The War of the Test Frameworks: XCTest vs Espresso vs Appium

Introduction

We know the running war of the test frameworks: XCTest vs Espresso vs Appium. When deciding to start your journey towards mobile test automation, it’s important to have all of the information in front of you before choosing your automation tooling. If you (like many other teams out there) are currently trying to decide between Appium, Espresso, and XCTest / XCUITest, it’s very helpful to weigh the pros and cons of each approach. To try and make this easier, we have done the hard work for you and compiled information around what each solution is and what the pros and cons are.

Looking at the market, the predominant tool is still Appium, but native frameworks like Espresso (for Android) and XCTest/XCUITest (for iOS) are beginning to gain ground for a variety of reasons.

Looking for the best mobile application testing framework for your business case?

Download the Buyer’s Guide to Mobile Test Automation

XCUITest/XCTest

What is XCUITest/XCTest?

XCUITest is a native framework for UI testing on iOS mobile devices. Since XCUITest is baked right into Xcode IDE, it provides the fastest possible testing for iOS devices. However, there are several limitations to be aware of in addition to the several major benefits.

Pros

• As mentioned above, XCUITest / XCTest is fast. As of now, it is your fastest solution for testing iOS mobile applications.
• XCUITest is a great “anti-flake” option, since it helps develop UI tests that reflect the user interface by working directly at the object level. In doing so, tests have the ability to verify UI elements and their properties and navigate through the element hierarchy of your application.This helps create intuitive tests that are easy to maintain and reduces flakiness tremendously.
• XCUITest also provides XCUITest Recorder that records navigation through the user interface and helps users create  scripts to quickly build test suites that can be run across actual devices or emulators/simulators.
• While traditionally more difficult to integrate into CI/CD pipelines than its Android counterpart, Espresso, XCUITest has made way headway into becoming easier to integrate into your CI processes.

Cons

• XCUITest is a great “anti-flake” option when testing on emulators and/or running unit tests, but this benefit does start to dwindle when testing on real devices.
• XCUITest is less stable and reliable when testing on real devices, making it not ideal for smoke and regression testing.
• XCUITest is limited in it’s support for a variety of languages. As of now, XCUITest is limited to Objective C and Swift.
• XCTest / XCUITest is pure iOS and cannot help the team that needs to test both iOS and Android devices.
• XCUITest was built for the iOS and Xcode Developer in mind and focuses less on the QA Automation Engineer.

Espresso

What is Espresso?

Like XCTest, Espresso is a native framework for UI testing. However, instead of iOS devices, Espresso is pure Android. Espresso is baked right into the Android SDK and integrated into Android Studio. Espresso has major benefits for teams looking to develop and test on Android devices. However, there are a few limitations to keep in mind. Let’s explore the pros and cons.

Pros

There are several major pros to using Espresso for your mobile automation testing:

• Espresso is 100% free to use
• Much like XCUITest, Espresso is fast. Since it is a native framework, baked right into Android SDK and integrated with Android Studios, and doesn’t require a server, Espresso can be much quicker than a solution like Selenium or Appium.
• Espresso’s core API is small and compact, making tests easy to maintain, change, and customize.
• With Espresso’s seamless UI synchronization, tests are less flaky and more intuitive with assertions.
• Espresso also offers Espresso Test Recorder that allows you to navigate through your application, add in assertions, and Espresso Test Recorder will help generate a scripted test. This helps quickly build test suites that can be modified later, as well as help learn Espresso scripting for the junior QE.
• Since Espresso is deeply integrated with Android, it also offers clear failure reporting and provides debugging information.
• Espresso can also be easily integrated into the CI/CD pipeline.

Cons

• Espresso is limited to Java and JUnit languages.
• Espresso is not compatible with iOS devices.
• Espresso is only for UI testing.
• You must have access to your Application’s source code
• Using Espresso (as well as the Espresso Recorder) does require some prerequisite basic knowledge of Android applications, including how they are built and how to create tests for them.
• Espresso has mostly been built with the Android Developer in mind, and is not near as QA-friendly as solutions like Appium.
• Espresso is pure Android and cannot help the team that needs to test both Android and iOS devices.

Appium

What is Appium?

Appium is an open-source automation framework, or collection of client “automation libraries” as Appium calls it, used primarily to test both Android and iOS native mobile applications, as well as hybrid and web applications. Appium is part of the Selenium WebDriver family, thus utilizes Selenium WebDriver API that specifies a client-server protocol that uses vendor-provided frameworks under the hood. With no need to compile a third-party framework or code, you’ll be able to deploy the same app you have tested across cross-platform devices with confidence.

Pros

There are several major pros to using Appium for your mobile automation testing:

• Appium is open-sourced and comes with a strong, supportive community to help facilitate your testing needs.
• Using native frameworks, such as XCUITest for iOS, UiAutomator or Espresso for Android, and WinAppDriver for Windows, Appium allows you to write tests against multiple platforms all in one sitting.
• Appium’s cross-platform accessibility and code reusability is extremely beneficial in mobile app automation and eliminates device fragmentation in your testing.
• Appium client libraries are available in most major languages, including Java, JavaScript, Python, C#, and Ruby. Along with supporting multiple frameworks, this allows you the freedom and flexibility to set up and manage your test environment however you see fit!
• Appium allows you to test on real devices and emulators/simulators with assisted setup and no extra dependencies needed.
• Appium comes with Appium Desktop for those who prefer to live outside the command line. Appium Desktop features include
  • Easily start/stop Appium server and view logs & screenshots
  • Appium Inspector allows you to view your app’s source tree: find any element and its locator within your application and perform basic interactions with them
  • Step Record: Record actions imposed on your application and let Appium auto generate code for said actions
• Appium supports end-to-end testing, including complicated gestures and behaviors for mobile applications.
• Appium can easily be integrated with the CI/CL pipeline
• Because Appium is the most popular framework used for mobile test automation, most Mobile Testing vendors have the most support for Appium, making Appium testing more convenient when using it alongside solutions that allow for real device access and Session Exploration.

Cons

• Appium is script-based that comes with its own learning curve, potentially challenging for the junior QE.
• Although Appium allows the freedom to manage your environment, environment setup can still be challenging, tedious, and time-consuming.
• Appium only offers real devices locally, making management of devices difficult if you’re in need of multiple, real devices.
• Appium performs slower test executions since it involves a remote webdriver, which applies potential network issues and processing command time.
• Appium scripting bears the issues of brittle and flaky tests resulting in instability, inconsistency, and involves more time in maintenance than development.

AI-enhanced Appium with Kobiton

While we believe that teams should choose whatever framework suits them best (and we will help make Appium, Espresso, and XCTest / XCUITest better), we have focused heavily on the Appium automation engineer. This year, we started releasing what we call “AI-enhanced Appium,” and this makes your Appium automation easier to author, more comprehensive in its coverage, faster in its execution (hallelujah!), and easier to remediate for when issues are encountered.

Test authoring with Kobiton AI-enhanced Appium automation

It all starts with test authoring. With Kobiton, we’ll let you inject our NOVA AI Engine into your existing Appium automation scripts, but we can also help you get to that script faster to begin with by leveraging our auto-Appium generation or Appium Export technology.

Kobiton’s Appium Export technology allows you to run a manual test on a Kobiton real device, tell our AI to auto-generate an Appium script, and then export that script into your IDE for quick “base script” authoring. Not only is this easy, but it’s also a huge time saver for teams who are trying to quickly build out their libraries of Appium automation.

Beyond just getting to your Appium “base script,” we also make it easy to author assertions into your Appium script. We actually just released the Beta version of our “Inject NOVA Visual Assertions into your Appium” capability, and will be releasing the General Availability version in our April 19th, 2021 release. This allows you to take advantage of the Visual Assertions and Visual Testing capabilities previously unique to the Kobiton Scriptless Automation flow and easily inject them into your Appium scripts. And, unlike other solutions that do similar things to this, Kobiton allows you to do this by simply making a quick change to your Desired Capabilities and also doesn’t require setting a new baseline for each and every mobile device. With Kobiton, you can run a Visual Test using one device’s visual output (let’s say a Samsung Galaxy S9) as a baseline and compare that baseline across multiple different device types (let’s say LG and Moto G4 devices).

Yes, that means no more making API calls, no more source code instrumentation, and no more endless baseline screenshots.

Test execution with Kobiton AI-enhanced Appium automation

After the test is authored, it’s got to execute quickly. That’s why Kobiton offers “souped-up” Appium that can run greater than two times as fast as a standard Appium execution. In addition to this, injecting NOVA into your script also allows for advanced adaptation and auto-correction capabilities that make your Appium much more reliable and stable when running in parallel across different device types.

Test remediation with Kobiton AI-enhanced Appium automation

Okay, so you’ve run NOVA AI-enhanced Appium and it’s been able to run across a bunch of different devices while also invoking Visual assertions to capture visual issues and discrepancies.

Now what?

With Kobiton, you’ll actually be able to easily remediate those visual issues identified by Appium with our Assertion Overview and Remediation interface. With this interface, you can easily see where NOVA identified visual discrepancies across device types and/or application versions, and you can even give NOVA instructions on how to better execute your tests going forward with directions around the strict and relaxed comparison.

To see this all in action, check out this snippet of a “visual assertions in Appium” demo run by Leslie Tan, Global Director of Solutions Engineering at Kobiton:

https://videos.sproutvideo.com/embed/449ddcb51818e4cecd/bf992867c81121ef

AI-enhanced Appium Pros

• Teams can take existing Appiums scripts and inject NOVA’s assertions without any need to do any complex scripting.
• Injecting assertions only requires changing certain Kobiton-specific desired capabilities’ values from “false” to “true”.
• Easily turn already-existing Functional test scripts into:
  • Visual Tests
  • Performance Tests (coming 2H 2021)
  • Crash identification tests (coming 2H 2021)
  • Accessibility Tests (coming 2H 2021)
• Remediate for issues raised by Kobiton’s assertions within the Kobiton remediation menu interface.
• Kobiton offers souped-up Appium that is >2-3x faster than normal Appium, making Appium on par with native frameworks like Espresso and XCUITest (both mentioned earlier).
• Insert Kobiton’s benefits into your already-existing scripts alongside existing parameterization and dynamic data.
• Kobiton’s Appium generations and Export capabilities make authoring Appium automation much easier.

AI-enhanced Appium cons Cons

• Unless users leverage the Kobiton Appium Export capability, creating Appium scripts from scratch requires technical know-how and time.
• Kobiton’s AI-assisted script-based approach only works for Appium, and does not yet support alternative frameworks, such as Espresso and XCUITest.

Wrapping up

There are many different test automation solutions out there, but only a handful that are truly optimized for mobile application automation. While we understand Kobiton’s offering as providing the most feature-rich, comprehensive, and flexible solution, we recognize that teams might have specific needs that fit better alongside other solutions. This is why we recommend taking a thorough look at your team’s make up, skill-set, and needs before evaluating and ultimately choosing a solution.