Top 10 Mobile Test Automation Tools

09 Apr Top 10 Mobile Test Automation Tools

Posted at 19:45h in Blog by

Introduction

The mobile application market has skyrocketed within the past decade. From mobile gaming to big enterprise applications, we live in a time where ‘there’s an app for that’ holds true. The Google Play store currently has roughly 3 million applications, and the Apple Store has roughly 2 million applications in the market. It is safe to say mobile app development has become one of the leading trends within the DevOps world.

Did you know that more than 50% of your peers are looking to automate up to 75% of their test cases?

Download the State of Automation 2021 Report

Given its huge and competitive market, it is imperative that mobile app developing teams create an exceptional, bug-free, user experience to retain customers and ensure top ratings within respective stores. To achieve this, mobile app developing teams aim to invest in quality assurance to perform a multitude of testing prior to releases.

However, choosing to invest in testing and actually executing that strategy are two different things. While testing tools and capabilities have flourished in the world of desktop web, Mobile Testing has proven increasingly difficult, due to many different factors, including:

  • Lack of comprehensive device infrastructure
  • Changing XPath selectors and technical identifiers across different device types
  • Complex gestures and behaviors that must be tested for (double press, infinite scroll, etc.)
  • Device fragmentation
  • Existing tool-stacks are optimized for desktop-web and often are mobile-immature

As a result, teams struggle to find the proper balance between manual testing and automation testing. Manual testing is “easy,” but incredibly tedious and time-consuming. However, automating tests can also be time-consuming, flaky, inconsistent, and excruciating to set up and maintain.

Luckily, mobile automation frameworks and tooling have begun to pop up in the market to relieve these mobile app pain points, making mobile app automation easier to achieve. While these frameworks and tools take different approaches to test automation, their goal is to significantly reduce testing time and provide optimal coverage.

In this article, we will focus on the top 10 best mobile automation tools to help your team release faster and with more confidence across multiple devices.

10. Mabl

What is Mabl?

Mabl is a revolutionary test automation solution that makes automated functional testing easier, faster, and more scalable for the enterprise. Through advanced AI, low code, and self-healing solutions, Mabl focuses on making functional UI and API test automation possible for both technical and non-technical teams.

Pros

  • Mabl’s offering includes self-healing, and this makes maintaining test scripts much easier.
  • Mabl’s low code offering makes automation a possibility for less technical teams.
  • Mabl’s Data-driven Testing engine allows you to parameterize and inject datasets into your team’s test automation for more coverage without the need for complex code.
  • Mabl is primarily a cloud-based solution, making its services easily accessible and scalable.
  • Mabl offers one solution for Desktop, API, and Mobile-web/Mobile-browser test automation authoring.

Cons

  • Mabl is still very limited for Mobile-first teams and just recently launched their first-ever Mobile compatible solution.
  • Mabl’s mobile solution only offers mobile-browser testing, and offers nothing for teams who need to automate testing for native or hybrid mobile applications.

9. XCTest / XCUITest

What is XCTest / XCUITest?

XCUITest is a native, as well as hybrid, framework for iOS mobile devices. Since XCUITest is baked right into Xcode's IDE, it provides the fastest possible testing for iOS devices. However, there are several limitations to be aware of in addition to its major benefits.

Pros

  • XCUITest is 100% free to use.
  • As mentioned above, XCUITest / XCTest is fast. It is the fastest solution for testing iOS mobile applications.
  • XCUITest is a great “anti-flake” option, since it helps develop UI tests that reflect the user interface by working directly at the object level. In doing so, tests have the ability to verify UI elements and their properties and navigate through the element hierarchy of your application. This helps create intuitive tests that are easy to maintain and reduces flakiness tremendously.
  • XCUITest provides XCUITest Recorder that records navigation through the user interface and helps users create scripts to quickly build test suites that can be run across actual devices or emulators/simulators.
  • While traditionally more difficult to integrate into CI/CD pipelines than its Android counterpart, Espresso, XCUITest has made headway into becoming easier to integrate into your CI processes.

Cons

  • XCUITest is a great “anti-flake” option when testing on emulators and/or running unit tests, but this benefit does start to dwindle when testing on real devices.
  • XCUITest is less stable and reliable when testing on real devices, not ideal for smoke and regression testing.
  • XCUITest is limited languages. As of now, XCUITest is limited to Objective C and Swift.
  • XCUITest is not compatible with Android devices.
  • XCUITest was built for the iOS and Xcode Developer in mind and focuses less on the QA Automation Engineer.

8. Applitools

What is Applitools?

Applitools is an automation framework that offers Applitools Eyes: a tool used for UI automation testing for web and mobile applications powered by visual AI. Offering 40+ SDKs to easily integrate with your existing automation test scripts, Applitools Eyes creates a DOM snapshot at designated checkpoints of your application UI and saves & sends each snapshot to Applitools robust, AI-driven server. There, Applitools breaks down your application UI snapshots into their layout and structure creating a baseline of images within each test. Thus, when running future tests on new versions and across different browsers and devices, Applitools is able to automatically detect and report visual differences.

Pros

  • As mentioned above, Applitools offers 40+ SDKs, so it can easily be integrated with other tools, such as Selenium WebDriver or Appium, as well as all major programming languages.
  • Applitools Eyes offers root cause analysis by clicking on a visual difference and seeing its corresponding, relevant DOM and CSS rule differences. This allows developers and testers to save significant time in digging for cause of a UI bug.
  • Applitools Eyes reduces coding significantly while still maintaining testing confidence- convert brittle, multi-lined UI assertions into one line of code calling Applitools Eyes and let Applitools do the rest.
  • Applitools is also great for data-driven testing. No need to code out conditional logic to try and cover all of a user’s potential flow. Instead, create visual snapshots of each variation to validate.
  • Applitools provides a streamlined dashboard that is easy to read with concise reporting. Each report clearly marks visual differences in compared screenshots.
  • Applitools also provides smart assist to help maintain tests - when the visual AI engine detects a way to improve tests, you are notified and can be implemented in a single click.
  • Applitools offers secure and reliable Ultrafast Grid- a cloud based grid that houses a combination of browsers, devices, and viewports. This allows you to run tests once, but in parallel across multiple configurations with virtually zero setup.
  • Applitools can also be easily integrated into the CI/CL pipeline, as well as other applications, like JIRA.

Cons

  • Applitools is script-based, thus some experience in scripting automation is needed to get started.
  • Applitools is limited to UI testing.
  • Applitools main focus is web application testing on different browsers and devices, provided by the Ultrafast Grid. Thus, Applitools is restrictive in mobile application testing. However, Applitools can be integrated with Appium to produce mobile application automation testing.
  • Since Applitools is mobile immature, it does not report application crashes.
  • Applitools does not offer real devices.
  • Applitools requires you to set specific baselines for each device type.
  • For mobile, Applitools requires teams to make API calls within their test automation and/or instrument their source code, both of which can cause serious problems (even in production)

7. Espresso

What is Espresso?

For teams exclusively developing applications for Android devices, Espresso is a great framework for automation testing at the UI level. However, there are a few limitations to keep in mind. Let’s explore the pros and cons.

Pros

  • Espresso is 100% free to use.
  • Much like XCUITest, Espresso is fast. Since it is a native framework, baked right into Android SDK and integrated with Android Studios, and it doesn’t require a server, Espresso can be much quicker than a solution like Selenium or Appium.
  • Espresso’s core API is small and compact, making tests easy to maintain, change, and customize.
  • With Espresso’s seamless UI synchronization, tests are less flaky and more intuitive with assertions.
  • Espresso also offers Espresso Test Recorder that allows you to navigate through your application, add in assertions, and Espresso Test Recorder will help generate a scripted test. This helps quickly build test suites that can be modified later, as well as help learn Espresso scripting for the junior QE.
  • Since Espresso is deeply integrated with Android, it also offers clear failure reporting and provides debugging information.
  • Espresso can also be easily integrated into the CI/CD pipeline.

Cons

  • Espresso is limited to Java and JUnit languages.
  • Espresso is not compatible with iOS devices.
  • Espresso is only for UI testing.
  • You must have access to your Application’s source code.
  • Using Espresso (as well as the Espresso Recorder) does require some prerequisite basic knowledge of Android applications, including how they are built and how to create tests for them.
  • Espresso has mostly been built with the Android Developer in mind, and is not near as QA-friendly as solutions like Appium.

6. Appium

What is Appium?

Appium is an open-source automation framework, or collection of client “automation libraries” as Appium calls it, used primarily to test both Android and iOS native mobile applications, as well as hybrid and web applications. Appium is part of the Selenium WebDriver family, thus utilizes Selenium WebDriver API that specifies a client-server protocol that uses vendor-provided frameworks under the hood. With no need to compile a third-party framework or code, you’ll be able to deploy the same app you have tested across cross-platform devices with confidence.

Pros

  • Appium is open-sourced and comes with a strong, supportive community to help facilitate your testing needs.
  • Using native frameworks, such as XCUITest for iOS, UiAutomator or Espesso for Android, and WinAppDriver for Windows, Appium allows you to write tests against multiple platforms all in one sitting. This cross-platform accessibility and code reusability is extremely beneficial in mobile app automation and eliminates device fragmentation in your testing.
  •  Appium client libraries are available in most major languages, including Java, JavaScript, Python, C#, and Ruby. Along with supporting multiple frameworks, this allows you the freedom and flexibility to set up and manage your test environment however you see fit.
  • Appium allows you to test on real devices and emulators/simulators with assisted setup and no extra dependencies needed.
  • Appium comes with Appium Desktop for those who prefer to live outside the command line. Appium Desktop features include
    • Ability to easily start/stop Appium server and view logs & screenshots
    • Appium Inspector which allows you to view your app's source tree: find any element and its locator within your application and perform basic interactions with them
    • Step Record: Record actions imposed on your application and let Appium auto-generate code for said actions
  • Appium supports end-to-end testing, including complicated gestures and behaviors for mobile applications.
  • Appium can easily be integrated with the CI/CL pipeline.

Cons

  • Appium is script-based that comes with its own learning curve, potentially challenging for the junior QE.
  • Although Appium allows the freedom to manage your environment, environment setup can still be challenging, tedious, and time-consuming.
  • Appium only offers real devices locally, making management of devices difficult if you’re in need of multiple, real devices.
  • Appium performs slower test executions since it involves a remote web-driver, which applies potential network issues and processing command time.
  • Appium scripting bears the issues of brittle and flaky tests resulting in instability, inconsistency, and involves more time in maintenance than development.

5. Functionize

What is Functionize?

Functionize is a scriptless visual automation framework that utilizes AI and machine learning algorithms for not only UI testing, but also for performance and load testing, as well. The power of AI with Functionize measures visual completion of each page, versus only the DOM, to help understand how users fully interact with your product.

Pros

  • Functionize intuitively records scriptless tests with their Intelligent Architect Product, which is useful for teams who need to quickly set-up testing.
  • In addition to recording, Functionize also provides an NLP-based (Natural Language Processing) test authoring solution that allows you to create tests using natural language (i.e. “Click” the “Log-in” “button”) and also provides word input for assertions using NPL. This simple, straightforward approach is great for junior quality engineers and allows teams to spend less time scripting and more time testing.
  • Functionize offers self-healing analysis and smart fix that identifies and presents most likely fixes. This helps reduce maintenance, save time, and create sturdy, non-flaky tests across the board.
  • Functionize provides insight into UI usability and performance with detailed break-downs of element load time and comparisons.
  • Since Functionize uses AI to collect data from live users and quality engineers, it is also able to generate performance and loading insights, such as real traffic, and helps improve user experience. This is extremely useful in shift-right testing to achieve continuous testing for your team.
  • Functionize uses cloud-based devices for iOS, Android, and major web browsers for optimal scalability.
  • Functionize can easily be integrated into CI/CD pipeline.

Cons

  • Functionize has been created with the cross-browser testing team in mind. Functionize only offers mobile testing at the browser-level and does not support native and/or hybrid mobile applications.
  • Functionize does not offer any real devices for automation testing, limiting use to virtualized devices that functionize offers via nested virtualization. While this technology is interesting, real devices are still necessary to test against true, real-world conditions and scenarios.

4. Katalon

What is Katalon?

Katalon offers a flexible platform aimed at web, API, mobile, and desktop app testing that fits teams and projects of any size, for any purpose — from creating tests, execution, reports, to seamless integration with the CI/CD ecosystem.

Pros

  • Katalon offers a free version.
  • Beyond the free version, Katalon also offers more enterprise-grade features like TestOps management and orchestration.
  • Katalon offers one platform for web, API, mobile, and desktop application testing.
  • Katalon’s automation capabilities are built using a “Step Composition” method, and this method has the following pro’s:
    • This can work well when creating automation for one device
    • Saves time when compared to scripting
    • Can create reusable components
    • Tester has more control over the test and assertions when compared to bots

Cons

  • Katalon started with desktop web and API testing, and its mobile test automation offering is still less mature.
  • Katalon does not offer real devices and must integrate into existing device providers and device lab management solutions.
  • While Katalon’s Step Composition has its pro’s (see above), there are also several cons:
    • With mobile, there are many-layered elements, and this can leave the tester asking “which element should I choose?”
    • Can only reliably create scriptless automation for one device at a time
    • Very time consuming
    • Cannot handle device and operating system-specific popups

3. 21Labs

What is 21Labs?

21Labs is a scriptless automation tool aimed at autonomous testing and maintenance to provide instant coverage in less time. 21Labs promises AI-based learning to learn the structure of your application through functionality of each screen and navigation. In doing so, 21Labs claims to execute a wide range of self-teaching mechanisms to keep track of elements and their relative hierarchy, allowing you to create stable, consistent, self-healing, and autonomous tests.

Pros

  • 21Labs comes pre-integrated with devices and emulators.
  • 21Labs can also be customizable via Appium scripting, and provides a wide range of actions and assertions.
  • 21Labs offers the ability to apply production data to tests. 21Labs will identify flows in prod that are more frequent and have a high traffic rate. Production data also highlights test coverage and areas that lack-there-of, providing more insight into user experience and coverage that can be applied in shift-right testing.
  • 21Labs offers a root cause analysis - if a run fails, 21Labs will automatically look at system messages to confirm if failure was due to a pop-up. If so, 21Labs will attempt to disable the pop-up and rerun the test again.
  • 21Labs comes with reporting that includes screenshots & recordings, Appium logs & logcat, and reproduction steps that can be easily integrated with project management tools, like JIRA
  • 21Labs also provides a dashboard that offers KPI metrics like total tests run, success rate, coverage index, and pass/fail, etc.
  • 21Labs is easily integratable into the CI/CL pipeline, as well as other 3rd-party device labs.

Cons

  • Based on user feedback, 21Labs is not “click through and automate” as advertised and can involve complex coding.
  • 21Labs promises scriptless, AI-based authoring, yet in reality, 21Labs utilizes the  “Step Composition” method, similar to Katalon. Please see Step Composition pros & cons above.
  • Given the “Stop Composition” method, this makes finding an element difficult and becomes technical identifier dependent, which results in less intuitive, flaky tests, especially for dynamic mobile applications.
  • Emulators and device offering is plan-dependent.
  • Automation must be built per device type, resulting in tedious testing setup.

2. Waldo.io

What is Waldo.io?

Waldo is a no-code platform for automation mobile application testing by building end-to-end regression suites that can be automatically run against new versions of your mobile app. Waldo.io's codeless recorder records "flows" to create one test flow, or chain together multiple test flows to create dependencies, when building your tests. Create "flows," or test cases, and every time you upload a new version of your app, Waldo.io will automatically kick off your tests.

Pros

  • Waldo.io claims easy onboarding. You just need to upload your mobile app file to Waldo.io in a browser and begin recording tests. This is great for quickly getting testing setup and started.
  • Waldo.io does not require in-depth knowledge of scripting in order to start creating automation tests. A great solution for non-technical teams or for teams who do not have testing capacity.
  • Waldo.io offers Visual Testing solutions that can help catch bugs that traditional, functional tests cannot.
  • When tests fail, Waldo.io provides quick, auto-generated assertions to help maintain tests. Additionally, when a test fails, given a different behavior of the new version, Waldo.io makes it easy to update the ‘source of truth’ directly from a failed run.
  • Since Waldo.io runs behind the scenes, when new versions of your application are compiled, recorded tests automatically kick off. A solution alternative to achieve continuous integration without the technical setup.

Cons

  • Very little information about Waldo.io on their website around capabilities, especially when it comes to real device access.
  • Waldo.io’s pricing is complicated.
  • Relying on only end-to-end UI testing can make it tricky and tedious to catch and investigate back-end bugs, resulting in a less efficient iterative process by finding, fixing, and compiling from the surface level.

1. Kobiton

What is Kobiton?

Kobiton is the industry leader in pure-play mobile test automation and mobile Continuous Testing. With support for both AI-driven scriptless and AI-assisted script-based approaches, Kobiton offers solutions around test authoring, execution, and remediation that will fit the needs of any mobile development or testing team.

The best way to understand the pros and cons of Kobiton automation testing is to break up pros and cons by the two workflows possible: scriptless and script-based. Let’s take a look at both workflows, as described by Kobiton's CTO Frank Moyer in Kobiton's Innovation Panel Session at Odyssey 2021:

Scriptless automation Pros

  • Manual “Capture and Execute” approach makes scriptless automation scalable and can easily run in parallel across multiple devices. This allows for baked-in Compatibility Testing with every Kobiton scriptless automation test that you run.
  • Manual Capture and Execute makes “Run once, test anywhere” possible by allowing you to easily re-run one manual test across your library of real devices.
  • Users can easily make the move from manual testing to Appium automation without any need to code using Kobiton’s 100% open-standard Appium Export technology.
  • Each scriptless test comes with baked-in Visual, Text, Performance, Crash, and UX Design assertions, allowing one Kobiton test to do the leg-work of what would traditionally require 3-4 different kinds of testing.
  • The NOVA AI Engine behind Kobiton scriptless automation is adaptable and makes scalable execution across multiple devices possible.
  • Kobiton Scriptless automation can run in parallel across multiple devices.
  • Kobiton’s baked-in assertions allow for intelligent remediation within our Remediation Menu. This allows teams to remediate for Functional, Visual, Performance, and Crash issues all within one place in the Kobiton UI.
  • Kobiton acts as the “AI layer on top” of existing real device infrastructure, allowing you to run Kobiton scriptless automation against both Kobiton devices and devices from other DLM providers that we support (currently Sauce Labs, but we are adding more throughout 2021).
  • On top of Functional, Visual, Crash, and Performance assertions, Kobiton is adding in Accessibility assertions later in 2021.
  • Kobiton offers Data-driven testing for scriptless automation that allows you to easily parameterize your scriptless automation test without needing to code.
  • Kobiton offers a runtime environment and real device infrastructure.

Scriptless automation Cons

  • As of now, Kobiton Scriptless automation doesn’t allow for as granular of parameterization and control as very technical teams can accomplish with scripts.

Script-based Pros

  • Teams can take existing Appium scripts and inject NOVA’s assertions without any need to do any complex scripting.
  • Injecting assertions only requires changing certain Kobiton-specific desired capabilities values from “false” to “true”.
  • Easily turn already-existing Functional test scripts into:
    • Visual Tests
    • Performance Tests (coming 2H 2021)
    • Crash identification tests (coming 2H 2021)
    • Accessibility Tests (coming 2H 2021)
  • Remediate for issues raised by Kobiton’s assertions within the Kobiton remediation menu interface.
  • Kobiton offers a custom Appium server that is >2-3x faster than normal Appium, making Appium on par with native frameworks like Espresso and XCUITest (both mentioned earlier).
  • Insert Kobiton’s benefits into your already-existing scripts alongside existing parameterization and dynamic data.

Script-based Cons

  • Unless users leverage the Kobiton Appium Export capability, creating Appium scripts from scratch requires technical know how and time.
  • Kobiton’s AI-assisted script-based approach only works for Appium, and does not yet support alternative frameworks, such as Espresso and XCUITest.

If you're interested in seeing the script-based workflow in action, check out this demo run by Leslie Tan, Global Director of Solutions Engineering at Kobiton, at Odyssey 2021:

Wrapping up

There are many different test automation solutions out there, but only a handful that are truly optimized for mobile application automation. While we understand Kobiton’s offering as providing the most feature-rich, comprehensive, and flexible solution, we recognize that teams might have specific needs that fit better alongside other solutions. This is why we recommend taking a thorough look at your team’s makeup, skill-set, and needs before evaluating and ultimately choosing a solution.

Did you know that more than 50% of your peers are looking to automate up to 75% of their test cases? Find out how test automation is affecting your competitors.