Mobile Testing Maturity Model

Organizations are quickly coming to the realization that a customer’s negative experience using a mobile app manifests in many forms: Massive app abandonment (i.e. switching to competitors), amplification of negative reviews (i.e. less prominent positioning in the app store), and even negatively impacting Net Promoter Scores (NPS).

With this realization also comes the harsh truth: Mobile testing and mobile DevOps isn’t trivial.

Fortunately, taking a systematic approach makes Mobile Testing less daunting. And by not using a ‘boil-the-ocean’ approach, organizations can quickly be on the road to broad and deep mobile testing. In this article we explore the Mobile Testing Maturity Model – organizations will find themselves at various points of mobile testing maturity and this model can serve as a good roadmap to achieving mobile testing nirvana.

Mobile Testing Maturity

The model depicted above does not imply that organizations must move through the maturity model in a sequential sequence. It does however let an organization frame where they currently are in their mobile testing maturity and help to form a game plan for reaching the highest levels of maturity. Let’s explore each tier in turn.

01 Manual Testing

A level 1 organization is starting to understand the importance of testing application or web behavior on mobile devices. This is also realizing the need to start incorporating real device testing into the process and not relying exclusively on emulators. Manual testing at this phase may be as simple as physically executing the app on a device – however, a better approach would be to have a more scalable approach with consolidation of test results. That is, allowing developers or testers to do their manual testing without being physically tethered to the device, and to report all test logs and results in a centralized manner.

02 Automated Testing

While testers have long mastered the art of automation testing, it isn’t as easily replicated on mobile devices. Challenges include the need to have test scripts have access to the devices which, when relying on physically present devices, cause severe scalability limitations. Moreover, automation testing requires automated feedback – again, very challenging when dealing with physical devices in multiple locations. This is part of the appeal of testing on Emulators in that they integrate into automated testing in a seamless manner. However, as organizations mature in their mobile testing capabilities, the requirement to test on real devices becomes imperative.

03 Automated Testing at Scale

It should be somewhat self-evident that the complexities inherent at level 02 only become severely magnified on level 03. It is for this reason that organizations should implement a mobile testing platform before attempting automated testing at scale. A natural question organizations may ask is “Why do we need to do mobile automated testing at scale” – and the answer is simple: Your customer base has over 4,500 combinations of phones and operating systems. At a minimum, at least 300+ combinations need to be tested to ensure adequate coverage across the most used devices. The only way to achieve effective testing is via scaled automation testing. Organizations reaching this level of maturity have no other viable option other than implementing a mobile device lab. It should be noted however that this does not necessarily mean physically hosting a device lab. Modern mobile testing platforms offer the ability to completely virtualize physical devices, combined with on-site devices to create a true hybrid test cloud.

04 DevOps and Mobile Testing

These organizations have reached mobile testing nirvana! This level indicates the successful inclusion of automated mobile testing at scale within sophisticated CI/CD processes. These environments are normally a combination of best-of-breed systems that are well integrated and well orchestrated. Companies at this level of maturity are able to deliver apps with certainty (confidence of quality and acceptance), and efficiency (quickly and at a lower cost).

How do we get there?

The recommendation for embarking on mobile testing and improving along the maturity model is this: Do so pragmatically. If you haven’t even started at level 1 yet, don’t jump into level 4. However, it is prudent to plan top-down (from level 4), but implement bottom-up (from level 1). This ensures that the technology choices you make now are compatible and in synch with the broader DevOps strategy that is in place or being implemented.