24 Jul How Much Should You Be Paying For Real Device Testing?
If you have questions about real device testing, you’ve come to the right place. Read on for insights on:
- The rising importance of quality when it comes to releasing new mobile apps
- Different options for mobile app testing, including real devices, mobile emulators and simulators and the pros and cons of each option
- Insight on how to realize the benefits of real device testing while minimizing the costs
- Four factors to consider when determining how much you should really be paying for real device testing, including plan capabilities, types of devices, length of access and level of accessibility
Mobile now rules the world.
Walk into any public place and you’re likely to see more people using mobile devices -- be they phones or tablets -- than not. Most of us sleep with our phones right next to us, and checking our phones is the last thing we do before we drift off for the night and the first thing we do when we wake up. Essentially, our mobile devices have become our preferred way of accessing the digital world, and it’s a world we’re often hard-pressed to leave.
This backdrop has created the perfect breeding ground for mobile apps that we can use for anything and everything. For proof of the proliferation of mobile apps, look no further than the Apple App Store, which is predicted to reach 5 million apps by 2020. Remarkably, this number represents a 99,900% increase compared to the number of apps available in 2008 when Apple first launched the App Store.
Notably, apps aren’t just flooding the market for no reason: Consumer usage and spend on mobile apps is predicted to scale alongside this growth. Just look at the following recent research from App Annie:
- The user base for mobile apps will nearly double from 2016 to 2021 (going from 3.4 billion people to 6.3 billion)
- Time spent in mobile apps will increase from 1.6 trillion hours to 3.5 trillion hours in 2021 (yes, you read that right… trillion hours).
- App spending will rise to an average of $1,008 per user in 2021
Together, this growth will contribute to a 385% increase in the market value of the global app economy, which is predicted to be worth $6.3 trillion by 2021.
As consumers, this growth sounds exciting, if not a bit overwhelming. But as software developers and testers, this growth sounds like an opportunity, since every one of those mobile apps will require testing.
The Rise of Mobile Testing
Given the explosion of mobile apps in such a short time period, it’s not surprising that mobile testing has experienced a meteoric rise in importance in the broader software testing world.
In a few short years, mobile testing capabilities, including mobile testing tools for both iOS and Android apps, have become a must have for any organization that’s serious about launching a successful mobile app. After all, today’s tech savvy consumers no longer tolerate poor quality software. These users expect apps to load in seconds (typically two seconds or less), are quick to delete apps that are ripe with bugs or crash often and are highly inclined to leave ratings and reviews in app stores (and those ratings and reviews go a long way toward app success). As a result, comprehensive testing on mobile devices is an absolute must.
Options for Mobile Testing
As important as it is, mobile testing presents a unique challenge: How do you create the right kind of mobile testing experience to ensure you’re releasing the highest possible quality product to your users?
There are now so many types of devices out there (a host of devices running iOS, Android and even Google Pixel and Windows) and so many different operating systems (and versions of those operating systems) on which those devices might run. Nevertheless, to ensure proper testing on mobile devices and bring the best possible version of apps to market, organizations need to test on all of these mobile devices (or as many as possible).
To meet this need, three solutions have emerged for conducting mobile testing, each of which has its pros and cons:
1) Real Devices
The most straightforward solution involves testing on real devices. These may be real mobile devices that you can hold in your hand or real mobile devices that you can access via the cloud (the latter option is known as a mobile device cloud, and while you may not be able to “hold them in your hand,” these are still real, physical devices, offering access to all of the same features and operating systems).
Pros of real device testing include:
-
- Full re-creation of the user experience (e.g. by testing battery usage, push notifications, etc.)
- Better visualization of the look and feel of the app, including different screen resolutions and brightness
- Better reliability and device speed
Cons of real device testing include:
- Expensive (especially in cases where you own your own devices, these devices can be extremely costly to purchase and maintain)
- The need to account for a variety of devices and operating systems
2) Mobile Emulators
Next we have mobile emulators, which are designed to mimic different mobile devices in terms of both software and hardware. They are simulated environments that take on most of the same features as mobile devices, attempting to allow mobile apps to run as they would on the real device.
Pros of mobile emulator testing include:
- Relatively inexpensive and often, free (many options are open source)
- Makes testing on a variety of devices and operating systems less cumbersome
- Mimics both the software and hardware of a device (as opposed to a simulator, which just mimics the software)
Cons of mobile emulator testing include:
- Often comes with its own bugs that are native to the emulator software and show issues that don't occur on the real hardware, leading to false positives
- Typically requires a long setup process and is slow to use once the system is up and running (Android emulators, in particular, tend to be significantly slower than the real device)
- Access to a limited number of operating systems and versions
- Unable to mimic all real life scenarios, such as battery usage and call interruption
3) Simulators
Finally, we have simulators, which create a similar experience to the real device in question, but only consider software, not hardware. Because simulators only mimic the software of a device, not the hardware too, mobile apps may run slightly differently on a simulator than they would on a real device.
Pros of simulator testing include:
- Speedy setup and testing experience
- Makes testing for a variety of devices and operating systems less cumbersome
Cons of simulator testing include:
- Apps don’t always run as they would on a user’s device since the system only mimics the software, not the hardware
- May surface false positives
- Does not provide a realistic simulation of performance under memory constraint
While each option certainly has use cases in which it works best, most testing teams have shown a preference for real device testing, as this option allows for the most comprehensive QA.
Despite this preference, many teams are limited in how often they can do real device testing because testing apps on real devices can be expensive and tedious. Most importantly, teams either have to own the devices and have physical access to them or they have to pay a lot of money to get access to real devices through a service. And the cost of purchasing and maintaining these devices can get extremely costly very quickly.
Many teams settle for emulators without truly understanding what alternatives are available and how those options can increase the quality of their product.. But as the app market continues to explode, with users spending more time and money on apps, and consumer expectations continue to rise, the type of comprehensive mobile testing for which real devices allow will become increasingly important. Conversely, the shortcomings of mobile emulator testing will become even more glaring.
This situation begs the question: How do you know how much you should really be paying for real device testing? The answer? It depends.
4 Factors to Consider When Evaluating the Cost of Real Device Testing
Device fragmentation is very real in today’s technology-driven world, and that makes it cost and time prohibitive to try to buy or access all the device hardware and software configurations that you need to do thorough testing on mobile devices.
For instance, the cost of running your own device lab is typically tens of thousands of dollars a year due to (a) the costs of the devices themselves and (b) the people costs of managing and maintaining those devices.
As a result, the best option for mobile testing teams is to work with a partner who can provide access to those real devices for you. A partner alleviates many of the pain points associated with real device testing by not only providing access to devices, but also by handling all of the maintenance and upkeep. This setup then allows your team to focus on the testing and what needs to get fixed so that you can release your products quickly without getting bogged down in the behind-the-scenes maintenance.
The bottom line? Your team shouldn’t have to settle for mobile emulators when you can get affordable, easy access to real devices. It’s not just that you shouldn’t have to settle. As app quality becomes even more important, you won’t be able to settle since emulators will not deliver the type of comprehensive mobile testing necessary to confidently deploy apps to savvy consumers.
However, not all of these partners’ platforms are equal in terms of the variety of devices they offer (real devices vs. mobile emulators vs. simulators) and the costs and dimensions of their plans. For example, some partners do offer access to real devices but restrict what kind of testing you can do, what kind of access you have to the devices and how long you have until your minutes expire.
So what do you need to know to properly evaluate different options and make an informed decision about how much you should be paying for real device testing? It’s all about looking at the bigger picture. Specifically, you should consider the following four factors:
1) What testing is included in the package?
Start by looking at the type of testing options in the package or pricing plan. For example, you want to pay careful attention to what type of testing you can run . Many services restrict the amount of automated testing that you can run on a single plan, so it’s important to distinguish between those that set limits for automated testing and those that allow for unlimited manual AND automated testing.
Other important test capability considerations include:
-
- Device control: What level of control will you have over the devices during testing?
- Reporting capabilities: What type of reporting is available for manual and automated testing? How detailed is the reporting?
- Long term access: Once you finish with a device, what happens next? Will you be able to view session history a few weeks or months down the line?
2) What kind of devices are included?
After you review testing options, it’s important to do some device validation to understand whether you’ll have access to real devices or mobile emulators. Some services that offer real devices only do so for enterprise teams, preventing access for independent testers and small to mid-size teams.
While mobile emulators do a very good job of mimicking real devices, they’re still not the real deal. Think of it this way: When consumers use an app, they’re not sitting still in a perfectly lit room with nothing else running on their phones. More likely, they’re on the go (moving from one Wifi network to another or in and out of Wifi) and have text messages, calls or push notifications coming in from other apps. An emulator essentially thinks that the user experience is the first of these options and offers mobile testing that mimics that type of siloed experience. Only a real device can take into account all of the external factors that consumers actually experience while using apps.
3) What will your access to the devices look like?
Finally, it’s important to consider device access, including who can access the devices, where they can access the devices and what happens with the devices you own.
In terms of who can access the devices, most providers indicate the number of users allowed under each plan. This consideration is important whether you’re a team of one or a team of 20, as you need to make sure there’s a plan that is small enough or large enough to accommodate your needs. For example, some providers don’t even offer automated testing to single users or smaller teams (unless of course you want to pay for a plan priced for a bigger team).
Next you need to consider accessibility, and this consideration is a big one. Today, it’s common for teams to be spread out across different locations and for the same testers to work from different places throughout the day, and mobile testing should be no different. As a result, if you want to maintain the high level of productivity that comes from being able to collaborate and work from anywhere at any time, you need to ensure that you have anywhere access to the mobile devices.
Last but not least, what happens if you own any devices? If you own a few devices, you should certainly be able to do testing on those too, right? Make sure that you can (a) add your own devices to the vendor’s device cloud and (b) continue to access those devices from anywhere. Of course you’ll also want to determine if there’s any additional cost for adding in and testing on mobile devices you already own.
Affordable Real Device Testing
Even though real device testing is the preferred approach for testing on mobile devices due to its ability to provide the most comprehensive testing experience, too many teams are forced to make do with emulators and simulators due to the high price of real device testing.
But the truth is, real device testing doesn’t have to be so unattainable. Fortunately, if you know how to evaluate different options for real device testing -- from owning your own devices to partnering with a device lab provider to some combination of the two -- you can find a plan for real device testing that doesn’t have to break the bank. It’s simply a matter of knowing where to look and how to break down what each partner provides.