20 Jul Emulators and Simulators vs Real Devices
Mobile device usage continues to grow exponentially. There are currently more than 3 billion mobile devices and tablets used worldwide today. So, it is no surprise that businesses are making sure they have a foothold in this ever-expanding market by creating mobile applications to be used on hand-held devices: eCommerce, banking & finance, healthcare, gaming, entertainment - all are making the move to include mobile to engage with the customer; customers who expect a seamless, bug-free experience. This brings a call to action to mobile application testing, and for the most part, the playbook has been written given web-based testing experience.
However, although basic testing strategies are largely the same, the tools needed for mobile application testing can wildly differ from that of the web, including the need for devices to test on. It is every tester's goal to execute fast, reliable, high-quality, and cost-efficient testing, so choosing the right devices, virtual or physical, can have a huge impact on delivering just that. Today, two solutions are offered: emulators/simulators and real devices. Let’s take a moment to explore what emulators/simulators actually are, the pros and cons of each solution, and which one to use when.
Ready to start testing on real devices now?
Start a Free Trial with Kobiton
What is an Emulator?
An emulator is a software program that imitates the hardware and/or software of a target device on your computer. Emulator programs behave in the same manner as a real device, as if the device was plugged right into the computer but set in an all-virtual environment. By creating a virtual environment that mimics a target device’s hardware and/or software, developers and testers are able to test basic functionality of an application running on an emulator. For example, Android offers an Android Emulator that is available to install, configure, and use from Android Studio SDK manager to test mobile applications.
What is a Simulator?
A simulator is a program that creates a virtual environment that mimics the internal behavior and configurations of an application on a target device’s operating system. A key difference between emulators and simulators is that simulators do not mimic the hardware of a target device. Instead, a simulator spins up a virtual environment of a target device’s operating system to run applications within the virtual operating system to build and test against. Like emulators, this virtual space can be used by developers and testers to test basic functionality of an application, though simulators lack the hardware functionality testing that emulators offer. An example of a simulator includes Apple’s Apple Simulator that is offered as part of Xcode tools and promotes developers and testers to test various iOS applications on simulated iPhone, iPad, Apple Watch, or Apple TV iOS environments.
Pros & Cons of Emulators/Simulators
There are many pros to using emulators/simulators. They are best used in the early stages of the SDLC to build and test applications as they are being developed. However, emulators/simulators come with quite a handful of drawbacks and limitations. Let’s take a look at the pros and cons of emulators/simulators:
Pros:
- Free, easily available, and open-source - Emulators/Simulators are usually provided by development manufacturers, so they are easily available, free to install, and are within an open-source community.
- Wide range of devices/ operating systems offered - Emulators/simulators offer a wide range of different devices and operating systems to emulate/simulate to help combat device fragmentation. However, these tend to be very basic, vanilla-type devices/ operating systems that may not emulate/simulate the real-world version.
- Provides functional and UI testing - As mentioned before, emulators/simulators are great for functional testing as well as testing UI layout of an application. Thus, emulators/simulators promote a free and easy way of testing within the early stages of development.
- Rich debugging and catching unexpected, functional behavior - Since emulators are within the SDK/ coding environment, they provide rich, step-by-step debugging information when testing an application on an emulated device. Simulators, however, do not provide as rich debugging info since simulators are used more for developer exploration on how an application is running within a simulated system to help find unexpected behaviors.
- Quick and convenient - Need to quickly test your code or test basic functionality of an app on a device? Once fully set up/configured, emulators/simulators are your go-to, since they are quick to spin up and easy to use. However, the initial setup of an emulator/ simulator can be tedious.
- No additional dependencies needed - Since emulators/simulators are offered within coding environment IDEs, there are no additional dependencies needed to use an emulator or simulator.
Cons:
- Cannot emulate/simulate real-time data - Emulators/Simulators cannot, or are restrictive in, producing real-time data, such as battery usage, network connectivity, bio authentications, GPS, sensors, etc.
- Lack of testing coverage - It’s great that emulators and simulators promote a free and easy way of testing UI layout and basic functionality across a wide range of target devices/OS, but functional and UI testing is only a fraction of overall testing that needs to be executed prior to releasing to production. The crucial, additional testing - like performance testing, user-based testing, system testing, etc - cannot be executed on an emulator/simulator; Thus, a huge drawback when considering using emulator/simulators for all testing needs.
- Testing accuracy, reliability, and speed - Emulators/Simulators are far less reliable and slower than testing on real devices. Each emulator/simulator tends to have its fair share of false positives/ negatives when testing applications, leaving developers and testers second-guessing and in need of re-testing for confidence, which proves to be counterproductive.
- Does not accurately reflect real-world scenarios - A tester will see drastically different results when executing regression on an emulator/simulator compared to a real device. This is because nothing can beat the real deal: testing all user-based scenarios as if the device was in the user's hand. Emulators/Simulators are far more limited in capabilities compared to real devices.
Pros & Cons of Real Devices
Of course, as said before, nothing beats the real deal like real devices. Testers are able to attain the truest, most accurate testing when executed on real devices, along with collecting real-time data to further be analyzed for optimization. However, real devices can be hard to come by and have their own share of drawbacks. Let’s take a look at the pros and cons of real devices:
Pros:
- The most reliable in testing - False positives and negatives are minimal compared to emulators/simulators when testing on real devices. On top of that, real devices offer real OS versions and not plain, vanilla-like OS versions only offered by emulators/simulators. With reliable testing comes increased confidence in testing efforts.
- Full testing coverage - Testing on real devices provides far more testing coverage than emulators/simulators. Not only can you also test UI and basic functionality on real devices, but also test real environments and conditions, along with more complicated testing scenarios. This includes bio auths, network/network failures, various weather conditions, push notifications, the device’s built-in sensors, camera functionally, geolocation, interruption (SMS/ incoming call) testing, and so much more. It goes without saying it’s imperative to execute mobile application testing, at the very least end-to-end and regression testing, on real devices.
- Most accurate understanding of UX - With the ability to test in real environments and conditions, you are able to accurately reproduce and understand user flow and experience. This enables you to put yourself in the shoes of your customer, so that you may deliver the best experience possible and reduce app abandonment when released.
- Performance testing - Only real devices offer accurate, real-time data of exhaust your application is putting off. This includes battery usage, CPU usage, network conditions and bandwidth, network payload transfer and processing speeds, latency speeds etc. Coupled with the most accurate understanding of user experience, this valuable data can be analyzed and optimized to deliver fast and efficient applications without compromising any functionality.
Cons:
- Costly - To put it bluntly, real devices cost A LOT. Not to mention the ever-impending doom of device fragmentation that mobile application testers face - back in 2015, there were 24,000 different Android smartphone and tablet models on the market. To try and supplement even a fraction of popular devices would cost quite a bit. So, we are now entering an era that recognizes this drawback, and thus, cloud-based device farms are now being offered at a subscription price. Although this lower price point helps resolve the device fragmentation blocker, it's still more costly than emulators/simulators.
- Limited availability - With how many different devices and OS versions there are on the market and each being pricey, real devices are extremely limited in availability. Even cloud-based device farms can fall short in offering the most real devices.
- Hard to manage physical resources - Real devices come with their own setup and storage that take up space, physically and virtually. They are in need of constant upkeep, with specific configurations and rapid released updates, which makes it tedious to manage multiple, physical, real devices. Many do not have the capacity to take on more than a few real devices because of this.
Wrapping Up - Which one to use?
It is clear that In the most ideal world, real devices would be in abundance and readily available to be used for all stages of testing at no cost. But, unfortunately, that is not the world we live in today. So, that begs the question 'which one to use?'. Do I put forth more money for real devices and get all my testing done on said devices? Or do I save my money and test on emulators/simulators and hope for the best when I push to production? The answer is: the two are not mutually exclusive and a combination of both is recommended. The true value of emulators/simulators is that they are a placeholder for testing needs until the need has exceeded what emulators/simulators offer. From there, it is ideal to procure real devices, in some way, to complete the continuous testing cycle. From very early stages of development and far-left testing, emulators/simulators will allow you to complete your testing at minimal cost to you. As we shift right, through the testing cycle, there is no replacement for the real deal that are real devices, even at a price; because the money put forth for real devices will save you money in the long run as you release cleaner, more efficient, and enjoyable mobile applications.
Looking to understand which mobile solution to use?