Addressing Testing in the Face of Device Fragmentation

A wide variety of devices at different price points is great for consumers. But this fragmentation poses a major challenge for all mobile developers trying to make sure their apps work well across all these devices. It is an especially thorny problem for an independent developer. Constrained by financial resources and facing pressure to get an app out quickly can lead to hasty, inadequate real device testing, a critical mistake when launching or updating an app.

By some estimates, there are upwards of 50,000 mobile devices types in use today, running varied combinations of operating systems. Developers and testers in large firms may have access to robust labs of many actual devices to test on, but that’s just not a realistic option for an independent developer. But not testing on enough real devices leaves a real risk of a terrible first impression and negative reviews that ruin any chance they may have for success.

But, if you are an independent developer, here are some steps you can take to help mitigate the challenge posed by device fragmentation.

Step 1: Test first on an emulator, early and often

We all know there is no replacement for testing on actual devices, but using an emulator early on is a budget-friendly way to identify initial bugs that need to be addressed. If you’re not already doing this, I strongly recommend it as a good first step.

Step 2: Prioritize the devices you must test on and those that would be nice to test on

After the emulators, you do need to get your app on real devices for testing before releasing it to the public. We’ve already agreed that you can’t test on everything, so what devices should you test on?

Step back for a moment and think about your intended audience, and from there develop a realistic and manageable set of configurations that you MUST test on:

1. Will your customer access your app through iOS or Android, or both?
2. Are you focused exclusively on mobile phones or will your app be available on tablets? Or both?
3. Are they more likely to have updated hardware and OSes, or are they likely to be late upgraders?

By considering these three questions, you can quickly create a list of device combinations that you need to cover as part of your real device testing.

If you’re wondering what devices are new and popular, a site such as gsmarena.com can provide an up-to-date list of Android devices that are popular.

A combination of minimal number of devices, form factor and OSes can quickly be overwhelming for an indie developer:

Android:

3 phones and 2 tablets, 2 OSes = 10 configurations

iOS:

2 phones and 2 tablets, 2 OSes = 4 configurations

You probably don’t have 14 devices on hand, and you don’t want the cost and hassle of upgrading your hardware and keeping OSes up to date.

You DO need to have several key devices on hand to perform much of your real device testing. You can cover your other real device testing needs by leveraging a mobile device cloud.

Step 3: Mobile device cloud services

Mobile device clouds are a relatively recent offering, but they are becoming a viable option for access to actual devices. They allow you to purchase access to devices, either on a pay-per-use basis or a monthly minutes subscription. Then you can access their library of devices through your browser. You can load your app or mobile site on these devices, and perform tests remotely. Many will also let you run automated test scripts.

So now you don’t have to keep on-hand a large number of mobile devices to verify your app on a critical set of real hardware and ensure a high-quality experience for your most important users.

Following are several popular mobile device cloud solutions available today:

1. AWS Device Farm
2. Perfecto Mobile
3. SeeTest
4. Xamarin
5. Saucelabs
6. BrowserStack

We are also launching our own mobile cloud solution – Kobiton. It’s currently in a beta, during which you can test your application for free. We aim to create a solution that’s intuitive to use, flexible and well integrated into the mobile developer’s workflow. We want to make it easy for you to produce great apps, and we’re looking for your feedback.

Don’t let your great idea and hard work fail because of device compatibility issues. Test your app on the devices your customer uses and ensure a great experience. Happy hunting!

About Vu Lam

Vu Lam is the Founder of Kobiton and has been a technology entrepreneur for more than 20 years. In addition to Kobiton, Vu founded KMS Technology, an Atlanta-based onshore/offshore product development and testing firm with service operations out of Vietnam, as well as QASymphony, a modern software testing platform built for agile teams. Early in his career, Vu co-founded Paragon Solutions, Inc., a US-based technology consulting company with international development centers in Bangalore, India and Ho Chi Minh City, Vietnam. His establishment and guidance of Paragon Solutions Vietnam (PSV) led to his recognition as a pioneer of the Vietnam IT industry. Following the acquisition of Paragon by First Consulting Group (FCG), a NASDAQ traded company, Vu led FCG’s Global Shared-Services group and led the growth of the organization from 400 to 1,300+ resources. Vu is also currently a Board Member of several IT startup companies.