Reducing Mobile App Testing Costs by Maximizing Shared Device Access
With an abundance of floating hardware split amongs various business units, we needed a way to centralize devices in order to reduce bottlenecks and increase productivity. Kobiton did just that, making us more efficient and allowing us to build a better product for our users." Brian Baumgartner, Quality Assurance Manager
Founded in 2006, Charleston, South Carolina-based Boomtown is a fast growing, web-based software company that specializes in end-to-end marketing solutions for real estate agents. Mobile apps are a big part of Boomtown’s automated marketing tools that help customers close deals and grow their businesses. Although the company was growing quickly, Boomtown’s success led to some inefficiencies as the mobile team performed more mobile app testing.
The Challenge: Lack of Shared Device Access and High Volume Testing Stretched Boomtown’s Budget and Time
Consisting of three automation engineers and six developers who work across both Android and iOS platforms, Boomtown’s mobile team could not purchase every device they wanted to test every possible configuration. Yet, they still needed a way to test as many devices as possible to meet their users’ needs. As the volume of tests (about 1600 each month) continued to increase, the team wasted a lot of time and energy physically sharing devices—time and energy that could have been spent resolving issues.
In addition to struggling with the best way to share devices across teams, QA and Support often did not have the right device in hand. Boomtown’s limited inventory of in-house devices, while making financial sense, restricted their total test coverage potential. Boomtown had to run many of their automated tests in simulated environments to save money and time on continuous testing—even though they knew simulations weren’t as good as testing on real devices.
Boomtown knew a better way must exist for their team to get the testing they needed while working more effectively. Brian Baumgartner, Quality Assurance Manager, said, "We didn't want our team 'testing scared'—overly worried about minutes and time limitations that potentially interfered with the quality of their testing."
When Boomtown explored options other than Kobiton, they hit limitations when providers:
- Featured a plethora of real devices in the cloud to test, but with prohibitive costs.
- Did not offer internal device access or if they did, it was complicated and costly.
- Offered testing services at a low cost but with unreliable simulators and emulators.
Ultimately, Boomtown wanted to run more automated tests on real devices in the cloud while still using and easily managing the devices they already owned.
And Boomtown needed a better way to share its test devices and increase collaboration across business units.
The Solution: Provide Shared Internal Device Access and More Device Configurations at a Low Cost
Kobiton provided a solution that helped Boomtown in the two most important areas that mattered to them.
- Easy-to-Use Shared Device Access and Management
Kobiton allowed Boomtown to easily share devices across teams and locations. Testers could plug real devices into Kobiton’s platform and any authorized team members could access those devices through the cloud. By working with Kobiton, Boomtown optimized the investment they already made in their internal devices, scaled their testing capacity, and increased productivity across their teams. Plus, Boomtown can also track and collaborate on the testing activity related to those devices.
- More Device Configurations Available Without Increasing Costs
Boomtown has access to a real device library that includes more devices and operating systems that customers actually use. For testing real devices in the cloud, Boomtown benefits from unlimited users, access to all of Kobiton’s real device library, and unlimited concurrent sessions.
Results: Boomtown Saves Time and Money With Kobiton’s Customized Solution
By reducing testing costs while expanding device configuration coverage, Boomtown has maximized its mobile app testing investments through Kobiton.
- Boomtown offloaded two full 24-hour days’ worth of scheduled automation runs each month from emulators to Kobiton’s real devices, saving almost $500 every quarter.
- Boomtown decreased the frequency of hand-to-hand device sharing. By plugging all internal devices into the Kobiton cloud platform, those devices became accessible to all and tested more quickly. This sped up bug identification and fixes.
- By using the Kobiton cloud to manage their devices, Boomtown has unlimited device minutes on any of their internal devices and they can run infinite automation tests on them without any cost anxiety.
Boomtown can now handle its growth and increased number of monthly tests with ease. And the most important result? Optimized and expanded mobile app testing helps improve the apps that real estate agents rely upon to grow their businesses. By taking its own mobile app testing to the next level, Boomtown continues to help do the same for its customers.
Replika Replicates Its Real Devices Over Kobiton’s Mobile Test Cloud
I picked Kobiton over the competition because of their excellent performance, quality UI, and extremely responsive customer support. Kobiton listens to its customers and does a great job of giving us an affordable way to test on real devices without sacrificing any of the performance and features we need.” Alexey Doroshko, Replika QA
A small yet scrappy startup of 28 people, Replika’s talented engineers specialize in AI. Their innovative tool learns to mimic the preferences and personalities of its users through online conversations with that user or information the user provides (such as photos). It’s a fascinating experiment in machine learning, but the AI will also ultimately help users with mundane tasks in their life such as scheduling appointments and even encourages work-life balance with mindfulness exercises.
Challenge: Supporting 1.6 Million Device-Diverse Users with a Mobile Team of Five
Developing such a pioneering AI product means that Replika’s mobile app must meet an unusually high standard of success for more than 1.6 million registered users. Yet, Replika’s mobile team is no more than five people, including a single tester. As a small startup, that means Replika must maximize its already minimal testing resources during such an early, crucial stage of the company’s growth.
With a limited budget and only two in-house test devices, it became obvious that covering a wide variety of device configurations was impossible.
mostly operate the app on new and emerging devices (including dozens of Android platforms and hundreds of iOS platforms), Replika did not want to sink costs into owning devices that could very soon become obsolete.To solve the problem, the company needed a cost-effective way to expand its testing footprint and cover as many devices as possible—including rare configurations—as they prepared to release a new version of the app.
After examining many options in the marketplace, Kobiton emerged as a perfect fit for Replika’s needs.
- More Available Device Configurations Without Increasing Costs:
Replika now has access to a real device library that includes hundreds of the most popular device configurations for manual and automated testing—while also giving Replika the ability to share its two internal devices anytime, anywhere with its team over the cloud. Replika now has access to a real device library that includes hundreds of the most popular device configurations for manual and automated testing—while also giving Replika the ability to share its two internal devices anytime, anywhere with its team over the cloud.
- Improves Testing Efficiencies:
With a distributed team in different locations, Kobiton allows the Replika team to work more efficiently. Although there is only one tester, with Kobiton, the team can easily view testing activity and results. They can also easily share bugs that have been discovered via screenshot, video and logs, in order to resolve issues more quickly, despite being in different locations.
- Kobiton Support Serves as an Extension of Replika’s Small Testing Team:
Kobiton provided Replika access to a responsive customer support team ready to answer questions and provide help—allowing Replika’s small QA team of one tester to get more done with less resources.
Just as Replika’s AI learns and adapts based on information from its users, Kobiton’s team is constantly evolving and adding features to support Replika as they test their mobile app. Today, Kobiton works like an extra member of Replika’s team—automating important testing functions, centralizing information, and offering customer service support. As Replika evolves its AI app, Kobiton’s platform features, variety of device configurations, and support will ensure that this startup continues to rapidly grow along with its enthusiastic user base.