The Problem with Fragmented QA Processes  

Does this scene sound familiar? Imagine it’s Friday afternoon, and a mobile QA team is frantically trying to finish testing before giving the green light to release the latest version of the app. It’s an all-hands-on-deck effort to work through regression tests. Unfortunately, bugs are being flagged at the last minute, and now there’s a growing concern about other defects leaking into production. 

The clock’s ticking closer to 5 p.m. The QA team is pushing its operational limits, and the delivery team is growing impatient. As time zips by, business stakeholders face a ‘rock and a hard place’ decision. Should the release be deployed early with more risk? Or should it be delayed again to give more time for testing and debugging? Either way, the consequences of another bug leakage or delay can lead to unhappy customers or even revenue loss. 

If you’ve ever observed or experienced this – you’re certainly not alone. This chaotic scenario is all too familiar for teams that haven’t fully embraced test automation and streamlined their CI/CD process. 

The Solution: Embracing Automated CI/CD Workflows 

Fortunately, mobile app delivery doesn’t have to be this hectic. When automated CI/CD workflows are put in place, the difference can be night and day. A process that was previously fragmented and inefficient can evolve into a high-speed system that reliably pushes new features to customers—all in a matter of hours. 

Here’s the catch. More often than not, organizations express an interest in ‘adopting a CI/CD-driven approach,’ or they want to ‘embrace DevOps’. When the time comes to define and execute a strategy, the well-intentioned effort can quickly fall off the rails, 

In this article, you’ll find practical examples of how you can utilize tools that are likely in your existing tech stack alongside mobile device clouds like Kobiton. 

• CI/CD Pipeline → Jenkins, Bitrise, Gitlab, GitHub Actions
• Functional Automation Framework → Appium, Katalon
• Performance Testing Framework → JMeter, K6, OctoPerf, NeoLoad

Once we cover the five must-have CI/CD workflows, you’ll then have a clear roadmap for enhancing your delivery process. That way, you won’t have to experience working nights & weekends to keep up with the next release. 

Let’s start with an easy square #1. 

Mobile CI/CD Workflow#1: Trigger Automated Smoke Tests After New Builds

As a delivery partner for Kobiton, we often hear that automated test suites are manually triggered every time a new build is ready. For a quick win, we recommend configuring your test suite to automatically kick-off as a post-build action. This can be accomplished with an open-source stack like Jenkins + Appium or with commercial solutions like GitLab + Katalon. 

CI/CD Workflow Implementation Steps

1. Kobition Setup: Integrate Kobiton into your CI/CD pipeline using the API library
2. Configure Tests: Write and configure Appium tests to execute against Kobiton’s real devices
3. Automate Execution: Set up your CI tool (GitLab, Jenkins, GitHub Actions, etc) to trigger these tests after each build 

Benefits 

• Increased Device & OS Coverage: Test on a variety of real devices to catch device-specific or OS-specific issues.
• Boost Operational Capacity: Reallocate time spent on manual testing into exploratory testing to catch bugs off the ‘happy path’
• Faster Feedback: Quickly identify and fix issues, accelerating the release cycle.

Mobile CI/CD Workflow#2: Schedule Nightly Automated Regression Tests 

Typically, when a new release is stuck in QA, it really just means that regression testing took much longer than anticipated. While automation is the obvious answer to this – the real key is to have these tests run on a continuous basis. We’ve had clients who would struggle to remediate defects because their testers would catch them later in the day. When they run tests overnight and can triage in the morning, it gives their developers valuable time to fix the issue and still give time for retesting. 

This, of course, makes it much easier to keep a release on schedule. 

CI/CD Workflow Implementation Steps

• Schedule Tests: Configure your CI tool to run regression tests nightly.
• Maintain Test Suite: Regularly update and maintain a comprehensive set of regression tests.
• Analyze Results: Automatically analyze and report test results to the team.

Benefits 

• Early Bug Detection: Catch and fix bugs before they reach production.
• Continuous Validation: Ensure the app remains stable with every code change.
• Efficient Use of Time: Utilize off-hours for extensive testing.

Mobile CI/CD Workflow#3: Initiate Load & Performance Tests

When functional testing is draining time and resources, it can be rather difficult to find time for executing load & performance tests. This is often the reason we see clients push load testing to the final stages of a test cycle. While this tends to be a norm, it’s certainly not ideal. Instead, we also recommend embedding your load/performance tests into the CI/CD pipeline. 

This is a best practice for any software project but is especially important for mobile projects. With the extra dependencies on 3rd party hardware, APIs, and operating systems, mobile tends to pose far more complexity than a simple web app. Without proper performance testing, apps can suffer from slow response times and crash while under significant load. 

By keeping load testing in the CI/CD workflow, teams can ensure they test earlier and more often.

CI/CD Workflow Implementation Steps

1. Set Up Tools: Configure K6 or JMeter for API performance testing and integrate Kobiton for functional testing.
2. Define Scenarios: Create performance test scenarios that simulate various load conditions on your API layers.
3. Parallel Execution: Set up your CI/CD pipeline to run API load tests and Kobiton functional tests concurrently.
4. Analyze Results: Monitor and analyze both performance metrics and functional test outcomes to identify any issues.

Benefits

• Optimal Performance: Ensure the app can handle expected loads on the API layer.
• Functional Integrity: Detect functional issues caused by performance problems.
• Comprehensive Test Coverage: Provide a holistic view of the app’s performance and functionality under load.

Mobile CI/CD Workflow#4: Parallel Test Execution Across Multiple Device Configurations

Let’s entertain another example. Imagine there are 2 teams, each dedicated to mobile test automation for iOS and Android. Pretend both have 1,000 test cases in their full automated regression suite. We’ll also assume each test takes 2 minutes to complete. 

Now, one team has 1 device available for testing sequentially. The other has 20 devices available for testing in parallel. What difference would this make in practice? 

	Team 1	Team 2
Total Tests	1,000	1,000
Average Test Duration	2 Minutes	2 Minutes
Total Devices	1	20
Tests per Device	1,000	50
Total Execution Time (minutes)	2,000	100
Total Execution Time (hours)	33.33	1.67 (20x efficiency)

While this is a hypothetical example, it underscores the importance of having both test automation and real devices embedded into the CI/CD pipeline. Team 2 is able to complete the same deliverables in 5% of the time while ensuring coverage is established for a variety of device + OS configurations. Not only will they finish their cycle a day early, it also means their colleagues on the development side will have an entire extra day to remediate any new defects. Or, their colleagues on the delivery team may be able to release a day earlier.

It’s a win for all involved. 

CI/CD Workflow Implementation Steps

1. Setup Parallel Execution: Configure your CI tool (e.g., Jenkins, GitHub Actions) to support parallel test execution.
2. Distribute Tests: Divide your test suite into smaller batches that can be executed concurrently.
3. Monitor Results: Aggregate and analyze results from parallel test runs to ensure comprehensive coverage.

Benefits

• Reduced Feedback Time: Get test results faster, enabling quicker iterations and more agile development cycles.
• Enhanced Efficiency: Utilize testing infrastructure more effectively by running tests concurrently.
• Accelerated Development: Speed up the overall development cycle, allowing for more frequent releases and faster time-to-market.

Mobile CI/CD Workflow#5: ChatOps Integrations & Defect Triage

A proper discussion on improved testing efficiency would be incomplete without reviewing the topic of defect triage. At a tactical level, it would be impractical to trigger a bug ticket to be sent to Jira or ADO Boards every single time an automated test failed. It’s important to ensure human intervention stands between 

To make this more realistic, we recommend Kobiton customers leverage their CI tool to trigger a notification in a dedicated channel in Slack or Teams. After tests run overnight, you’ll be able to review bug notifications in the morning. With the click of a link, it’s easy to refer back to a recording of the test in the Sessions area of the Kobiton Portal. Then a tester can decide if triaging a bug to development makes sense. 

CI/CD Workflow Implementation Steps

1. Configure Webhooks: Set up webhooks to send CI/CD notifications to your chosen communication tool.
2. Customize Alerts: Define specific alerts for different stages of the CI/CD pipeline (e.g., build success, test failures).
3. Automate Notifications: Ensure notifications are automatically sent to relevant team channels, providing instant updates.

Benefits

• Improved Collaboration: Keep the entire team informed and aligned, enhancing teamwork and communication.
• Faster Response: Quickly address issues as they arise, reducing downtime and improving overall efficiency.
• Enhanced Visibility: Provide real-time insights into the CI/CD process, ensuring transparency and better decision-making.

Wrap-Up

Implementing these five essential CI/CD workflows can transform your mobile app delivery process, making it faster, more efficient, and more reliable. By automating and streamlining testing and deployment, you can ensure high-quality releases and avoid the chaos of last-minute bug fixes and delayed launches.

Need Help Setting Up CI/CD Workflows? 

If you’re interested in learning more about best practices or need support setting up your mobile CI/CD workflows, use this link to book an initial strategy session with one of Perform’s architects. Check out the replay of Zach’s MTES session “Elevating Mobile App Quality: 5 Essential Steps for Optimal App Performance” to learn more.