Device Queuing for Mobile Test Automation

Mobile test automation teams often face the challenge of orchestrating test executions when device capacity runs out. A common question arises: Should the device lab platform itself queue up incoming test requests until a device is available, or should such logic be handled at the CI/CD or automation framework level?

In this post, we’ll explore the concept of device queuing, discuss why Kobiton (and other platforms) may prefer not to provide an enforced queue, how Kobiton’s Device Reservation feature offers an alternative, and show you how to implement an effective polling strategy in your CI/CD pipelines—complete with code snippets for GitLab CI.

What is Device Queuing?

Device Queuing is a mechanism by which test sessions that exceed the current device capacity are placed in a waiting line until a device becomes free. When all devices are busy:

• A platform with built-in queuing will place new test sessions in a queue.
• A platform without built-in queuing might return an error if no devices are available.

While a built-in queue sounds convenient, it can come with downsides, including hidden timeouts, unintended bottlenecks, and reduced flexibility to prioritize or reroute tests.

Downsides of Platform-Level Queuing

Although automatic queuing is useful in some scenarios, it can introduce complexities:

1. Hard Timeout Limits
   Many platforms enforcing queueing have strict timeouts (e.g., 30 minutes). If a device doesn’t free up in time, your test fails automatically—even if waiting longer would have been acceptable.
   
2. Single Choke Point
   A built-in, one-size-fits-all queue can cause high-priority tests to wait behind lower-priority ones, stalling your entire test pipeline.
   
3. Reduced Flexibility
   By letting the platform handle queuing, you lose the freedom to create custom retry intervals, dynamic rerouting to other clusters, or advanced prioritization logic.
   

Why Kobiton Encourages CI/CD-Level Queuing

Kobiton’s philosophy is to give you full control over how your tests are queued and retried:

1. Greater Control
   • You decide how many times to retry, how long to wait between retries, and whether to escalate certain test suites.
   • Your pipeline or automation framework remains the single source of truth.
2. Avoiding Arbitrary Platform Limits
   • Kobiton does not impose strict queueing timeouts or device usage constraints that don’t align with your needs.
   • You can scale your device usage as needed and only fail tests on your own terms.
3. Integration with Existing Workflows
   • CI/CD systems like Jenkins, GitLab, and GitHub Actions are already designed to handle custom logic.
   • You can monitor device availability, schedule tests, and gather analytics in one place.

Leveraging Kobiton’s Device Reservation Feature

While Kobiton does not implement a first-in, first-out queue for devices, it does provide a Device Reservation feature that allows you to reserve specific devices in advance. This can be particularly beneficial for critical or high-priority test suites that must have guaranteed access to a device at a specific time.

How Does Device Reservation Work?

• You can designate certain devices to be “reserved” for your critical tests or regression runs.
• When these tests begin, they automatically get those reserved devices without competing for availability.
• Other tests can still run on the remaining pool of devices. If everything else is occupied, those non-reserved tests can implement polling or retry logic in the CI/CD pipeline until a device frees up.

Benefits of Device Reservation

1. Guaranteed Device Access
   • Critical or time-sensitive tests won’t be blocked by other concurrent tests.
   • Minimizes the risk of failing a crucial build just because no device was free.
2. Fewer Waiting Sessions
   • By guaranteeing that certain devices are always available to specific tests, you reduce the chance of a large queue forming.
   • Non-critical tests still rely on your pipeline’s retry/poll mechanism, ensuring flexible scheduling.
3. Combines Well with CI/CD Polling
   • Even with reservation, you can still use CI/CD-level queueing for non-reserved tests.
   • This hybrid approach helps you prioritize certain runs while giving others a smart fallback.

Important Note: Reserved devices that are left idle mean fewer devices are available to the general pool. Striking a balance between reserved and non-reserved devices is key to avoiding overall device underutilization.

How to Implement CI/CD-Level Queuing or Polling

Even with Device Reservation, you may still need to handle concurrency for non-reserved devices. Below is the general approach for setting up CI/CD-level polling:

1. Check Device Availability: Use the Kobiton API to determine if at least one desired device is free.
2. Retry Mechanism: If no device is free, wait (e.g., 30 seconds) and check again. You can implement either a fixed interval or an exponential backoff.
3. Maximum Retries / Timeout: Decide how many times you want to retry or how long you’re willing to wait.
4. Proceed or Fail: If a device becomes available, move on to your test execution. If not, fail gracefully or fall back to another environment.

Note: Replace “<BASE64_ENCODED_CREDS>” with your actual Kobiton authorization header (Basic Auth token) or whatever credentials your organization uses.

5.1 GitLab CI/CD Example

stages:

  - check_devices

  - run_tests

check_devices:

  stage: check_devices

  image: alpine:3.16

  script:

    - apk add --no-cache curl jq

    - |

      max_retries=5

      wait_interval=30

      device_available=false

      for i in $(seq 1 $max_retries); do

        available_count=$(curl -s -X GET "https://api.kobiton.com/v1/devices" \

          -H "Authorization: Basic <BASE64_ENCODED_CREDS>" \

          | jq '[.devices[] | select(.isAvailable == true)] | length')

        if [ "$available_count" -gt 0 ]; then

          echo "Device is available."

          device_available=true

          break

        else

          echo "No available device. Retrying in $wait_interval seconds..."

          sleep $wait_interval

        fi

      done

      if [ "$device_available" = false ]; then

        echo "No device became available after $max_retries retries."

        exit 1

run_tests:

  stage: run_tests

  script:

    - echo "Running tests..."

    - # e.g., "mvn test"

Key Points

• We install curl and jq in an Alpine container so we can query Kobiton’s API.
• The job exits with code 1 (failing the pipeline) if no device is available within the specified retries.

Customizing Your Strategy

While these examples illustrate a straightforward polling mechanism, you can adapt them to better suit your organization:

• Exponential Backoff: Increase the sleep interval each time (e.g., 30, 60, 120 seconds) to reduce API calls.
• Filtering for Specific Devices: If you need a particular OS or device model, refine the jq filter or Kobiton API query.
• Prioritization: You can write logic to prioritize critical test suites, pushing them ahead in the queue.
• Alerting & Monitoring: Integrate Slack or email notifications when you’re consistently hitting concurrency limits.
• Device Reservation: Reserve specific devices in Kobiton for your most critical runs to avoid blocking or wait times altogether.

Conclusion

Platform-level device queuing might sound attractive, but it can bring rigid timeouts, hidden bottlenecks, and reduced flexibility. Implementing CI/CD-level queuing or polling for mobile test automation places you in full control—letting you tailor the concurrency logic to your unique workflow and keep an eye on performance bottlenecks directly in your pipelines.

• Avoid Arbitrary Timeouts
• Manage Complex Priorities
• Maintain Full Transparency & Control

By leveraging Kobiton’s API alongside features like Device Reservation and your existing CI/CD pipeline, you can seamlessly manage when and how to run tests. The result is a more optimized automation flow, reduced risk of contention for devices, and confidence that critical tests will always have a device ready when they need it. Give Kobiton a try today!