Mar 22, 2026
Agentic Quality Engineering Ends Mobile Device-in-Hand Testing
Frank Moyer
Read More
Why Agentic Quality Engineering Breaks Without Real Devices
Reading Time : 22 min read
The shift nobody’s talking about
Agentic Quality Engineering is the next major shift in software development, but Agentic Quality Engineering Breaks Without Real Devices when AI-driven workflows lack access to the environments where users actually interact with applications. AI agents are writing code, generating tests, and running development workflows with minimal human involvement.
The conversation around it is focused on the wrong thing. The analysts are paying attention. Gartner predicts that 40% of enterprise applications will feature task-specific AI agents by the end of 2026, up from less than 5% in 2025. Forrester renamed its testing platform category to Autonomous Testing Platforms in 2025. Most of the attention, though, is on what agents can produce.
Everyone wants to know about test generation. How fast can an AI write a test case? How many scripts can it produce in an hour? How well does it understand the requirements?
Fair questions. But they skip the part that actually matters.
Where do those tests run?
In 1998, Motorola launched Iridium, a $5 billion satellite constellation that promised phone calls from anywhere on Earth. The technology was real. Sixty-six satellites in low Earth orbit, all working as designed. But the phones were bricks. They didn’t work indoors. The ground infrastructure for seamless handoffs barely existed. Motorola built the network in the sky and treated the place where humans actually hold phones as an afterthought. Nine months after launch, Iridium filed for bankruptcy.
The agentic testing industry is headed down the same path. Impressive capabilities at the top of the stack, a gap at the bottom where tests actually need to run. For mobile applications, that gap breaks the whole thing.
What agentic quality engineering actually means
The full lifecycle, not just generation
Agentic Quality Engineering is not AI-powered test generation. It’s a closed-loop system where AI agents participate across the development lifecycle. They analyze requirements. Generate test cases. Write automation scripts. Execute those scripts against real environments. Interpret the results. Then iterate: modify the tests, rerun, and validate fixes.
Each step feeds the next. The agent learns from execution results to improve future test generation and adjusts automation scripts based on what actually happened when they ran. This is where Agentic Quality Engineering Breaks Without Real Devices, because AI agents need reliable feedback from real-world device conditions to validate results and continuously improve their testing decisions.
It helps to see where agentic QE sits next to what came before it.
| Traditional automation | AI-assisted testing | Agentic quality engineering | |
| Who writes the tests | Engineers write every script | AI suggests, engineers approve | Agents generate from requirements |
| Who decides what runs, and where | Engineers, via pipeline config | Engineers, with AI recommendations | Agents, within rules humans set |
| How failures are handled | Engineers investigate and fix | AI suggests a fix | Agents classify, fix or file, then rerun |
| What it learns from | Nothing; scripts don’t change themselves | Mostly the codebase and past scripts | Execution results, every cycle |
| How much it depends on the execution environment | Moderately; a bad environment means flaky scripts | Moderately | Completely; the environment is the agent’s only view of reality |
That last row is the one the rest of this article is about.
Why the loop only works with real environments
Take out execution against real environments and the loop doesn’t collapse. It just never completes. The agent still generates tests. It still produces automation scripts. Those scripts might even run somewhere. But without real-world execution data, the machine never learns from real-world scenarios. The automation gets brittle because it can’t account for what actually happens in the field: poor network conditions, unexpected application crashes, Apple’s permission popups disrupting test execution mid-flow.
The agent optimizes for a controlled environment that doesn’t represent your users. Every iteration takes it further from reality.
The missing layer in Agentic Quality Engineering
The stack everyone’s building
The agentic testing stack looks roughly like this:
Most platforms invest heavily at the top. AI coding agents are getting better fast. Code generation improves month over month. Test generation is the feature that gets the stage at every product launch.
Same pattern Motorola followed. They built better satellites and a more sophisticated network. The part of the system that actually touches the user, the handset, the ground infrastructure, indoor coverage, was someone else’s problem.
Where execution infrastructure disappears
Look at the bottom of the stack. Execution infrastructure and validation environments are treated as assumed, as if they’ll just be there when you need them.
For web applications, that assumption mostly holds. Browsers are standardized. Selenium Grid can be set up in a day. Headless Chrome runs anywhere.
For mobile? Not close.
What breaks when agents can’t reach real environments
When AI agents can’t access real execution infrastructure, a few things go wrong.
Tests pass in synthetic environments and fail in production, giving you false confidence. Agents generate tests but never learn whether those tests reflect reality, so the feedback loop stays open. Each iteration builds on the last, and if the first execution didn’t reflect real conditions, every adjustment after that drifts further from what users experience. Meanwhile the agents burn compute generating, modifying, and regenerating tests that validate nothing.
The architecture has a hole in the middle, and everything on top of it is unstable.
Why mobile is the hardest problem for Agentic Quality Engineering
Device fragmentation at scale
Mobile testing is a different engineering problem than web testing.
Start with fragmentation. Thousands of active Android device models. Multiple iOS versions across different hardware generations. Screen sizes, pixel densities, chipsets, memory configurations all vary, and all affect behavior.
Fragmentation isn’t only hardware. It also breaks the way an agent reads the screen.
In a browser, the agent has a DOM: a stable, standardised map of the page. On mobile, it works from the platform’s accessibility and view hierarchy, and that hierarchy isn’t the same everywhere. A few places where it shifts under the agent:
- OEM skins rework system UI. The permission dialog, the share sheet and the notification shade look and nest differently on a Samsung, a Pixel and a Xiaomi running the same Android version.
- OS updates change the tree. An element the agent located reliably last quarter can sit at a different depth, or carry different attributes, after an update.
- Hybrid apps switch surfaces mid-flow. A checkout that moves from native screens into a WebView needs the agent to handle two different automation contexts in one journey.
- Screen size changes the path. A button visible on a large phone needs a scroll on a small one. A test that never scrolled passes on one and fails on the other.
- System interruptions arrive uninvited: permission prompts, low-battery warnings, OEM ‘optimise this app’ dialogs.
This matters more for agents than for scripts, because agents self-heal. An agent that learns its locators on a stock emulator image will ‘repair’ tests toward a UI most of your users never see. Every heal makes the suite more confident and less accurate.
Then add the things that don’t exist in a browser: sensors, GPS, Bluetooth, NFC, cameras, biometric hardware, carrier networks. Each one depends on physical hardware.
What emulators can’t reproduce
Simulators and emulators are useful for development. They’re fast to spin up, easy to automate, and they let developers iterate quickly.
They can’t fully represent real-world conditions.
An emulator won’t show you the thermal throttling on a three-year-old Samsung running in direct sunlight. It won’t reproduce the Bluetooth pairing failure that only happens on specific chipsets. It won’t simulate push notification differences between OEM Android skins.
For mobile users, none of that is an edge case. It’s a Tuesday.
The agent perspective
A human tester knows the gap between emulator and real device. Experienced QA engineers know which tests need real hardware and which are safe on simulators.
AI agents don’t have that judgment yet. They’ll run every test wherever you point them. Point them at emulators and they’ll report passing results for scenarios that would fail on real devices. With complete confidence.
Think of it as a different kind of hallucination. The agent isn’t fabricating an answer like a language model might. It’s reporting exactly what happened. The problem is that what happened on the emulator has nothing to do with what your users will experience.
Giving agents the judgment they don’t have yet
If agents can’t tell which tests need real hardware, you have to tell them. Not test by test. As rules they apply on every run.
The point isn’t to push everything onto real devices. Emulators are fast and cheap, and an agent iterating on a fix should use them. The point is that an emulator result should never be allowed to stand in for a real-device result on a path where the hardware matters. Five rules cover most of it:
1. Tag every test that touches hardware-dependent behaviour: notifications, camera, biometrics, Bluetooth, NFC, GPS, sensors, network transitions, background and battery behaviour. Tagged tests always get a real-device run before a release decision.
2. Treat an emulator pass on a tagged test as unverified, not passed. The agent can report it, but it doesn’t count toward release readiness.
3. Once a test has failed on a real device and passed on an emulator, it never goes back to emulator-only. That divergence is the most valuable thing the agent has learned about it.
4. Route by pipeline stage, not just by test type. Fast feedback stays virtual; decisions get made on hardware.
5. When a new device or OS version enters your user base, promote it into the real-device matrix before the agent starts generating tests against it.
| Pipeline stage | Where the agent runs | What the run is for |
| Agent iterating on a change | Emulators and simulators | Fast functional feedback while generating and fixing tests |
| Pull request or merge | Emulators, plus a small real-device smoke set on your top devices | Catch obvious device-specific breakage before it lands |
| Nightly or scheduled | Real-device matrix for all hardware-tagged tests | Build the real-world execution data the agent learns from |
| Release candidate | Full real-device matrix | The results the release decision actually rests on |
An experienced QA engineer applies these rules without writing them down. An agent needs them written down. That’s the difference between an agent with access to real devices and an agent that actually uses them.
Why real device infrastructure is so hard to build
Device exhaust at millisecond latency
Here’s what most people outside mobile infrastructure don’t understand: running a test on a real device isn’t the hard part. Capturing everything that happens during that test is.
Real device infrastructure has to collect device exhaust in real time. Screenshots, view hierarchies, system logs, video, CPU usage, memory consumption, network activity, battery drain. All at millisecond latency. While the device runs automated scripts from an AI agent that expects immediate feedback.
This isn’t a configuration problem. It’s a systems engineering problem that takes years to solve.
All those data streams, all at once
Each stream has its own capture mechanism, timing constraints, and failure modes. Screenshots need to sync with test steps. View trees need to reflect the actual rendered state, not a cached version. Video has to be continuous without dropped frames. System metrics need to correlate precisely with test events.
Now do that across thousands of devices simultaneously, running different OS versions, on different hardware, over network connections with variable latency.
Why companies walk away after years of trying
I remember a partnership conversation with test.im back in 2018, before Tricentis acquired them. Their rep told me the engineering team was building out mobile testing capabilities. That was eight years ago. Today, neither test.im nor Tricentis has real device infrastructure.
They’re not alone. A real device cloud takes specialized hardware management, custom firmware integration, device provisioning systems, and the kind of low-level systems engineering most software companies aren’t set up to do.
Browser infrastructure takes a day. Mobile takes a decade.
I could stand up browser testing infrastructure with Selenium Grid in a day. Spin up containers, configure nodes, connect browsers, run tests. Done.
Mobile real device infrastructure is a decade-long engineering investment. There’s no container that holds a physical phone. There’s no npm package for managing a rack of devices with thermal management, USB hubs, and carrier SIM cards.
Iridium learned this. The satellites were the impressive part, the thing investors and press pointed to. But the ground segment, the handset engineering, the infrastructure that connects the network to a human hand? That took decades of telecom experience. Motorola thought they could skip it.
The execution infrastructure layer isn’t something you bolt on later. For mobile, the number of organizations that have actually built it and run it at scale is very small.
Why real devices are essential for AI test automation
What only exists on physical hardware
AI agents will generate tests for every feature your application supports, including the ones that only work on real hardware.
Push notification delivery and display behavior vary by device and OS version. Camera-dependent features like AR and barcode scanning need actual camera hardware. Biometric authentication, Face ID, fingerprint sensors, iris scanning, can’t be meaningfully simulated. GPU differences cause visual discrepancies you’ll only catch on a real screen. Network transitions between WiFi and cellular, dead zones, carrier-specific quirks, all depend on real radio hardware.
For any consumer mobile app, those are core user journeys, not corner cases.
When test generation meets reality
An AI agent generates a test for push notification handling and runs it on an emulator. Clean result. The notification arrives instantly, displays perfectly, dismisses cleanly.
On a real device, that same test might reveal the notification gets delayed by OEM battery optimization. Or it doesn’t render correctly at a specific screen resolution. Or the deep link fails because of a manufacturer’s custom intent handling.
The agent needs those messy, real results to get better. Without them, it optimizes for a world that doesn’t exist.
What a real-device result gives the agent back
A failed test on its own tells the agent almost nothing. Did the app break? Did the test break? Did the device have a bad moment? An agent that can’t answer that question either files noise as bugs or ‘heals’ a test that was correctly catching a defect.
That’s what the device exhaust is for. Screenshots, view hierarchy, logs and system metrics, captured in sync with each step, are what let the agent classify a failure instead of guessing at it.
| What the agent sees | What it most likely means | What the agent should do |
| Fails on one OEM or OS version, passes everywhere else | Device-specific defect | File a bug with the device, OS version and artifacts attached. Keep that device in the matrix. |
| Fails at the same step on every device, and the element is missing from the view hierarchy | The app changed, or the test is out of date | Compare against the change. Update the test only if the change was intended; otherwise file a bug. |
| Fails intermittently on the same device across reruns | Environment or timing: network, thermal state, a system dialog | Correlate with network, CPU and thermal data before touching the test. Don’t self-heal a flake. |
| Passes, but CPU, memory or battery drain spikes against the previous build | Performance regression | Flag it even though the functional check passed. |
| Crash or ANR in the system log | Application stability defect | File immediately, with the log excerpt and the video leading up to it. |
There’s a second payoff. Once results are tied to real devices, you can separate the coverage you designed from the coverage you proved. A hardware-dependent test that exists but has only ever passed on an emulator is designed coverage. It counts as proven only when it has passed on a real device. Reporting the first number as if it were the second is exactly the false confidence agentic testing is prone to.
Continuous testing requires continuous access
Agentic quality engineering is not a one-shot process. Agents run continuously. They generate, execute, learn, and iterate around the clock.
That means continuous access to real devices. Not a shared lab available during business hours. Not a manual checkout process. A real device cloud with API-level and MCP access, on-demand provisioning, and capacity for parallel execution at scale.
Continuous doesn’t mean wasteful. Two kinds of cost add up in an agentic pipeline: device time and agent reasoning. Both are manageable.
Agents shouldn’t re-reason a test every time it runs. Once a test is authored and has passed on real hardware, it can replay as a deterministic script, with the agent stepping back in only when something fails or the app changes. And not every run needs the full matrix. Routing by pipeline stage keeps real-device capacity for the runs that inform a decision, while emulators absorb the iteration.
What you can’t economise on is availability. When the agent needs a real device, it needs one now, not when the lab opens.
How good the agent is depends entirely on how good its execution environment is.
The future of agentic quality engineering
The architecture that will actually work
The future isn’t smarter AI. It’s smarter infrastructure.
The systems that deliver on AI-driven testing will need four things working together: AI agents for reasoning and test design, automation frameworks that turn AI decisions into executable scripts, execution infrastructure that can run tests reliably at volume, and real validation environments (real device clouds included) that ground the AI’s feedback in what actually happens on a user’s phone.
Iridium eventually survived. After bankruptcy, restructuring, and a completely different business model focused on maritime, aviation, and military users who actually needed satellite coverage. The technology worked. It only became viable when someone built the right infrastructure around it and pointed it at the right problem.
Agentic quality engineering will go the same way. The AI is real. The capabilities keep improving. But the winners won’t be the platforms with the best test generation. They’ll be the ones that connect generation to execution environments that reflect the real world.
Real device cloud as infrastructure, not accessory
Agentic workflows won’t replace testing infrastructure. They’ll depend on it more than human-driven workflows ever did. Humans compensate for missing infrastructure with judgment and manual effort. AI agents can’t do that.
A real device cloud is load-bearing in this architecture. Take it out and the system produces unreliable results.
Where humans still belong in the loop
Real devices don’t remove humans from agentic QE. They move them.
Developers are already cautious about AI output. In Stack Overflow’s 2025 Developer Survey, 84% of respondents said they use or plan to use AI tools, yet more distrusted their accuracy (46%) than trusted it (33%), and only 3% said they highly trust it. Agentic testing doesn’t get a pass on that scepticism. It has to earn trust with evidence.
In a working setup, humans stop executing tests and start owning the decisions the agent can’t make for itself:
- What to test. Approving which requirements and user journeys the agent generates tests for, so it doesn’t invent scope.
- Where to test. Owning the real-device matrix and the routing rules, because both depend on knowing who your users are.
- What a failure means. Reviewing failures that only appear on real hardware, since those are where the agent’s model of the app is weakest.
- Whether to ship. Making the release call on proven coverage, with the device evidence in front of them.
The agent does the volume. The human does the judgment. Real devices are what make the agent’s evidence worth judging.
Is your agentic testing pipeline real-device ready?
If you’re putting agents into your mobile testing workflow, check these before you trust what they report.
1. List your hardware-dependent user journeys. Notifications, payments with biometrics, camera, location, Bluetooth, anything that crosses a network transition. These are the tests that can’t be settled on an emulator.
2. Build the device matrix from your own analytics. Cover the devices and OS versions your users actually run, plus your lowest-spec supported device.
3. Give agents programmatic access to real devices. API and MCP access, on-demand provisioning, enough parallel capacity for continuous runs. Not a shared lab or a checkout queue.
4. Write down the routing rules. Which tests run where, at which pipeline stage, and which results count toward a release.
5. Capture evidence on every run. Screenshots, view hierarchy, logs, video and system metrics, attached to each result so the agent and your team can triage it.
6. Report proven coverage separately from designed coverage, and gate releases on the proven number.
7. Review real-device-only failures regularly. Each one is a rule you haven’t written yet.
If you can’t tick the third item, the rest don’t matter much. The agent will produce results either way. They just won’t describe your users’ phones.
Conclusion
Agentic Quality Engineering is a real evolution in how software gets built and tested. But the conversation is lopsided. Too much focus on generation, not enough on execution. Agentic Quality Engineering Breaks Without Real Devices because AI-driven testing still needs to validate how software performs in real-world conditions.
For mobile, execution means real devices. Not instead of emulators and simulators, but on top of them. Emulators are where developers iterate. Real devices are where software gets validated. Both matter. Only one tells you what your users will actually experience.
The organizations that invest in execution infrastructure alongside AI capabilities will ship software people trust. Everyone else will have sixty-six satellites in orbit and no one able to make a call.
Frequently asked questions
What is agentic quality engineering?
Agentic quality engineering is a closed-loop approach to testing where AI agents analyse requirements, generate tests, write automation, execute it, interpret the results and iterate, with humans setting goals and making release decisions. What separates it from AI test generation is the loop: the agent learns from execution results, so the quality of the execution environment directly determines the quality of the agent.
Can AI testing agents use emulators instead of real devices?
For fast feedback while an agent iterates, yes. Emulators and simulators are quick and cheap. For validation, no. They can’t reproduce real notification delivery, biometric hardware, thermal throttling, carrier networks or OEM-specific UI, so an emulator pass on those paths says little about what users will experience. Most teams use both: emulators for iteration, real devices for the results that decide a release.
How is agentic QE different from AI test generation?
AI test generation produces test cases or scripts. Agentic QE also runs them, classifies failures, fixes or files them and reruns, feeding what it learned into the next cycle. Generation is one step in the loop; the loop is what makes it agentic.
How do AI agents get access to real devices?
Through a real device cloud that exposes provisioning, test execution and artifacts programmatically, typically by API, and increasingly through MCP so coding agents can call device tools directly. The access needs to be on demand and parallel, because agents test continuously rather than in scheduled windows.
Which mobile tests should always run on real devices?
Anything that depends on physical hardware or device-specific software: push notifications, camera and AR, biometric authentication, Bluetooth and NFC, GPS accuracy, network transitions, battery and thermal behaviour, and UI rendering across OEM skins. Treat emulator passes on these as unverified.
Does agentic testing remove the need for human testers?
No. It shifts their role. Humans approve what gets tested, own the device matrix and routing rules, review failures that only appear on real hardware, and make the release decision. The agent handles volume; people handle judgment.
