Out-of-Context Failures: Why a Second AI Agent Cannot Catch Them

The doubt AI was supposed to remove

A QA lead at a wealth-management firm said it plainly, in the middle of evaluating AI-assisted testing tools. Asked what was genuinely hard about automation, he answered:

Even if it passes nine out of 10 times, there’s that one time where it fails.

Then, about the promise that an AI agent would catch that tenth time on its own:

And I think the power in the agent mode is that it’ll know when it fails.

And then, immediately, before anyone could answer, he asked what happens when the agent is wrong about that too, when the pass it reports is a false positive. That question is the whole reason this piece exists. If you are evaluating agentic testing and you want to know where it stops, you are in the right place.

Here is the part worth sitting with: that doubt is not new, and it was never really about AI. A QA leader at a healthcare distribution enterprise put the older version of the same complaint this way, talking about ordinary automation rather than AI agents:

Automation can’t tell me if the screen doesn’t look right. It doesn’t know that. It’s only as smart as how I code it. It’s just following a spec. And so you always have to look at it.

The complaint did not change. It changed clothes. QA leaders have long known that automation checks only what you thought to tell it to check. Agent mode was sold as the fix, because the agent would know when it failed. The instinct of the person accountable for quality is to ask what happens when the agent is confidently wrong. Nobody has named what sits underneath that instinct, so this piece does.

Reasoning independence is not origin independence

In agentic testing, the common answer to the worry that an AI should not grade its own homework is a second AI: a separate evaluator, a different model, a fresh conversation with no shared history, an independent reviewing agent. It is a good answer to the problem it solves. A second agent genuinely catches intent mismatches and logic errors that the first agent talked itself into, because a fresh reasoner is not invested in the first one’s conclusion.

That is independence of reasoning. It is real, and this piece does not dispute it.

What it disputes is narrower. A second agent reasoning over the same artifacts the first one had shares the first one’s blind spot, because the missing evidence was never in those artifacts to reason over. When a defect’s decisive evidence has not been produced yet, a second opinion about the same absent evidence changes nothing; two agents reading the same missing fact reach the same wrong verdict with the same confidence. The gap is not that the reasoning was weak. It is that the evidence had never been produced. That is a different axis, and it needs its own name: independence of origin, not independence of reasoning.

What an out-of-context failure is

An out-of-context failure is a defect whose evidence does not exist until the app runs on real hardware. No model can produce that evidence, however large its context window or however capable it gets.

The name carries the reason it will not yield to a better model. The gap is about where the evidence comes from, not how well it is reasoned over: until a run produces the artifact, there is nothing in any context window to read, and a larger window only holds more of the same absence.

How these findings were produced

Every failure below came out of one loop. An AI coding agent dispatched test runs through Kobiton’s MCP server onto real devices, and the results came back as session video, device logs, and captured state that the model did not write. The agent could ask for a run and read what returned; it could not author the evidence, because the evidence was produced by hardware reacting to the build. That division is the whole argument in miniature, which is why the method is worth stating rather than hiding: the agent dispatches, the hardware produces the evidence, and the verdict follows from what the run actually returned. A reader can check how each finding was generated.

There is an irony in the mechanism. The agent reached those devices through the Model Context Protocol, the protocol built specifically to extend what a model can reach. The failures it surfaced are precisely the ones whose evidence no model could have produced on its own. The tool for extending reach is what exposed the limit of reach.

The taxonomy: what only a real run produces

Five categories follow. Each is a distinct kind of out-of-context failure, a defect whose decisive evidence a model cannot originate, and each carries a failure I hit firsthand. A sixth, related class sits just outside the taxonomy; I come to it after these five. The table is the short version; the sections give one case each.

Category	What the model can read	What only a real run produces
Cached artifact state	The code, keys, and the artifact it built and served	Which build the device is actually running right now
Install provenance	The binary, byte for byte	How the OS believes that binary got installed
Sensor and host dependency	The build, verified correct	The state of the specific machine that ran it
Host state drift over time	A prior passing result	What changed on that machine since
Contested input regions	The layout the app drew	Which region the OS reserves for itself, and who wins the touch

Cached artifact state

The archetype is a build that reports itself misconfigured while every artifact the agent can read says it is fine. The code is correct, the keys are loaded, the served bundle has the right values inlined and verified. The model inspects its own work and passes it. The device is quietly running an older bundle it cached days earlier, and nothing in the code, the keys, or the served artifact records which bundle is actually resident on the phone at this second. A force-stop and relaunch pulls the current one and the defect evaporates with zero code changes. The resident-build state lives on the device and expires there; no run had produced that fact, so nothing carried it.

Install provenance

Take a binary that is byte-for-byte identical across two installs. On one, in-app purchases work; on the other, they die. The bits are the same, so nothing in the binary explains the difference. What differs is how the operating system believes the app was installed, which decides how it validates store signing. A byte comparison of the artifact cannot recover a fact about the install path that the artifact does not contain. This one is not hypothetical for customers either: a vacation-ownership company opened a support case where the version the device reported as installed diverged from the version actually released, the same class of gap showing up in production rather than in my lab. For the mechanics of running these checks against real hardware, the field-log companion covers the how; here the point is that install provenance is a property of the device, not the file.

Sensor and host dependency

The same APK, disassembled and verified correct at the bytecode level first, so the code is provably not the variable. It succeeded on one Kobiton device host and threw a connection error on another host of the same model. The differentiator lived in the state of the specific machine, not in anything the build carried. No artifact records the condition of the host it happens to land on, so no amount of re-reading the build surfaces it. It took a run on the failing host to produce that fact, and a run on the passing host to prove the build was not at fault.

Host state drift over time

A host that had passed the identical build cleanly later failed it with no app change at all. A background port had moved and two services began contending for it. Nothing about the binary changed, and nothing about the earlier passing result predicted the later failure, because the variable was the machine’s state at a later moment. This is the one that kills “we validated it once.” A verdict is true for the host state the run observed, and host state is not frozen.

Both of those hosts were Kobiton’s, where I work. The fault was found, filed, and fixed. It stays in because the argument requires it: the flaw was real, it was invisible in the build, and only a run on the actual machine surfaced it.

Contested input regions

The app drew its Next and Calculate footer button on top of the system gesture-navigation bar. On screen the button was visible and clearly on top; the home pill and back chevron were drawing inside it. A tap at the button’s coordinates registered as a system home press, because the operating system reserves that strip regardless of what an app paints over it, and wins the touch. The control was present, visible, and on top, and still could not be reached. A follow-on variant appeared on a newer device on Android 16, which reported zero insets where the app expected them.

Keep the claim honest here. A model reviewing the layout code might well flag a missing bottom inset; this is not a defect no model could ever suspect. What no model could produce on its own was the manifestation and the diagnosis: that on this OS version, on this device, that region is system-reserved and swallows the touch as a home gesture. My own written diagnosis of the bug was wrong until the screenshots corrected it. The document said one thing and the picture said another, and the picture won. That is the thesis in one sentence: the record generated by reasoning disagreed with the record generated by the device, and only one of them had touched the phone.

A related failure: what nobody thought to check

The five above are failures whose evidence no run had produced yet. This last one is a different mechanism, and it is worth separating rather than smuggling into the taxonomy. In a single ordinary session, automatic accessibility scanning surfaced 87 issues across 27 screens, after both my tests and the agent’s tests had passed green: contrast failures, touch targets under the recommended size, form inputs with no label. Here the evidence did exist and a scanner produced it; I could read every finding. What failed was not origin but coverage. No assertion I or the agent had written was aimed at accessibility, so nothing flagged it. That is an attention gap, not an origin gap, and I include it because it has the same practical effect on a release owner: a green board over a real defect. You cannot write an assertion for a risk you have not thought of, and an agent inherits exactly the assertions it was given.

"Just put the device state in the context"

The strongest objection to all of this is also the most reasonable, and it deserves a straight answer rather than a dodge. It goes: fine, but screenshots, logs, video, insets, resident-build hashes, all of it can be captured and piped into the model's context. So this is a tooling gap that better plumbing closes, not a permanent boundary.

The answer is that a model can consume device evidence but cannot generate it. Every one of those artifacts exists only because something ran on real hardware first and produced them. That leaves three things you cannot pipe in:

• A fact you never captured, because nothing recorded it at the moment it was true.
• A result from a device you never ran on, because the condition lived in that host's state.
• A signal for a condition you did not know to check, because no capture was aimed at it.

Capture, execution, and attention. Feeding device evidence into a model is genuinely useful, and it is not in dispute; it is also downstream of a physical run that already happened. Hardware stays load-bearing, permanently, and the question is never whether the model is smart enough. It is whether the evidence was ever produced. That absence is what makes an out-of-context failure permanent rather than a tooling gap.

Evidence provenance: who produced the record

Here is the version of this a VP of Engineering or a CTO can act on. Audit trails are routinely computer-generated; that was never the problem. What makes a trail trustworthy is a set of controls: integrity, traceability, retention, and separation, so that the actor whose work is under review is not also the one who creates and can silently alter the records that vindicate it. An AI agent that writes the code, writes the tests, runs the tests, and files the results fails that last control by construction. The records exist, but the thing being judged produced every one of them and could revise them, so they evidence effort, not independent verification.

In a regulated setting that is not a preference, it is a signing problem. In banking, in medical devices, in gaming, someone with signing authority has to attest that a release is sound, and a record the system produced about itself is not something they can lean on. This is why the older, human version of the doubt kept a person in the loop: a quality lead at a major US airline, working under accessibility regulations with real financial penalties, described needing a remediation plan and independent proof that a fix works, and being the person who could say "I validated it" and mean it. That was never bureaucracy. It was provenance.

And the exposure grows as a share, even as it shrinks in count. As generation scales, intent mismatches and logic errors increasingly get caught by the second agent that is genuinely good at catching them, so origin failures seem likely to become a larger fraction of what escapes, even as better capture and wider device coverage keep reducing how often they happen. Teams betting their QA strategy on better models alone will find the last mile is the only mile left.

The verdict has to originate outside the model

Go back to the question the QA lead asked before anyone could answer him: what happens when the agent is wrong about the pass it just reported? His instinct was right, and now the reason has a name. Some failures are out-of-context failures, and a second agent reasoning over the same artifacts cannot help, because the evidence was never produced for either agent to reason over.

The conclusion follows without anyone having to assert it. For this class of failure, the verdict, and the evidence behind the verdict, has to originate somewhere the model cannot influence, on hardware it did not drive, in artifacts it did not author. That is not a claim about any one vendor's tooling. It is what the failures themselves require.

So take the five categories, and the coverage gap beside them, to your own escaped-defect log this week and sort last quarter's incidents into them. The ones that land in these rows are the ones a smarter agent would not have caught, and they are the ones worth building your gate around.

If you want to see where the wider practice stands, not just against these five, Kobiton's Mobile Maturity Assessment benchmarks a team's coverage, automation, and CI/CD in about twenty minutes.