AI Testing in the AI Development Lifecycle

AWS Labs’ AI-DLC spec starts in the right place

AI can cut ambiguity before a line of software is written

AI Testing has a new problem: AI can now generate working software that is wrong.

An anonymized prospect came to Kobiton with the cost of that problem. They were a $130M ARR company with 99% of revenue coming from their mobile app. They had shifted quality checks onto developers using Maestro, then pushed straight to production. Two months later, churn was up fivefold, monthly recurring revenue (MRR) was down 8%, and the public reviews told the same story.

The issue was not that developers owned quality. Strong product teams want developers close to quality. The issue was that the company lost the runtime evidence it needed to know whether the mobile experience still worked before release.

That is why AWS Labs’ AI-DLC (AI-Driven Development Lifecycle) workflow specification is worth taking seriously. It gets the first part of the problem right: software does not begin with requirements. It begins with human intent, and human intent is almost always incomplete.

A recent anonymized Kobiton customer came to us last month with an app Apple had rejected in review. Their requirements were detailed. They were not vague one-liners like “we need driving-speed records for each driver,” but data collection, annotations, escalations, approvals, the full set of workflows. They’d started with five pages and refined them with Claude until they had twelve. The app those twelve pages described was about thirty screens.

That move from five pages to twelve, through AI-assisted iteration, is basically AWS Principle 2 in action. The framework says every business intent starts out ambiguous because “humans are naturally imprecise when expressing requirements, and natural language is inherently ambiguous,” carrying a pile of unstated assumptions. AWS argues that AI shouldn’t jump straight to solution mode. It should ask clarifying questions, gather context, and disambiguate intent before it tries to solve anything.

That’s a sound principle, and it’s how we work at Kobiton.

One of the most common failure modes in AI-generated software is the model producing plausible code for the wrong reading of the problem. The better models get at implementation, the more dangerous that gets. Bad assumptions now become working software faster. Working software that happens to be wrong.

So the first job of AI in the lifecycle is clarification. A good system asks what a strong product manager (PM), designer, or engineer would ask: What is the user trying to do? What are the expected states? The error conditions? The constraints? What does success look like, and what must never happen?

This is where AWS Labs’ AI-DLC is directionally right. It treats ambiguity as the natural starting point and makes the work about reducing that ambiguity before design and implementation begin.

But it only handles one kind of ambiguity: the kind you can find before the application exists. That matters. It also isn’t enough.

Acceptance criteria are the real foundation of quality

The most important artifact in an AI-driven lifecycle isn’t the user story, the test case, or even the generated code. It’s the acceptance criteria.

Acceptance criteria are where product intent becomes verifiable, the bridge between what the business wants and what the running app has to prove. Without them, AI Testing is just a faster version of the old problem: more automation scripts, still no clear answer on whether the product does what it’s supposed to.

Traditional QA orgs tend to put the test case at the center of everything. A requirement gets interpreted by a PM, translated by a tester, encoded into a test case, automated into a script, then maintained as the product shifts underneath it. Every translation leaks context and costs money, and over time teams are left managing test artifacts that may or may not still reflect what the product was ever meant to do.

That model was already fragile. AI makes it worse. If the point of AI Testing is to generate more test cases faster, we haven’t changed the model. We’ve just sped up the artifact factory.

The better path is to make acceptance criteria the control point. In that setup they drive design (they shape the generated interface and system behavior), they drive code (they define what developers and coding agents have to implement), and they drive verification (they’re the basis for deciding whether the running app satisfies the original intent). The test case, if there is one, drops to an implementation detail: a generated artifact whose only job is to verify a criterion. It shouldn’t be the source of truth.

This is central to how Kobiton thinks about Agentic Quality Engineering. The claim isn’t “AI writes better test cases,” which wastes most of what AI can do for quality. The point is that AI Testing should connect acceptance criteria to runtime proof. The question was never “did the tests pass?” It’s “did the running application satisfy the acceptance criteria?” That’s the line between automating QA work and actually proving product quality.

The missing layer: runtime creates new ambiguity

Some requirements can’t be found until the app runs

AWS Labs is right that ambiguity lives at the requirement stage. What the framework misses is that ambiguity also shows up at runtime.

You can’t surface every missing piece of context by asking better questions upfront. Better questions expose assumptions, sharpen requirements, and cut avoidable waste, but some context simply doesn’t exist until the app runs and gets compared against the acceptance criteria.

Take a requirement that says login failure should show a helpful message. Everyone agrees, the criteria get written, the code gets generated. On a Galaxy S22 the user sees a clean “Your credentials are invalid.” Fine. Run the same flow on an iPhone 16 Pro Max and the retry button clips half the message. That’s a defect you had no way of knowing about until runtime. Or take a primary action button that looks fine in the static design but, in the running app, doesn’t read as tappable because the color, spacing, and state treatment don’t signal that you can touch it. The requirement was technically met. The experience failed anyway.

That’s runtime ambiguity, the ambiguity that only appears when intent meets behavior. It isn’t a failure of requirements or of the product manager, and it isn’t proof the team didn’t ask enough questions. It’s just what real software does in real environments. Mobile is the worst offender: different devices, OS versions, screen sizes, gestures, permissions, networks, battery states, user states. Runtime doesn’t only validate requirements. It expands them.

This is where Kobiton’s Agentic Quality Engineering extends the AI-DLC. AWS gives you a strong framework for reducing ambiguity before implementation. We add the runtime loop that finds ambiguity after implementation and routes what it learns back into code, design, requirements, or verification.

Runtime verification isn’t the same thing as test execution

Most people hear “runtime verification” and mentally translate it to “test execution.” Before the agentic era, that was fair. It isn’t anymore.

In traditional QA, a failed test is mostly a triage problem. The automation broke, and nobody knows why yet. Product defect? Flaky script? Changed selector? Environment? Incomplete requirement? A test that no longer matches how the product is supposed to behave? One anonymized Kobiton customer runs a team of ten software development engineers in test (SDETs), out of sixty, whose entire job is figuring out the root cause of test failures. For them, pass/fail is where the work starts, not where it ends.

Agentic Quality Engineering starts from a different premise: the system shouldn’t just report that a test failed. It should gather enough runtime evidence to explain what happened, where, and why the observed behavior did or didn’t satisfy the acceptance criteria.

That matters most in mobile. Code is one part of the story. The running experience is screens, gestures, device state, OS version, permissions, network behavior, UI hierarchy, logs, visual state, real interaction. A generic coding agent can reason about source files. It can’t see whether the actual app behaved correctly on a real device under real conditions, which is exactly why AI mobile testing can’t be reduced to script generation. The signal you want isn’t “automation ran.” It’s what the app actually did.

A script can tell you a password-reset flow failed. Runtime verification can show that the app received the reset callback, went into a loading state, never rendered the confirmation screen, and never offered a recovery action. Those are very different signals. One says the test failed. The other explains the product’s behavior. And once you can explain the behavior, you can route the issue.

Because the fix isn’t always in the same place. Maybe the app code is wrong. Maybe the design is unclear, or the requirement was incomplete, or the verification step itself was generated badly, or the device exposed a condition nobody thought about. Runtime verification is the next logical step in AI Testing precisely because the goal isn’t running more tests. It’s turning observed behavior into product understanding.

AI Testing needs runtime intelligence

AI-generated software moves the bottleneck from creation to confidence

The better AI gets at generating software, the less code creation is the constraint. That doesn’t make development easy. It moves the hard part.

Teams are shipping more code, more screens, more flows, and more variations in less time. That is real productivity. But unless the organization can verify that all this generated software works as intended, speed just converts into risk.

That was the lesson from the mobile-first prospect at the opening of this post. Their problem was not a lack of output. They had plenty of output. Their problem was confidence. They could ship, but they could no longer reliably prove that the running mobile experience still worked for customers.

That is the new bottleneck AI Testing has to solve.

The best AI Testing systems will classify failure, not just detect it

Detection is table stakes. A failed check tells you there’s a problem. It doesn’t tell you what kind of problem. In modern teams, that distinction is the whole game.

Go back to the login message clipped by the retry button on one device. A traditional automation system reports that the expected text wasn’t visible. Useful, but incomplete. The team still has to work out whether it’s a layout bug specific to the iPhone 16 Pro Max, interference from something else running on the device, or something else entirely.

This is where AI Testing has to grow up. The system should be able to say: this looks like a code issue. Or: this satisfies the written acceptance criteria, but the criteria missed a real-world condition. Or: the UI element is present, but the visual design doesn’t communicate that it’s interactive. Or: the verification plan assumed a precondition the runtime session never set up.

That kind of classification is worth far more than another pass/fail report, because it changes how work moves through the org. Code issues go to engineering. Design issues go to product design. Requirement gaps go back to product management. Verification issues update the quality system. Environment problems get separated from real defects. The whole org points at improving the product instead of arguing about the test.

It’s also the practical bridge between AWS Labs’ AI-DLC and Kobiton’s Agentic Quality Engineering. AWS is right to focus on reducing ambiguity before implementation. But once the app runs, the system has to learn from what it saw, not in a hand-wavy “continuous improvement” way, but concretely: this behavior was observed, this criterion was or wasn’t met, and here’s the artifact that needs to change. That’s the loop AI Testing has been missing. The future isn’t AI that writes more tests. It’s AI that understands product intent well enough to verify runtime behavior and explain the gap.

AWS found the first ambiguity. Runtime reveals the second.

AWS Labs deserves credit for an important paper on the AI-driven lifecycle. It names the first ambiguity: intent ambiguity, the kind that shows up before implementation and lives in vague requirements, thin user stories, missing constraints, and unstated assumptions. AI helps reduce it by asking better questions and turning intent into acceptance criteria.

There’s a second one, though: runtime ambiguity. It shows up after implementation, in the gap between what the team thought it specified and what the app actually does when it runs. You only find it through execution, observation, evidence, and verification against the acceptance criteria. That’s the missing layer.

Before implementation, AI asks, “What should this mean?” After implementation, it asks, “Did the running application actually satisfy it?” Both questions matter. A requirements doc can describe intended behavior, a design can represent intended experience, code can implement intended logic, but only the running app can prove the user got the intended outcome.

So AI Testing can’t stop at generating test cases, scripts, assertions, and reports. Those are useful; they aren’t the goal. The goal is confidence that the product works as intended, and confidence doesn’t come from generation. It comes from verified behavior.

AWS is right that every business intent starts with ambiguity. The part it leaves out is that some of that ambiguity only becomes visible at runtime. The winners in AI Testing will be the platforms that handle both.