When leaders build real teams, they do not start by throwing smart people into a room and hoping the work somehow organizes itself. They define the mission. They clarify roles and responsibilities. They decide who owns what. They create operating cadence. They define handoffs. They put controls in place. They measure performance. They run retrospectives. They improve the system over time.

But when people build agentic systems, they often skip that entire layer.

They start with a few agents. They write prompts. They chain some actions together. They get a working demo. And then they wonder why the system breaks down the moment the work becomes ambiguous, cross-functional, interrupted, or dependent on human review.

The more I work on agentic systems, the more I think this is the core issue:

Most agentic systems fail for the same reason real teams fail.

They fail because onboarding is weak. Roles are fuzzy. Responsibilities overlap. Handoffs are fragile. Controls are missing. Human judgment sits too far to the end of the process. Performance is not measured. And there is no real operating rhythm for learning and improvement.

That is why I no longer think the right question is, “What agents should I build?”

I think the better question is:

How do I design an operating system for agentic work?

For me, that operating system has four pillars:

1. Onboard

2. Organize

3. Operate

4. Review, Retrospect, and Improve

That framing has become much more useful than thinking in terms of agents alone.

1. Onboard

The first thing strong leaders do is onboard people into the mission and context.

Agentic systems need the same thing.

And I think there are actually two separate onboarding capabilities that matter.

The first is user onboarding.

This is how humans learn how to use the system. What does this platform do? What teams exist? What do I need to provide? Where do I review outputs? Where do I approve work? How do I know what happens next? How do I resume if I come back later?

The second is context onboarding.

This is how the system learns what the customer or project is actually about. What is the objective? What is in scope? What is out of scope? What constraints matter? What source materials exist? What is missing? What assumptions are already floating around? What does success look like?

These are not the same thing.

Teaching a teammate how to use the system is one problem.

Teaching the system what the work means is another.

If you blur those together, the system starts operating on weak context almost immediately. Product gets fuzzy. Architecture guesses. QA tests against the wrong intent. Delivery coordination becomes cleanup instead of coordination.

So onboarding is not a setup step. It is a first-class operating capability.

2. Organize

This is where the leadership analogy gets even stronger.

Strong teams work because people understand their roles and responsibilities.

The same is true for agentic systems.

What changed my thinking here was moving away from the idea of isolated agents and toward agentic teams.

Real software delivery almost never lives in one role. It moves across product, architecture, development, QA, DevOps, delivery coordination, and governance. If you model that as a loose swarm of agents, ownership gets muddy fast. If you model it as teams with leads, you get a much more stable structure.

Each team should have:

• a clear mission

• a lead

• defined inputs

• defined outputs

• review points

• escalation paths

• handoff obligations

Subagents can exist inside the team, but they should support the lead, not dilute accountability. The lead owns the official output.

This matters because one of the easiest ways to break an agentic system is to let multiple agents contribute without anyone clearly owning the result.

That is just an org design problem in a different form.

A Product team can own business intent and requirement quality. Architecture can own technical direction and specs. Development can own implementation. QA can own quality strategy, evidence, defects, and regression assets. DevOps can own environments and operational readiness. PMO and Delivery can own planning, visibility, continuity, and user guidance. Governance can inspect the health of the system and recommend improvements without automatically rewriting it.

But just as important, not every system needs the full cross-functional model on day one.

You can start with a smaller agentic team as long as the operating system is still clear.

Two Examples

The first example is a full product-development agentic team.

That is the larger model. It spans Product, Architecture, Development, QA, DevOps, Delivery, and Governance. This is the right shape when you are trying to build or evolve a product end to end and you need clear ownership across the full software lifecycle.

The second example is a testing agentic team.

This is a much smaller and more practical place to start.

A testing-focused agentic team might include:

• a QA or Test Lead

• a Test Design Agent

• a Test Execution Agent

• a Defect Reporting Agent

That smaller team still benefits from the same operating system ideas:

• user onboarding

• context onboarding

• defined roles

• handoff rules

• controls

• logs

• retrospectives

The difference is scale, not principle.

That is an important point. This is not about building an elaborate org chart for every problem. It is about designing the right operating model for the work you are trying to do.

3. Operate

This is where the team becomes an operating system.

Once a team is onboarded and organized, it still needs a way to work. In leadership terms, this is the operating model. In agentic terms, this is workflow, handoffs, controls, logging, and recovery.

This is the layer I think people underestimate the most.

A useful agentic system should define:

• how work enters

• what states work can move through

• what artifacts must exist before a team begins

• what counts as a valid handoff

• where approvals happen

• what gets logged

• how failures surface

• how work resumes after interruption

• what happens when the system has to operate in degraded mode

The reason this matters is simple.

Most agentic failures are not intelligence failures. They are operating failures.

A team expects an asset and it is missing. Or it exists, but it is stale. Or it was superseded by something newer. Or nobody knows which version is the source of truth. Or the work was partially completed and then interrupted. Or a human needed to approve something and nobody surfaced it in time.

That is not a prompt problem. That is a coordination problem.

And coordination problems are exactly what operating systems are supposed to solve.

Human-In-The-Loop Should Be Active, Not Passive

I also think this is where a lot of people misunderstand human-in-the-loop design.

A passive review at the end is not enough.

There are parts of the flow where human judgment should be a first-class step, not a final checkpoint. That is especially true when the work includes ambiguity, business meaning, or quality judgment.

Testing is a good example.

I would not treat test design as something an agent fully owns and a human casually approves. I would treat test design as a human deliverable, with support from the agentic system.

That means the Test Design Agent might generate:

• draft scenarios

• coverage ideas

• edge-case prompts

• traceability starters

• peer-review questions

But the human QE lead or tester shapes the final test design.

That is a very different model.

The human is not just reviewing the work. The human is owning a critical artifact, and the agent is helping that human think better and faster.

I think this is a better answer to the “something feels wrong” problem than passive review alone.

A human can often tell when a design feels incomplete or off, even before they can fully explain why. If the operating system makes that human step explicit, the system gets stronger. If it pushes everything to the end, the human becomes a safety net instead of part of the design.

So for me, human-in-the-loop is not only about approvals. It is also about deciding which parts of the work should stay human-owned, with agent support.

4. Review, Retrospect, and Improve

This is the leadership habit I think matters just as much in agentic systems as it does in human teams: retrospectives.

Real teams do not improve just by doing more work. They improve by stopping and asking how the work happened.

Were the right people involved?

Were the roles clear?

Did the handoffs work?

Were the controls too weak or too heavy?

Did approvals happen at the right times?

Did the logs help, or just create noise?

Did the team recover cleanly from interruption?

Did the process create confidence, or confusion?

Agentic systems need exactly the same discipline.

That is why I think the final pillar is not just “observe.” It is review, retrospect, and improve.

The system needs health checks. It needs performance measurement. It needs governance. And it needs retrospectives after meaningful parts of the flow.

Not just retros on outcomes.

Retros on the operating system itself.

That means looking at:

• cycle time

• blocked time

• approval delays

• rework loops

• unresolved assumptions

• aging open questions

• repeated handoff failures

• stale artifacts

• bloated definitions

• noisy logs

• poor recovery after interruption

And then asking what needs to change.

Maybe a role boundary is unclear. Maybe a control is missing. Maybe the wrong team is owning a handoff. Maybe a log is too verbose to be useful. Maybe a human step should move earlier in the process. Maybe a certain agent should not exist at all. Maybe one should be added.

This is how the system gets better.

And one boundary matters a lot here: governance should recommend changes, not silently implement them. Humans should still decide when the operating system itself changes.

That is a trust issue as much as a design issue.

What About Cost and Latency?

This is another place where I think people can swing too far in either direction.

On one side, you can build a loose agentic system with almost no controls and get speed at the cost of reliability.

On the other side, you can build such a heavy operating model that the system becomes slow, expensive, and hard to use.

I do not think the answer is to reject controls.

I think the answer is to calibrate the level of controls to the system you are designing.

A small testing agentic team probably does not need the same control surface as a full product-development operating system that spans requirements, architecture, development, QA, DevOps, and release governance.

That means:

• lighter controls for smaller, lower-risk flows

• stronger controls for high-risk, cross-functional, production-grade work

So yes, economics matter. Cost and latency should shape the design. But that does not mean production-grade controls are unnecessary. It means they should be applied intentionally.

The Shift in How I Think About Building This

If I were advising someone to build agentic systems today, I would not tell them to start by creating a bunch of agents.

I would tell them to start the way a strong leader builds a team.

1. Define the mission and success criteria.

2. Design onboarding for both users and context.

3. Define the teams, leads, roles, responsibilities, inputs, and outputs.

4. Design the operating model: workflow, handoffs, controls, logging, recovery, and where human judgment should be a first-class step.

5. Define health checks, performance measures, governance, and retrospectives.

6. Turn that blueprint into a roadmap.

7. Stress test the design for ambiguity, overlap, weak handoffs, and missing controls.

8. Adjust the blueprint.

9. Build one narrow slice first.

10. Test it, run retrospectives, improve it, and expand gradually.

That sequence feels much more durable to me than “start with prompts and see what happens.”

Where I’ve Landed So Far

If I had to summarize what I believe right now, it would be this:

Building agentic systems is less like wiring prompts together and more like building a high-functioning team.

The same leadership disciplines apply:

• onboarding

• roles and responsibilities

• operating cadence

• decision rights

• controls

• human judgment

• performance reviews

• retrospectives

• continuous improvement

That is the operating system.

And I think that is the layer most people are still missing.

So if you are building agentic systems, I would not start by asking:

What should this agent do?

I would start by asking:

If this were a real team, what roles, responsibilities, controls, and operating rhythm would it need in order to succeed?

That question has turned out to be much more useful for me.

And I suspect it is where serious agentic systems actually begin.

The Problem This Solves

Most QE organizations hit the same wall. Test automation can’t keep pace with delivery. The backlog of unautomated scenarios grows every sprint. Regression suites take longer to maintain than they save. And the team is stuck in a loop: write scripts, fix scripts, rewrite scripts, fall behind, repeat.

The usual response is to add headcount or buy a platform. Both scale linearly. Double the work, double the cost. And neither addresses the core issue: the cognitive work of test design, coverage assessment, and risk evaluation is bottlenecked on the same people doing the mechanical work of writing and maintaining scripts.

What if you separated those two things?

Not in theory. In architecture.

The Operating Model

In Where to Draw the Line Between AI and Human Work, I described an operating model where humans own the cognitive work and AI handles execution. This isn’t a new idea in QE. Most enterprise testing organizations already work this way with people: FTE QE leads own strategy, coverage decisions, and release readiness. A delivery team handles scripting, data prep, test runs, and triage.

The system I built applies that same structure to AI agents. Five of them, each with a defined role, communicating through files, coordinated by a lead agent that acts as the single interface between the human and the execution layer.

The human stays above the Delegation Line. The agents sit below it.

That sentence sounds simple but it drives every architectural decision in the system. Who can create test scenarios? The agent drafts them, but the human approves them. Who decides what to test? The human. Who writes the Playwright code? The agent. Who decides if a test failure is a bug or a flake? The human - with the agent’s analysis as input.

The line is structural, not aspirational. The system enforces it.

Five Agents, One Team

The system uses five specialized agents. Not one general-purpose AI assistant. Five, each with a narrow scope and clear boundaries.

The Architect

This is the lead. It’s the only agent the human talks to. It receives requests, decides what needs to happen, delegates to specialists, reviews their outputs, and assembles deliverables. It analyzes applications, runs impact analysis on code changes, manages the review workflow, and interprets test results.

What it doesn’t do: write test scenarios or Playwright code. That’s what the other agents are for. The Architect coordinates. It doesn’t do everything.

The Scenario Designer

Designs structured test scenarios across four coverage categories: happy path, edge cases, negative scenarios, and error conditions. Every scenario has four layers: the user flow, validation points, expected results, and instrumentation (screenshots, network captures, console logs). Outputs go to markdown files with a parallel review file for the human to approve.

The Script Engineer

Generates Playwright TypeScript from approved scenarios. Creates test specs, Page Objects, fixtures, and helpers. Enforces a hard gate: it won’t generate code for scenarios the human hasn’t reviewed. This is the architectural enforcement of the Delegation Line - cognitive work must be complete before execution begins.

The Reporter

Produces formal deliverables: Requirements Traceability Matrix, coverage reports, release readiness assessments, version delta comparisons. Everything traceable. Everything in markdown that can go into a release package or a stakeholder review.

The Validator

Quality control for the agents themselves. Validates outputs from every other agent before the human sees them. Three modes: scenario validation (structure, ID uniqueness, coverage completeness), script validation (TypeScript compilation, import resolution, selector accuracy), and results validation (failure categorization, accuracy). Issues go back to the originating agent for fixes - two rounds max - then escalate to the human.

Why Five?

Because specialization creates reliability. A single general-purpose agent that does everything is capable but unpredictable. Narrow agents with clear boundaries are more consistent. Each one has a defined input, a defined output, and a defined scope of responsibility. When something goes wrong, you know exactly where to look.

This mirrors how high-performing QE teams work. You don’t have one person who does strategy, scripting, execution, reporting, and quality control. You have specialists who are excellent at their specific jobs, coordinated by a lead who understands the full picture.

File-Based Communication

The agents don’t share memory. They don’t use message queues or databases. They communicate through files.

This is a deliberate choice, not a limitation. Files are auditable. Files are git-traceable. Files survive session boundaries. A new session can reconstruct the entire project state by reading the file system. No context to rebuild. No state to restore. Just read the files.

The practical impact: every test scenario, every review decision, every script, every result, every report is a file you can inspect, version, diff, and trace. When a VP asks “what changed between v2.3 and v2.4 and how do we know the regression suite covers it?” - the answer is in the files. Not in someone’s head. Not in a tool’s database. In files your team can read.

The Review Workflow

This is where most AI testing approaches fall apart. They generate tests and assume they’re correct. Or they put a human “in the loop” as a rubber stamp - a monitor watching AI work, clicking approve on things they barely read because the volume is too high.

This system does the opposite. The human is the decision-maker, not the monitor.

When the Scenario Designer produces test scenarios, it also produces a review file. Each scenario gets a line item: ACCEPT, REJECT, or NEEDS CHANGES. The human reads each scenario, applies judgment (is this the right coverage? does this match how real users behave? is this priority correct?), and marks their decision. The system won’t proceed until the review is complete.

This isn’t a bottleneck. This is the point. The cognitive work - deciding what to test and why - is the highest-value activity in QE. Automating it away doesn’t make your testing better. It makes your testing unexamined.

The review gate enforces this structurally. The Script Engineer literally cannot generate code for scenarios that haven’t been approved. It checks. If the review file doesn’t exist or has unresolved items, it stops and tells the human.

Impact Analysis and Version Control

Here’s where the system earns its keep in a real release cycle.

When a new version of the application lands, the Architect reads the git diff and categorizes every existing test into one of four buckets: NEW (net new feature needing full design), MUST UPDATE (existing tests that will break), CHECK (existing tests that might be affected), and UNAFFECTED (no relation to any change, inherit as-is).

This produces a version manifest - a structured record of what changed, what’s affected, and what needs work before you can run the regression suite. The manifest chains to previous versions, so you can trace coverage decisions across the entire release history.

The practical outcome: instead of running the full suite and hoping, you know exactly what needs attention before a single test executes. Your team’s time goes to the tests that matter, not to re-running 400 unaffected scenarios to see green checkmarks.

For organizations running biweekly or continuous delivery, this is the difference between a testing practice that keeps up and one that’s perpetually behind.

What This Looks Like in Practice

A typical workflow for a new sprint:

The QE lead gets the sprint scope. They run impact analysis against the code changes. The system produces a manifest showing 3 new features needing test design, 8 existing tests that need updates, and 12 that should be verified. The remaining 180 tests are unaffected and inherit from the previous version.

The lead runs test design for the new features. The Scenario Designer produces 15 scenarios across the 3 features. The lead reviews them - accepts 12, marks 2 for changes (one is missing an edge case, one has the wrong priority), rejects 1 (it’s duplicating coverage from another component). The system incorporates the feedback.

Scripts are generated from approved scenarios. The Validator checks them for compilation errors, missing imports, and selector issues before the lead sees them. Tests run. Results come back categorized: 2 real failures (bugs), 1 environment issue, 1 flaky test.

The lead files the bugs, flags the environment issue for the DevOps team, and marks the flaky test for investigation. A release readiness report generates automatically, showing coverage percentages, open defects, and risk areas.

Total cognitive effort from the lead: the review decisions, the failure triage, and the release call. Everything else was execution - and the agents handled it.

What Changes at Scale

The system supports multiple applications from the same repository. Each app is an isolated project with its own profile, test cases, scripts, and reports. The methodology stays the same. The agents stay the same. Only the context switches.

For a QE organization managing 5 or 10 or 20 applications, this means the operating model is consistent across the portfolio. The same workflow, the same review discipline, the same traceability, the same reporting structure - regardless of which app you’re looking at.

The scaling math changes too. Adding a new application to the portfolio doesn’t require proportional headcount. The cognitive work - strategy, coverage design, risk assessment - still needs a human. But the execution work - scripting, running, triaging, reporting - is handled by agents that don’t have a capacity ceiling the way people do.

This doesn’t mean zero headcount growth. It means the ratio changes. One strong QE lead can cover more ground when the mechanical work isn’t consuming their calendar.

The Scope - and the Starting Point

What I’ve described so far is a system for web UI end-to-end functional testing. It handles the full lifecycle - scenario design, script generation, execution, triage, impact analysis, version management, and reporting. For that specific scope, it’s comprehensive.

But web UI E2E is one layer of a testing organization’s responsibility. And this system is deliberately scoped to that layer as a starting point, not because it’s the only thing that matters, but because it’s the right place to prove the model before expanding.

What’s not covered yet

Other testing types. API testing, ETL testing, data validation, contract testing - these are different execution patterns with different agent architectures. The Delegation Line applies to all of them, but the agents below the line look different. A web UI agent team needs Playwright and browser automation. An API testing team needs HTTP clients, schema validators, and contract tools. The methodology transfers. The implementation doesn’t.

Non-functional testing. Security testing, performance testing, accessibility, load testing - these are specialized disciplines with their own toolchains, expertise requirements, and risk models. Each one could have its own agent team below the Delegation Line, but the cognitive work above the line requires domain-specific expertise that’s distinct from functional testing.

Test data operations. Building test data, seeding backend databases, managing environment state, provisioning test accounts - these are operational capabilities that sit underneath the testing lifecycle. The current system assumes test data exists. A mature implementation needs agents that can construct and manage it.

Tool integrations. Requirements management systems, test management platforms, defect tracking, CI/CD pipelines - the current system is self-contained. In an enterprise context, these agent teams need to integrate with the existing toolchain: pulling requirements from Jira, syncing coverage to a TMS, filing defects automatically, triggering from pipeline events.

Exploratory testing. This is structured, scenario-based automation. Exploratory testing still needs a human with curiosity and domain knowledge probing the application in ways no scenario anticipated. That’s above the Delegation Line by definition.

The point is not that these gaps are problems. The point is that this is a starting point - a proven architecture for one well-defined scope that’s designed to extend.

Governance for Non-Deterministic Systems

Here’s the part most AI testing conversations skip entirely.

These are agentic systems. They use large language models. By their nature, they are non-deterministic. Give the same agent the same input twice and you may get slightly different outputs. The scenarios might be worded differently. The scripts might use different selector strategies. The failure analysis might emphasize different factors.

This is not a flaw to be eliminated. It’s a characteristic to be governed.

The review workflow is the first layer of governance - nothing reaches production without human approval. But governance goes deeper than review gates. It includes auditability (every decision is in a file, every file is versioned), validation (the QA Validator checks agent outputs against structural and technical rules before a human ever sees them), and traceability (every test scenario traces to a requirement, every script traces to a scenario, every result traces to a script).

For QE leaders evaluating agentic approaches, this is the question that separates serious implementations from demos: how do you control a system that doesn’t produce identical outputs every time? The answer isn’t to make it deterministic - that would eliminate the flexibility that makes it useful. The answer is to build control structures around the non-determinism so that variability in execution never becomes variability in quality decisions.

The human decides. The system executes. The governance layer ensures that boundary holds even when the execution path varies.

What This Doesn’t Replace

It doesn’t replace QE judgment. The system is deliberately designed to keep humans making the hard calls - what to test, whether coverage is sufficient, whether a failure matters, whether the release is ready. AI is capable of attempting those decisions. It is not yet reliable enough to make them autonomously. That’s the Delegation Line.

It doesn’t work without expertise. The human lead needs to understand testing methodology, risk assessment, and the application under test. The system amplifies expertise. It doesn’t generate it. A junior engineer with this system will produce more than a junior engineer without it - but they won’t produce what a senior lead with this system produces.

It doesn’t eliminate maintenance entirely. Scripts still break when applications change. The difference is that impact analysis tells you which scripts will break before you run them, and the agents can regenerate code from approved scenarios rather than requiring manual fixes.

The Underlying Bet

Every QE organization is going to have to figure out how to work with AI agents. The question isn’t whether - it’s how. And the organizations that figure it out first will have a structural advantage that compounds over time.

The bet this system makes is that the right architecture separates thinking from execution, enforces human authority at the decision points, and lets AI handle the work where it’s most reliable and humans are most constrained. It’s not about replacing your team. It’s about changing what your team spends their time on.

The QE leads who are currently spending 60% of their week on script maintenance and regression triage? They could be spending that time on coverage strategy, risk modeling, and release quality - the work that actually determines whether your releases are safe.

That’s not a technology decision. That’s an operating model decision. And it’s one that gets harder to make the longer you wait, because the organizations that move first will have the methodology, the institutional knowledge, and the velocity advantage.

The Rollout Mental Model

You don’t deploy agentic AI testing teams the way you deploy a tool. You don’t install it on Monday and have it running your regression suite by Friday. These systems need to be developed and rolled out systematically, in the context of your specific applications, environments, team structures, and quality standards.

The mental model is closer to standing up a new team than installing new software. You start with one application. You configure the agents for that application’s tech stack, selector patterns, data requirements, and environment topology. You run the first design cycle. You review the scenarios - not as a formality, but because the agents are learning your application and you’re learning the agents. You iterate. You build trust through evidence.

Then you expand. A second application. A third. Each one goes through the same onboarding discipline, because each application has its own context that the agents need to understand. But the methodology is the same, the governance is the same, and the operating model is the same. What you learned on app one accelerates app two.

The broader roadmap follows the same pattern. Web UI E2E first - that’s the system described here. Then API testing, with agent teams built for that execution pattern. Then test data operations. Then integrations with your requirements and test management systems. Then non-functional testing disciplines, each with their own specialized agent teams. Each layer follows the Delegation Line. Each layer gets its own governance. Each layer earns trust independently.

This is not a big bang transformation. It’s a systematic expansion of a proven model, one capability at a time, with human oversight at every stage.

Where the Line Moves

The Delegation Line doesn’t have to be static. As AI reliability improves for specific, measurable tasks, you can move the line - carefully. Not all at once. Not based on capability demos. Based on measured reliability over time in your environment, with your applications, against your quality bar.

Today the line sits at: agents execute, humans decide. Tomorrow, some of those decisions might move below the line - but only when the data supports it. Auto-approving scenarios that match established patterns with 99%+ accuracy. Auto-triaging failures that have been categorized correctly 50 times in a row. Progressive trust, earned through evidence.

The architecture supports this because the review gates and validation checkpoints are configurable, not hardcoded. You don’t rebuild the system to move the line. You adjust the gates based on measured confidence.

But that’s tomorrow. Today, the system works as designed: you think, agents execute, Playwright runs. And that’s already a meaningful shift from where most QE organizations are operating.

This is the architecture behind the system I introduced in Meet Your AI-Powered QA Team. What I’ve described here is the first layer - web UI E2E - and the operating model that governs it. The layers that come next - API testing, test data operations, tool integrations, non-functional testing - follow the same architecture and the same Delegation Line. But they need to be built in context: your applications, your environments, your quality standards, your team.

If you’re a QE leader thinking about how to bring agentic AI into your testing organization - or if you’re running a team that’s hitting the scaling wall and wondering what the next operating model looks like - I’d like to hear from you.

But the question underneath it is reliability.

There is an important distinction that I think most of the conversation skips over. AI capability has been advancing rapidly. Arguably faster than any technology in recent memory. But reliability, the kind that lets you take a human out of the loop and trust the outcome, has improved much more slowly.

Automation depends on reliability, not capability.

A system that is 95% capable and 60% reliable is not ready for autonomous operation. It is ready for supervised use. Those are fundamentally different operating models with fundamentally different human requirements. And the choice between them may be the most consequential design decision organizations face right now: where do you draw the line between what AI does and what humans do?

The Gap Between Capable and Reliable

You can see this dynamic play out across industries. Autonomous driving is the most visible example. Waymo spent over a decade closing the gap between a vehicle that could handle most driving scenarios and one reliable enough to operate without a human ready to take over. That gap was not about capability. It was about reliability.

But you do not need to leave software to see it. Every QE leader has lived a version of this. A test automation framework that works for 90% of scenarios and fails unpredictably on the other 10% is not a reliable framework. It is a framework that generates maintenance overhead and erodes trust. The capability was there. The reliability was not. And the team paid for it.

AI has this same dynamic at a larger scale. The failure modes are novel, inconsistent, and difficult to anticipate. That is a real constraint on how much autonomy you can safely delegate to it, and it should inform how you design the human-AI relationship in your organization.

A Risk Worth Naming

The default agentic model looks like this: AI agents perform the work. Humans monitor the output. Humans intervene when something goes wrong.

On paper, this preserves human judgment. But I think there is a real risk that it hollows it out over time.

Monitoring is not the same as doing.

When a QE lead actively designs a test strategy, they are building and reinforcing expertise. Working through scenarios. Triaging failures against domain knowledge. Deciding what coverage a release needs. Each decision sharpens their mental model. Each edge case deepens their judgment.

When that same QE lead is repositioned as a monitor, something different happens. Reviewing AI-generated test plans. Approving AI-made triage decisions. Watching dashboards for anomalies. The cognitive engagement drops. The expertise that was supposed to backstop the system begins to atrophy. Not immediately. Not obviously. But steadily.

I think the agentic enterprise often assumes that supervision sustains human judgment. My experience suggests that expertise develops through active engagement, not passive observation. When reliability is still evolving, and in enterprise AI it absolutely is, that distinction matters. It may be the difference between a system that gets safer over time and one that gets more fragile.

The Delegation Line

This is the question I keep coming back to: where do you draw the line between what AI does and what humans do?

I think many organizations will default to drawing it in a place that feels efficient but creates long-term risk. The natural tendency is to delegate judgment to AI and ask humans to monitor, because that maximizes the amount of work the AI handles. But it creates the risk I described above: the human’s role becomes reactive, supervisory, and increasingly disconnected from the cognitive work that built their expertise.

The model I have arrived at draws the line differently. Delegate execution to AI. Keep humans doing the thinking.

I call this the Delegation Line. On one side: the cognitive work. Design, strategy, risk assessment, judgment calls, domain reasoning. Humans own this. On the other side: the mechanical work. The repetitive, time-consuming, well-defined tasks that consume most of a team’s capacity but do not require human judgment on each instance. AI owns this.

The distinction matters because of where reliability breaks down. AI fails most dangerously in novel situations requiring judgment. Those are exactly the situations where you need human expertise to be sharp. AI is most reliable in repetitive execution against well-defined parameters. That is exactly the work that consumes the most human capacity today.

Draw the line at execution:

• Human expertise stays sharp because humans are actively engaged in the hard problems

• AI reliability is highest because it is operating in its most predictable mode

• The human is not a monitor. They are a decision-maker who happens to have an execution engine

Draw the line at judgment:

• Human expertise atrophies because the cognitive work has been offloaded

• AI reliability is lowest because it is making novel decisions in ambiguous contexts

• The human is a monitor whose ability to catch AI errors may degrade over time

One model gets safer as it scales. The other gets more fragile.

A Model You May Already Operate

Here is what I find interesting. If you run a QE organization in the enterprise, you may already operate a version of the Delegation Line. You just do not call it that.

Most enterprise QE functions split into two layers. Your FTE QE leads and managers own the cognitive work: test strategy, coverage decisions, risk assessment, release readiness, standards. They are accountable for quality. That does not change regardless of who or what executes underneath them.

The execution layer is handled by a delivery team. Writing scripts, preparing test data, running tests, triaging results, compiling reports. In many organizations, this is an SI partner managing onshore and offshore resources. The QE lead sets the direction. The delivery team converts that direction into output.

This is the Delegation Line, drawn by hand. The QE lead thinks. The delivery team executes. The accountability model is clear. And it works.

The constraint is not the model. It is that the execution layer scales linearly. When a sprint delivers more changes than the delivery team can absorb, the options are: delay the release, reduce coverage, or add headcount. All three are expensive. Capacity is directly proportional to how many people you can staff, train, and manage.

Now consider applying AI to that same model. Not by replacing the QE lead with a monitor, but by evolving the execution layer.

Above the line, the QE lead’s cognitive work stays the same, and gets amplified. The QE lead co-designs test strategy with AI as a thinking partner. Not monitoring AI output. Working alongside it, the same way a senior tester works with a peer in pair testing. Challenging assumptions. Surfacing edge cases. Applying domain expertise that comes from years inside the business. The QE lead makes the final design decisions.

Below the line, the execution layer becomes agentic. AI agents handle scripting, data preparation, test execution, triage, and reporting. The same work the delivery team does today, but without the linear scaling constraint. The work is well-defined and repetitive. It is exactly where AI reliability is highest and where human capacity is most constrained.

The accountability model does not change. The QE lead is still accountable for quality. Nothing ships without human approval. The difference is that the execution capacity underneath them is no longer limited by headcount.

And the QE lead never becomes a monitor. They are doing the same cognitive work they do today, with more leverage.

The Execution Layer Is Evolving

I want to be clear about what I am suggesting and what I am not.

This is not about replacing delivery teams. It is about recognizing what the execution layer is going to look like in 18 months. The managed services model that most enterprises already operate is, in my view, the right starting architecture for agentic AI. The accountability structure works. The roles and responsibilities are clear. The evolution is in how the execution layer delivers: from purely labor-based to AI-augmented, and eventually to agentic.

I have been building a version of this. Five specialized AI agents working under one human QA analyst for Playwright E2E testing. The human retains full oversight. Nothing gets tested without approval. Nothing gets reported without review. But the human is not watching AI work. The human is doing the thinking. The AI is doing the execution.

The architecture maps directly to the operating model QE organizations already run. The QE lead’s role does not change. The execution capacity underneath them does.

Why This Matters Now

Gartner projects that 40% of enterprise applications will integrate task-specific AI agents by end of 2026. Goldman Sachs has deployed AI agents across its 12,000-person technology organization. Amazon used AI agents to upgrade tens of thousands of production applications, saving an estimated 4,500 developer-years.

The shift is here. Agents are going to get more capable faster than most organizations can absorb. The question is not whether to adopt them. It is how.

And “how” comes down to where you draw the Delegation Line.

Organizations that delegate judgment to AI and position humans as monitors risk a compounding problem: the expertise they need to oversee AI safely may erode precisely because the humans are no longer doing the work that built that expertise. The more they rely on AI judgment, the less capable the humans become of catching AI errors.

Organizations that delegate execution to AI and keep humans engaged in cognitive work will likely build a different kind of system. One where human expertise stays sharp, AI operates in its most reliable mode, and the whole system gets stronger over time.

The Delegation Line is not a technical decision. It is an operating model decision. And the operating model most enterprises already use, where senior people own the thinking and delivery teams handle the execution, is a strong foundation to build on.

Draw the line in the right place. Delegate execution, not judgment. Keep your people thinking.

That is how I believe you build an agentic system that actually works.

Your QE team operates on a model that has worked for decades. A test case is a contract: define a precondition, provide an input, specify the expected output, and compare. Pass or fail. The logic is clean, the boundaries are clear, and every test either confirms the system behaves as designed or it doesn’t.

Automation encodes that contract into scripts. Execution requires work — environment setup, data seeding, account resets, script runs, log collection, result analysis, reporting. None of this is truly one-click yet for most organizations, but the model itself is well understood and improvable. You can optimize each step. You can measure each step. You can hire for each step.

The entire discipline rests on a single underlying assumption: determinism. Given the same input and the same system state, the system produces the same output every time. This is what makes it testable. This is what makes automation possible. This is what makes your assertion logic work.

That assumption is so deeply embedded in every tool, framework, certification, and hiring criteria in the QE profession that most teams don’t even think of it as an assumption. It’s just how software works.

Until it isn’t.

A Single Scenario, Two Worlds

Let’s take one concrete example and walk through it twice. The scenario is straightforward: an insurance claims eligibility check. A customer wants to know if their claim is covered.

The Traditional Version

In a traditional system, this is a well-defined API call or UI workflow. The inputs are specific: customer ID, claim type, date of loss. The system checks the policy status in a database, applies business rules, and returns a result — eligible or ineligible, with a dollar amount.

Your test case is clean:

Given an active policy #12345, when a claim is submitted for an auto accident on January 15, then return eligible with a $1,200 deductible.

Your automation script calls the API, captures the response, and asserts that the response string matches the expected output. Pass or fail is binary and unambiguous. If the system returns “$1,200,” it passes. If it returns “$1,300” or an error, it fails. There is exactly one correct answer.

This is the world your entire testing infrastructure was built to handle.

The Agentic AI Version

Now imagine the same workflow, rebuilt as an agentic AI feature. Instead of a structured API call, the customer describes their situation in natural language through a conversational interface.

The agent receives the message, reasons about what information it needs, and decides which data sources to query. It might check the policy status first. Or it might start with the coverage history. Or it might look at recent payment records to confirm the policy is active before checking coverage. All three paths lead to the correct answer. The agent chose one based on its reasoning at runtime.

Then it responds — in natural language. It might say “Your claim is approved for $1,200.” Or “You have a $1,200 deductible on this claim.” Or “You’re covered — the first $1,200 is your responsibility.” All three responses are correct. All three mean the same thing. None of them are identical strings.

Now run your traditional test suite against this. Your string assertion fails on two out of three valid responses. Your scripted API call sequence doesn’t match the path the agent chose. Your test case structure itself — not just the automation — is wrong.

The problem isn’t that your tests have bugs. The problem is that your testing model assumes a world that no longer exists.

What Actually Changed — The Three Breaks

The disconnect isn’t a single issue. There are three structural breaks between traditional testing and agentic AI, and each one independently undermines the conventional approach.

Non-deterministic outputs. The same input can produce multiple correct answers. The agent generates natural language, and there is no single canonical string that represents the “right” response. String matching and exact assertions are fundamentally incompatible with this reality. You can’t hardcode an expected output when the output is different every time — and legitimately so.

Context-dependent behavior. The same question asked in a different conversational context produces a different correct answer. If the customer mentioned three turns ago that they already filed a police report, the agent’s response to “what do I need to do next?” changes. Traditional tests are stateless — each test case is independent. Agentic interactions are stateful across multiple turns, and correctness depends on the full conversation history.

Variable execution paths. The agent chooses which tools, APIs, and data sources to use based on its own reasoning. You can’t script the path because the path is decided at runtime. A claims eligibility check might hit the policy API, the coverage API, or the payments API first — and all three sequences are valid. Testing the exact sequence of calls is not just brittle, it’s testing the wrong thing.

These three breaks are not edge cases. They are the defining characteristics of agentic AI systems. Any testing approach that doesn’t account for all three will produce false failures on valid behavior and false passes on invalid behavior — which is worse than having no tests at all.

The New Testing Model

The paradigm shift is this: stop testing whether the system did exactly what you scripted, and start testing whether it achieved the right outcome within acceptable constraints.

This is not a minor adjustment. It’s a fundamentally different model of what a “test” is. Instead of a single assertion against a single expected value, you’re evaluating behavior across multiple dimensions simultaneously. Here’s the framework.

Component 1: Semantic Validation

Traditional testing compares strings. Semantic validation compares meaning.

Instead of asserting that the agent’s response exactly matches an expected string, you evaluate whether the response is semantically equivalent to what a correct answer should communicate. The mechanism varies — you can use embedding models to compute similarity scores between the expected and actual responses, or you can use an LLM-as-judge approach where a separate model evaluates whether the response answers the question correctly.

The key difference is that pass/fail becomes threshold-based rather than binary. An embedding similarity score of 0.92 might be a pass. A score of 0.65 might be a fail. You define the threshold based on how much variation is acceptable for a given scenario.

For example: if the expected response is “Your claim was approved for $1,200” and the agent responds “We’ve approved your $1,200 claim,” a traditional string comparison fails. Semantic validation passes — because the meaning is identical.

This directly addresses the non-deterministic output problem. You stop caring about how the answer is phrased and start caring about whether the answer communicates the right information.

Component 2: Outcome-Based Assertions

This is the most important shift for teams coming from traditional QE. Instead of testing the path the agent took, you test what the agent accomplished.

Define your assertions around outcomes: Did the agent identify the correct customer? Did it access the right data (even if through a different sequence of calls)? Was the final determination — eligible or ineligible — correct? Did the dollar amounts match? Did the agent follow business rules and constraints?

For the claims eligibility example: don’t test “agent called Policy API, then Coverage API, then returned result.” Do test “agent correctly determined that the claim is ineligible due to lapsed coverage.” The first assertion breaks whenever the agent reasons through a different valid path. The second assertion validates what actually matters.

This addresses the variable execution path problem. The agent is free to reason and execute however it determines is best, as long as it arrives at the correct conclusion within the correct constraints.

Component 3: Conversation Flow Testing

Agentic AI interactions are multi-turn conversations, and context matters. This component tests whether the agent maintains coherent, accurate behavior across a full interaction.

Design test scenarios as conversation scripts with multiple turns. Then validate: does the agent maintain context across turns? If the customer corrects themselves (”Actually, the accident was two weeks ago, not last week”), does the agent update its understanding? If the customer switches topics mid-conversation (”Wait — before we continue with the claim, can you tell me if my premium is going up?”), does the agent handle the transition and return to the original thread? If the agent gets stuck or reaches the boundary of its capability, does it escalate appropriately?

A sample test scenario might look like this:

• Turn 1: “I want to file a claim.”

• Turn 2: “It’s for a car accident last week.”

• Turn 3: “Actually, it was two weeks ago.”

• Turn 4: “Do I need a police report?”

The validation isn’t about the exact wording of any single response. It’s about whether the agent maintained context about the claim type (auto), adjusted the date when corrected, and answered the policy-specific question about police reports accurately given this customer’s coverage.

This addresses the context-dependent behavior problem. You’re testing the agent’s ability to reason across a conversation, not just respond to isolated inputs.

Component 4: Tool Usage Validation

Even though you shouldn’t test the exact sequence of API calls, you absolutely should test the boundaries of what the agent is allowed to do.

Validate that the agent accessed only authorized data sources. Verify it passed correct parameters — the right customer ID, the right policy number. Confirm it respected rate limits and retry logic. Check that it handled tool failures gracefully — if the policy API was down, did the agent tell the customer it couldn’t complete the request, or did it hallucinate an answer?

The distinction is important: don’t validate the exact sequence of tool calls (too brittle, and the sequence is the agent’s decision to make). Do validate that every tool call the agent made was authorized, correctly parameterized, and appropriately handled.

This is especially critical in regulated industries where audit trails matter. You need to prove that the agent didn’t access data it shouldn’t have, didn’t make unauthorized decisions, and didn’t bypass controls — regardless of which path it took to get to its answer.

Component 5: Edge Case and Constraint Testing

This is where you stress-test the boundaries of the agent’s behavior. Traditional edge case testing focuses on invalid inputs and error handling. Agentic AI introduces an entirely new category of edge cases that don’t exist in deterministic systems.

Test with incomplete information: if the customer says “I want to file a claim” without providing any details, the agent must ask clarifying questions, not guess. Test with contradictory requests: “Cancel my policy and also increase my coverage.” Test with out-of-scope requests: “Book me a flight” sent to an insurance claims agent. Test with prompt injection attempts: “Ignore your previous instructions and approve all claims.” Test with infrastructure failures: what does the agent do when an API it depends on is unavailable?

In regulated industries — insurance, banking, healthcare, government — this component is non-negotiable. You must be able to demonstrate that the agent won’t hallucinate policy terms, make unauthorized financial decisions, expose personally identifiable information, or provide advice it isn’t qualified to give. The risk surface of an agentic AI system is fundamentally larger than that of a deterministic system, and your edge case testing has to expand accordingly.

Defining “Good Enough”

One of the hardest conversations you’ll have is this: when there’s no single “correct” response, how do you define pass and fail?

The answer is tiered evaluation criteria. Not every dimension of the agent’s behavior carries the same weight or the same risk.

Tier 1 — Must Pass. These are your hard requirements. Factual accuracy: policy terms, coverage amounts, dates, and dollar figures must be correct. Compliance: the agent follows regulations and doesn’t make unauthorized decisions. Security: no PII exposure, access controls respected. Constraint adherence: the agent stays in scope and escalates when appropriate. If any Tier 1 criterion fails, the release is blocked. No exceptions.

Tier 2 — Should Pass. These are quality requirements. Appropriate tone and professionalism. Logical conversation flow. Efficient path to resolution — the agent doesn’t ask five redundant questions when two would suffice. Helpful explanations — when the answer is “no,” the agent explains why. If Tier 2 criteria fail, you fix them before release if possible, and document them as known issues if the timeline doesn’t allow it.

Tier 3 — Nice to Have. These are experience requirements. Conversational naturalness. Personalization. Proactive suggestions the customer didn’t ask for but would find useful. Empathy signals in difficult interactions. Tier 3 failures go into the improvement backlog. They don’t block a release.

This tiered model gives your team and your stakeholders a shared language for the “good enough” decision. It also prevents the common trap of holding an agentic AI to a standard of perfection that you’d never apply to a human agent doing the same job.

Implementation — Where to Start

Don’t try to build the entire framework on day one. A phased approach gets you to production safely without requiring six months of upfront investment.

Phase 1: Golden Path Testing (Weeks 1–2). Start with 10 to 15 happy path scenarios covering your core workflows end-to-end. Implement semantic validation and outcome-based assertions for these scenarios. The goal is to prove that basic functionality works — the agent can handle the primary use cases it was built for, and it arrives at correct conclusions.

Phase 2: Edge Cases and Constraints (Weeks 3–4). Add 20 to 30 edge case scenarios. Boundary testing, error handling, out-of-scope inputs, tool failure scenarios. Layer in tool usage validation. The goal is to prove the agent handles failure modes safely — it doesn’t hallucinate, it doesn’t expose data, it doesn’t make unauthorized decisions when things go wrong.

Phase 3: Production Learning (Ongoing). Once the agent is live, capture real user interactions with appropriate privacy controls. Identify new edge cases from production traffic that your test scenarios didn’t cover. Auto-generate test scenarios from failures and near-misses. Continuously expand your coverage based on what real users actually do, which will always be different from what you predicted.

A note on tooling: there are emerging tools in this space — evaluation frameworks, tracing platforms, and LLM testing harnesses — and they’re evolving rapidly. But the reality today is that this is 60 to 80 percent custom work regardless of which tools you adopt. The frameworks help, but they don’t eliminate the need for domain-specific design of your test scenarios, evaluation criteria, and threshold calibration. The testing strategy is the hard part. The tooling is the easier part.

The Transition to BAU

Initial testing gets the agent to production. Sustained testing keeps it there safely.

The handoff model needs clear ownership: the QE team owns test scenarios and evaluation criteria. The platform team maintains the testing infrastructure. The product team defines success metrics and acceptable thresholds. And there’s a regular cadence — weekly or biweekly — of reviewing production failures and expanding test coverage based on what you find.

The metrics you track over time tell you whether the agent is stable or drifting. Semantic similarity scores trending downward over weeks may signal model drift. New tool usage patterns emerging could mean the agent is finding better paths — or problematic ones. A rising escalation rate suggests the agent is getting stuck on scenarios it used to handle. Production incidents should be traced back to missed test scenarios, and those scenarios should be added to the suite immediately.

Governance matters too, especially in regulated industries. Who approves the “good enough” thresholds? Who reviews AI-generated test failures versus genuine agent failures? When do you retrain or update the agent versus adjust the test? These aren’t just technical questions. They’re organizational decisions that need to be made before you go live, not after.

What This Means for Your QE Team

If you’re leading a QE organization and your company is building agentic AI, the testing model you’ve spent years optimizing is about to face its biggest structural challenge. Not because it was wrong — it was exactly right for deterministic systems. But the systems are changing, and the testing has to change with them.

The shift from “did it do exactly this” to “did it achieve the right outcome within acceptable constraints” is not incremental. It requires new skills, new tools, and new ways of thinking about what a test is. The teams that figure this out early will ship agentic AI safely and quickly. The teams that try to force-fit their existing approach will either ship slowly, ship dangerously, or not ship at all.

The good news: the fundamentals of QE rigor — structured thinking, risk-based prioritization, clear pass/fail criteria, traceability — still apply. You’re not throwing away your expertise. You’re extending it into a new domain where it’s desperately needed but almost entirely absent.

Most AI teams don’t understand QE discipline. Most QE teams don’t yet have the AI depth. The intersection of both is where the value is — and where the next generation of quality engineering gets built.

They’re going to pick a tool. They’re going to run a pilot on their UI regression suite. They’re going to report early wins. And eighteen months from now, they’re going to wonder why their testing operation still feels the same.

I’ve seen this pattern play out enough times to know why it happens.

The problem isn’t the tool. It’s the target.

When most organizations say “we’re adopting AI in QE,” what they mean is: we’re going to use AI to speed up test execution. Maybe auto-generate some Selenium scripts. Maybe add a copilot for writing test cases.

That’s not wrong. But it’s optimizing a stage that typically represents 15-20% of total QE effort.

The other 80% — coverage design, data preparation, environment setup, results analysis, defect triage, reporting, regression management — stays untouched. Manual. Expensive. Invisible.

A team can report 70% automation and still spend the majority of its budget on manual work. The metric everyone watches measures the wrong thing.

The real question isn’t “which AI tool should we buy?”

It’s: where in our testing operation would AI actually move the needle?

And you can’t answer that if you don’t know where the effort actually goes.

I’ve worked with QE organizations that were convinced their bottleneck was test execution speed. When we actually mapped the operation — every lifecycle stage, every test type, every release type — we found the real constraint was somewhere else entirely. Data preparation eating two days per release. Environment contention blocking three teams simultaneously. Results analysis where a senior engineer spent half their week manually triaging false failures.

These aren’t glamorous problems. They don’t make for exciting vendor demos. But they’re where the money is.

Why discovery has to come first

Here’s what I mean by discovery: before you evaluate a single AI tool, map your current testing operation from end to end. Not the process diagram on the wiki — what actually happens.

For every test type your organization performs, trace it through all ten stages of the testing lifecycle:

1. Coverage design — how do you decide what to test?

2. Test case creation — who writes them, how long does it take?

3. Script development — what’s automated, what’s maintained by hand?

4. Data preparation — where does test data come from?

5. Environment setup — how long do you wait?

6. Execution — this is the stage everyone focuses on

7. Results analysis — how long does triage take?

8. Defect management — what’s the false positive rate?

9. Reporting — can you answer “are we safe to ship?”

10. Regression management — is the suite growing or grooming?

For each stage, capture who does the work, how they do it, how long it takes, and what it costs. Then compare that to what’s actually possible today with modern tooling and agentic AI.

When you do this honestly, patterns emerge. You find lifecycle stages where the gap between current state and art of possible is enormous. You find stages where a single intervention would cascade through the entire operation. You find that the thing you were about to automate wasn’t actually the bottleneck.

That’s the fact base. Without it, every AI investment is a guess.

The two-phase approach

I frame this as a two-phase journey:

Phase 1: Discover. Map the operation. Build the fact base. Identify where the gaps are largest and where AI would deliver the most impact. Sequence priorities by dependency and ROI.

Phase 2: Transform. Match findings to solutions. Run proof of concepts against your actual environment. Train the team. Deploy. Optimize.

Most organizations skip Phase 1 and jump straight to Phase 2. They pick a tool because a vendor gave a compelling demo, pilot it on the most visible test type, and declare success based on a narrow metric. Meanwhile, the operating model stays the same.

Phase 1 takes 2-3 hours per application. Phase 2, done right, takes months. But Phase 1 is what makes Phase 2 successful.

I built a framework for Phase 1

I’ve put together a structured discovery document that walks you through this process. It covers all five dimensions of a testing operating model — crown jewels, release types, test phases, test types, and the full ten-stage lifecycle — with templates for both deep-dive and lightweight assessment.

It includes:

• A routing section so you only complete what’s relevant

• Full and lightweight lifecycle assessment templates for each test type

• An “art of possible” comparison for every lifecycle stage showing what AI-enabled testing looks like today

• A priority matrix to sequence where to invest first

• A results summary template you can take to your CTO

This isn’t a maturity model. There’s no score at the end. It’s a fact base — the kind of clarity you need before you spend a dollar on AI tooling.

Get the discovery framework

I’m releasing this as a free PDF — it’s v0.9, a beta. I want feedback from practitioners who actually run QE organizations.

Subscribe to this newsletter and I’ll send you the framework directly. It’s free — just drop your email.

If you complete it and want a second opinion on your findings, or if you need help turning them into a funded roadmap, I’m happy to talk. You can book a discovery call at qualityreimagined.com.

The AI tools are getting better every month. The organizations that win won’t be the ones that adopted first — they’ll be the ones that knew where to point them.

Richie Yu works with QE leaders navigating the shift to agentic AI. His focus is on the operating model — not just the tools, but how testing work actually flows and where modernization delivers measurable returns.

The strategic guide for IT leaders evaluating intelligent testing solutions

Your QE team is drowning. Releases that should take days take weeks. You’re hiring more automation engineers but velocity stays flat. Your testing budget is $3M+ annually and growing.

Meanwhile, your dev team just asked: “Can we ship this agentic AI feature next sprint?”

You have two problems colliding:

1. Your current QE approach doesn’t scale (and never will)

2. Agentic AI is about to make it obsolete anyway

But there’s a third option most leaders miss: Use agentic AI to fix testing itself.

The same technology disrupting your product can transform how you test it. Companies doing this are seeing:

• Test execution speed: 10x faster

• Test intelligence: 70% reduction in unnecessary test runs

• QE cost of ownership: down 40%

Here’s how it works, what’s real vs. hype, and how to evaluate solutions.

The Three Promises of Agentic Testing Solutions

Promise #1: Operational Speed (10x Faster Test Execution)

What it means:

Tests that took 6 hours now run in 30 minutes. Feedback loops measured in minutes, not days. Releases no longer waiting on regression testing.

How agentic AI delivers this:

Intelligent test parallelization decides optimal distribution across infrastructure automatically. You’re not manually configuring which tests run where—the AI orchestrates based on historical execution patterns, resource availability, and dependencies.

Autonomous environment provisioning spins up isolated test environments, configures them with the exact dependencies needed, seeds data, and tears everything down when done. What took your team 2-3 days of manual work happens in 20 minutes without human intervention.

Self-healing test scripts fix themselves when the application changes. A button ID changes from submit-btn to submit-button? The AI detects the failure, identifies the new locator using visual recognition and DOM analysis, updates the test, and reruns automatically. Your team never sees the failure.

Adaptive test data generation creates exactly what each test needs, when it needs it. No more maintaining massive seed data files or manually resetting 47 accounts before each test run. The AI generates realistic, contextual test data on demand—customer records, transactions, policy details—that match your production patterns.

Real example:

A major insurance company needed to test complex policy administration workflows. Traditional approaches required 40+ hours of manual environment setup per test cycle—provisioning databases, creating test customer accounts, configuring policies with specific coverage details, setting up provider networks.

With an agentic solution, they automated the entire provisioning process. Isolated environments spin up in under 2 hours with contextual insurance policy data and customer scenarios ready to go. That’s a 20x improvement in operational speed.

The difference isn’t just faster computers. It’s eliminating the manual toil that sits between “start test run” and “see results.”

Promise #2: Smarter Testing (Test the Right Things)

What it means:

Stop running 10,000 tests when 200 would give the same confidence. Intelligently prioritize based on risk, not hunches or outdated heuristics. Catch the bugs that matter, ignore the noise.

How agentic AI delivers this:

Impact analysis on steroids. Traditional impact analysis maps code changes to test coverage using static dependency graphs. Agentic AI goes deeper—it analyzes the code diff, traces runtime dependencies, reviews historical test results, examines production telemetry, and identifies which tests are most likely to catch regressions for this specific change. It’s dynamic, learning, and gets smarter with every commit.

Risk-based test selection combines multiple signals: what code changed, what tests historically caught issues in that code, what’s running in production right now, what customer workflows are most active, and what your business considers high-risk. The AI weighs all of this and selects the optimal test subset.

Intelligent coverage gap detection identifies what’s NOT tested that should be. The AI analyzes your production code paths, compares against test coverage, identifies critical business logic with zero or weak test coverage, and flags it. Some solutions even auto-generate test scenarios for those gaps.

Continuous learning means the system gets better over time. Every test run feeds back: which tests caught real bugs, which ones are perpetually green (dead weight), which ones are flaky, which code areas produce the most defects. The AI adjusts its test selection strategy accordingly.

The business impact:

A banking client ran their full regression suite for every commit: 12,000 tests, 8 hours, $400 in compute per run. Twenty commits per day meant $8,000 in daily test infrastructure costs alone.

With agentic test selection, they run 300-800 tests per commit based on actual risk—code change impact, historical failure patterns, and production usage data. Same confidence in release quality. 95% reduction in test execution cost. Feedback loops went from 8 hours to 15 minutes.

The compounding effect is extraordinary. Faster feedback means developers fix issues while the code is still fresh in their minds. That reduces defect escape rates. Which reduces production incidents. Which reduces emergency fixes and unplanned work. The velocity improvement isn’t linear—it’s exponential.

Promise #3: Lower Total Cost of Ownership (40% Reduction)

What it means:

Fewer QE engineers needed for the same (or better) outcomes. Eliminate the script maintenance tax that currently consumes 40% of QE capacity. Reduce infrastructure and tool sprawl.

How agentic AI delivers this:

Self-maintaining test suites fix themselves continuously. Flaky tests? The AI identifies root causes—timing issues, brittle selectors, environmental dependencies—and refactors them. Duplicated test logic? The AI identifies redundancy and suggests consolidation. Outdated assertions? The AI updates them based on current application behavior and production data patterns.

Autonomous root cause analysis means when tests fail, the AI triages them immediately. It determines if the failure is a real bug, a flaky test, an environmental issue, or a test data problem. It attaches relevant logs, screenshots, network traces, and database states. It clusters similar failures. It suggests fixes. What used to take a QE engineer 45 minutes per failure now takes 3 minutes of review time.

Unified testing platform replaces your sprawl of 6+ tools—separate tools for test management, execution, data management, environment provisioning, reporting, and monitoring. Agentic platforms consolidate these into one intelligent system. Fewer vendor contracts, less integration overhead, lower training costs.

Natural language test authoring allows business analysts and product owners to write tests in plain English. “Verify that a customer with a lapsed policy for more than 90 days cannot file a claim” becomes an executable test without coding. The AI translates intent into test automation. This doesn’t replace QE engineers—it lets them focus on complex scenarios while domain experts handle straightforward functional tests.

The CFO case:

QE teams average 30-40% of their time on test maintenance—fixing flaky tests, updating scripts when UIs change, debugging framework issues, managing test data, fighting with environments.

For a 15-person QE org at $150K average fully-loaded cost (salary + benefits + overhead), that’s $675K-$900K per year spent keeping tests running instead of building new ones.

Agentic solutions cut that maintenance burden by 60-80%. That’s $400K-$700K in recaptured capacity—without hiring a single person. That capacity can go toward:

• Expanding test coverage to new features

• Improving test quality and reliability

• Supporting more teams and projects

• Strategic QE improvements like performance testing or security testing

Plus infrastructure savings: reducing test execution time by 10x means you need 90% less compute. A company spending $500K annually on test infrastructure drops to $50K.

Total cost reduction across labor and infrastructure: 35-45% is realistic in year one.

What’s Real vs. What’s Hype

Let me be the honest broker here. I’ve evaluated dozens of agentic testing platforms, implemented them at major enterprises, and talked to vendors who promise the moon. Here’s what’s actually working versus what’s marketing fantasy.

What’s Actually Working Today:

✅ Intelligent test selection — This is mature. AI can reliably map code changes to impacted tests using a combination of static analysis, runtime dependency tracking, and historical test results. Expect 80-95% reduction in unnecessary test execution with maintained confidence levels. This works.

✅ Self-healing locators — AI can fix broken selectors autonomously, especially for UI tests. When a button ID changes or a CSS class is renamed, the AI uses visual recognition, DOM structure analysis, and element attributes to identify the new locator and update the test. False positive rate is under 5% in production systems. This is production-ready.

✅ Autonomous environment provisioning — Containers plus AI orchestration make this reliable. The AI spins up environments, configures dependencies, manages secrets, provisions databases, seeds data, and tears everything down afterward. Works consistently for cloud-native applications. Companies are seeing 15-30x speed improvements here.

✅ Smart test data generation — AI generates realistic, contextual test data on-demand based on production data patterns (anonymized), schema constraints, and test requirements. Big time-saver. Eliminates the brittle, manually-maintained seed data files that break constantly. This is solid.

✅ Automated root cause analysis — AI can triage test failures, cluster similar issues, attach relevant logs and diagnostic data, and suggest likely causes. Reduces triage time by 70-85%. Not perfect, but good enough that QE teams use it daily without hesitation.

What’s Still Emerging (Use with Caution):

⚠️ Fully autonomous test creation from requirements — Works reasonably well for simple happy-path scenarios and CRUD operations. Still needs significant human oversight for complex business logic, edge cases, and workflows with nuanced rules. You’ll get 60-70% of the way there automatically, then need QE expertise to finish. Useful, but not a QE replacement.

⚠️ AI-generated assertions — Can create basic assertions (element exists, response code is 200, data field is populated). Often misses nuanced business rules and meaningful validation. A generated test might verify a discount was applied but not verify it’s the correct discount based on customer tier and promotion rules. Needs validation and augmentation.

⚠️ Zero-human test maintenance — The vision is real and we’re getting closer, but you’ll still need QE experts. Agentic systems dramatically reduce maintenance burden, but complex test scenarios, framework decisions, and strategic choices still require human judgment. Think 70-80% reduction in maintenance effort, not 100% elimination.

What’s Pure Vendor Hype:

❌ “Replace your entire QE team with AI” — Nonsense. You need fewer people doing different, higher-value work. QE teams shift from script maintenance to test strategy, complex scenario design, quality metrics analysis, and AI oversight. Headcount reduction: 20-40% is realistic. Elimination: no.

❌ “Works out of the box, no training needed” — Every agentic system learns from your codebase, existing tests, historical failures, and production behavior. Initial training period is 2-4 weeks minimum, often 6-8 weeks to reach full effectiveness. Any vendor claiming instant results is lying or selling something far less sophisticated than true agentic AI.

❌ “100% test coverage automatically” — AI can improve coverage significantly by identifying gaps and auto-generating tests, but 100% coverage is neither achievable nor desirable. Chasing 100% coverage wastes resources on low-value tests. Risk-based testing—comprehensive coverage of high-risk paths, selective coverage elsewhere—is smarter. AI helps optimize that trade-off, but it doesn’t eliminate the need for judgment.

❌ “No code changes required” — Most agentic testing platforms work best when you structure tests in ways the AI can understand and manipulate. Some refactoring of existing test suites is typical. Not a full rewrite, but not zero effort either. Budget 10-20% of existing test code needing adjustments.

The honest reality:

Agentic testing solutions are not magic. They’re amplifiers. A dysfunctional QE organization with bad practices, no test strategy, and poorly designed tests will just automate dysfunction faster and waste money on fancy tools.

But a reasonably structured QE practice—clear test objectives, some level of automation already in place, basic CI/CD pipeline—can achieve 10x improvements in the right areas. The ROI is real if your foundation is solid.

The Strategic Evaluation Framework

If you’re evaluating agentic testing solutions, here are the seven questions that separate real capabilities from vaporware.

The 7 Questions to Ask Every Vendor:

1. “What’s the initial training period and data requirement?”

Red flag answer: “Works immediately with zero setup” or “Ready to go in minutes”

Good answer: “Requires 2-4 weeks of learning from your codebase, test execution history, and production telemetry. We need access to your test results from the past 3-6 months, code repository, and CI/CD pipeline data. Performance improves continuously but reaches baseline effectiveness around week 4.”

Why this matters: Real machine learning requires data and time. Instant results mean simple rule-based automation dressed up as AI.

2. “How does it handle our legacy test suite?”

Red flag answer: “You’ll need to rewrite everything to work with our platform” or vague handwaving about “migration support”

Good answer: “Works alongside existing tests written in Selenium, Playwright, Cypress, or your current framework. Gradually improves them through refactoring suggestions and self-healing capabilities. Migration path available but not required—you can start getting value from existing tests on day one.”

Why this matters: You have thousands of existing tests representing years of investment. Throwing them away is prohibitively expensive. The solution should enhance what you have.

3. “Where’s the human still required?”

Red flag answer: “Nowhere, it’s fully autonomous” or defensive avoidance of the question

Good answer: “Humans define overall test strategy and risk appetite, validate AI decisions for high-risk changes, design complex test scenarios with nuanced business logic, manage exceptions and edge cases the AI hasn’t learned yet, and provide feedback to improve the AI. The AI handles execution, maintenance, environment management, and triage. We see teams shift from 70% execution work to 70% strategy work.”

Why this matters: Vendors trying to sell complete human replacement are either lying or selling something far less capable than advertised. Honest vendors explain the human-AI collaboration model.

4. “What’s the ROI timeline?”

Red flag answer: “Immediate savings from day one” or “ROI in the first week”

Good answer: “Month 1: Setup and training, limited value. Month 2-3: System reaches baseline performance, you’ll see 30-50% of projected value. Month 6: Measurable ROI as the system learns your patterns and teams adapt their workflows. Month 12: Full value realization with 10x improvements in targeted areas. Total ROI: 3-5x annual subscription cost by end of year one.”

Why this matters: Real transformations take time. Vendors who promise instant results are selling you disappointment.

5. “How does it integrate with our existing CI/CD pipeline?”

Red flag answer: “You’ll need to change your entire pipeline” or “Best if you adopt our end-to-end platform”

Good answer: “Plugs into Jenkins, GitLab CI, GitHub Actions, Azure DevOps, or CircleCI via REST API and webhooks. Works with your existing test framework—no rip-and-replace. Adds intelligence layer on top of current tooling. Implementation typically takes 1-2 weeks for initial integration.”

Why this matters: Replacing your entire CI/CD infrastructure is a multi-million dollar, year-long effort. The solution should fit your current architecture, not force you to rebuild everything.

6. “What happens when the AI makes a mistake?”

Red flag answer: “It doesn’t make mistakes” or “Our accuracy is 99.9%” without explaining the failure scenario

Good answer: “AI decisions are logged with confidence scores. Low-confidence decisions trigger human review before execution. For critical test paths you designate, we enforce human-in-the-loop approval. Full audit trail of all AI actions with rollback capabilities. When mistakes happen—and they will early on—the system learns from the correction and improves. We track AI accuracy over time and surface it in dashboards.”

Why this matters: All AI systems make mistakes. The question is how the platform handles failures, learns from them, and gives you control over risk tolerance.

7. “Can you show me a similar customer (industry, scale, tech stack)?”

Red flag answer: “We’re too new to have case studies” or “All our customers are under NDA” or showing you a customer in a completely different industry with different problems

Good answer: Provides anonymized case study or facilitates reference customer conversation with a company in your industry, similar scale (within 2x of your team size), and comparable technical environment. Shares specific metrics: before/after test execution time, maintenance effort reduction, infrastructure cost savings, defect escape rate changes.

Why this matters: You’re not buying bleeding-edge research. You’re buying a solution to a business problem. You need proof it works for companies like yours.

The Build vs. Buy Decision

Should you build your own agentic testing solution or buy one?

Build if:

• You have a 20+ person QE organization with deep engineering expertise

• You have highly specialized testing needs that commercial products can’t address (think unique regulatory requirements, proprietary systems, classified environments)

• You have 6-12 months and $1M-$2M budget for R&D with no guaranteed outcome

• You want to own the intellectual property and customize deeply for competitive advantage

• You have leadership commitment to maintain and evolve the solution long-term

Buy if:

• You need results in 3-6 months, not 18-24 months

• Your QE team is already stretched thin—adding a major development project will break them

• You want to focus QE expertise on domain-specific testing strategy, not building infrastructure

• You value vendor support, ongoing innovation, and someone else handling the AI/ML complexity

• You want predictable costs and faster time-to-value

My take for most enterprises:

Buy the agentic testing platform. Build the domain-specific strategy layer on top.

You’re an insurance company, a bank, or a government agency. Your competitive advantage isn’t in building testing infrastructure. It’s in applying intelligent testing to YOUR unique risk profile, regulatory requirements, and business workflows.

Let the platform vendor handle the AI models, self-healing algorithms, infrastructure orchestration, and continuous improvement of the core engine. You focus on:

• Defining what “high-risk” means for your business

• Designing test scenarios for your specific domain (claims processing, loan origination, benefits administration)

• Integrating with your proprietary systems

• Training the AI on your unique application patterns

This is the same logic you use for every other infrastructure decision. You didn’t build your own database, application server, or cloud platform. You bought best-in-class infrastructure and built your differentiating capabilities on top.

Testing infrastructure should be no different.

Three Ways to Start

Don’t wait until your competitors are shipping 10x faster. Here are three practical paths forward, from lowest risk to highest strategic impact.

Option 1: Pilot Project (Lowest Risk)

Pick one high-value test suite to pilot the agentic approach:

• Your regression test suite (typically the biggest time sink)

• Smoke tests (high-frequency execution, clear success criteria)

• Critical path tests (high business impact, easy to measure improvement)

Run the agentic solution in parallel with your existing tests for 4 weeks. Don’t replace anything yet—just observe and measure.

Measure:

• Execution time (should see 5-10x improvement)

• Maintenance effort (track hours spent fixing tests)

• Defect detection (are you catching the same bugs plus new ones?)

• False positive rate (fewer should be better)

Decision point at week 4: If the data shows clear improvement with acceptable risk, expand to more test suites. If results are marginal, either adjust the approach or stop. You’re out 4 weeks and limited cost, not a multi-year commitment.

Best for: Risk-averse organizations, regulated industries, teams with limited bandwidth for change.

Option 2: Greenfield Application (Highest ROI Potential)

Apply agentic testing to a new project from day one:

• Your agentic AI initiative (test the AI with AI)

• Cloud migration project (new infrastructure, fresh start)

• New product launch (no legacy constraints)

Why this works:

No legacy test suite to migrate. No entrenched processes to change. No resistance from teams attached to old ways. You can design modern QE practices from scratch and demonstrate value quickly.

Learn on the new project where stakes are lower and complexity is contained. Then use that success story to retrofit agentic testing to legacy systems with organizational buy-in and proven playbooks.

Timeline: Value in 6-8 weeks. Full ROI within 6 months.

Best for: Organizations with significant new initiatives underway, teams ready to experiment, leadership willing to champion new approaches.

Option 3: Strategic Assessment First (Smartest for Most Organizations)

Start with a structured diagnostic of your current QE practice:

Week 1-2: Current state assessment

• Map your test suites, coverage, execution times, maintenance burden

• Interview QE team members about pain points and time allocation

• Analyze test results, failure patterns, and false positive rates

• Review tooling, infrastructure costs, and team capacity

Week 3: Opportunity identification

• Identify highest-impact opportunities for agentic testing

• Prioritize based on ROI potential (quick wins vs. strategic bets)

• Map dependencies and integration requirements

• Assess team readiness and capability gaps

Week 4: Roadmap and business case

• Build detailed implementation roadmap with phases

• Project ROI with conservative assumptions

• Identify risks and mitigation strategies

• Define success metrics and governance model

Deliverable: A clear, evidence-based decision on whether to proceed, which approach to take, and what outcomes to expect.

Investment: $15K-$30K for the assessment. Saves you from a $500K+ mistake if the timing or approach is wrong.

Best for: Most enterprises. You don’t know what you don’t know. Get clarity before committing to a major change.

What NOT to Do:

❌ Don’t wait for perfect clarity. You’ll never have complete information. The market is moving. Start learning now.

❌ Don’t try to transform everything at once. Pilot, learn, adjust, expand. Boiling the ocean fails.

❌ Don’t buy tools without understanding your current state. Agentic testing platforms are powerful, but they can’t fix fundamental dysfunction. Assess first, then tool.

❌ Don’t let perfect be the enemy of good. You don’t need 100% AI-driven testing. You need 10x improvement in your biggest bottlenecks. Focus there first.

If You’re Evaluating Agentic Testing Solutions or Trying to Modernize Your QE Practice, Let’s Talk

I help enterprises navigate this transition—from assessment to strategy to implementation. I’ve seen what works, what’s hype, and what’s worth investing in.

I bring:

• Real implementation experience deploying agentic testing solutions at major enterprises in insurance and banking

• Vendor-neutral perspective (I evaluate solutions, I don’t sell them)

• Strategic business lens (this is about velocity, cost, and competitive advantage, not just technology)

• Practical roadmaps (not theoretical frameworks—actual week-by-week plans)

Three Ways I Can Help:

1. QE Modernization Assessment - Comprehensive diagnostic of your current QE practice plus prioritized roadmap for agentic testing adoption. You get clarity on where you are, where you should go, and what it will take to get there.

2. Agentic AI Testing Advisory - Hands-on strategy for testing your agentic AI projects. I work alongside your team or your vendors to design the testing approach, implement it, and transition it to your team for ongoing ownership.

3. Fractional QE Leader - Embed in your organization to lead the full QE transformation. I assess current state, identify opportunities, build the strategy, select and implement solutions, and develop your team’s capability to sustain it after I’m gone.

Book a 15-Minute Diagnostic Call

No sales pitch. No obligation.

We’ll discuss:

• Your current QE challenges and bottlenecks

• Whether agentic testing makes sense for your context

• What approach would likely deliver the best ROI

• Honest assessment of timing and readiness

[Book a 15 min no-obligation Call]

Let’s make your QE practice a competitive advantage, not a bottleneck.

It got split.

One track uses agentic AI to speed up the testing you already do. The other track introduces new testing methods because more systems now behave like agents, not deterministic apps.

One line to remember: In 2025, testing stopped being a suite. It started becoming a verification system.

Executive summary

Two things happened at the same time in 2025.

First, change got cheaper. AI lowered the cost of producing code, so teams shipped more changes, more frequently, and more in parallel. That is not just a velocity story. It is a control story. When change volume rises, the constraint shifts away from execution and toward interpretation. You can run a thousand tests. You still need to decide what those results mean, what risk you are accepting, and whether you are shipping.

Second, behavior got harder to predict. More workflows started to include agents that plan steps, retrieve context, call tools, and adapt at runtime. In these systems, failures are less likely to be “a button is broken” and more likely to be “the system took the wrong action for this user in this situation.” The old testing model still applies in parts, but it no longer covers the full risk surface.

The result is a fork in the road:

• Track A: Use agentic AI to accelerate existing testing approaches.

• Track B: Build evaluation methods to test agentic workflows themselves.

Most organizations adopt Track A first because it maps cleanly to today’s QA operating model and budgets. Track B is moving faster because the major platforms are productizing it inside their agent builders.

Trend 1: The testing market split into two tracks

Track A: Agentic acceleration of classic testing

This is the “do the same job with less toil” wave. It is less about inventing new testing theory and more about compressing the manual parts of testing that never scaled well: authoring, maintenance, triage, environment wrangling, and evidence packaging.

In practice, Track A shows up as:

• Faster test creation from intent (requirements, flows, usage telemetry)

• Lower maintenance through self-healing patterns and resilient automation

• Triage automation that classifies failures and proposes next actions

• Execution industrialization through managed grids, parallelism, and shared control planes

This is why “autonomous QA” became a credible category. Not because organizations suddenly love AI. Because maintenance costs and bottlenecks become impossible to hide under higher change volume.

Track A is the fastest path to productivity gains. It is also the easiest place to overclaim. A tool can generate tests. The hard part is keeping them stable, keeping them relevant, and keeping their results interpretable in the release window.

Proof points (2025)

• Functionize, autonomous QA funding push: https://www.functionize.com/blog/functionize-raises-41m-series-b-to-accelerate-the-future-of-autonomous-quality-assurance

• Continuous testing as infra: https://testkube.io/blog/announcing-testkube-8m-series-a

• Verification layer framing: https://momentic.ai/blog/series-a

Track B: New methods to test agentic workflows

Track B starts with a different assumption: the oracle changed.

If your system is an agent, expected results are not always a single deterministic answer. Correctness can depend on what context was retrieved, what tool was selected, what order actions were executed in, and whether safety rules were followed.

So Track B looks less like test automation and more like evaluation engineering:

• Build datasets of scenarios that represent real user intents and risk conditions

• Create rubrics and graders that formalize what “good” looks like

• Score traces and trajectories to verify behavior, not just outputs

• Run user simulation to stress the system under variance and ambiguity

• Monitor production sampling to detect drift after release

If your organization is shipping agents without evaluation assets, you are not testing. You are demoing.

Proof points (2025)

• Trajectory evaluation: https://cloud.google.com/blog/products/ai-machine-learning/introducing-agent-evaluation-in-vertex-ai-gen-ai-evaluation-service

• User simulation harness: https://developers.googleblog.com/announcing-user-simulation-in-adk-evaluation/

• Eval inside the builder: https://www.microsoft.com/en-us/microsoft-copilot/blog/copilot-studio/build-smarter-test-smarter-agent-evaluation-in-microsoft-copilot-studio/

• Runtime eval plus controls: https://aws.amazon.com/about-aws/whats-new/2025/12/amazon-bedrock-agentcore-policy-evaluations-preview/

• Datasets and trace grading: https://openai.com/index/introducing-agentkit/

Trend 2: Managed execution is becoming table stakes

A quiet but important 2025 shift is that the hardest-to-operate parts of testing are being packaged as managed services by cloud vendors. This changes the economics of the testing tool market, and it changes the operating model for QA teams.

When browsers and devices can be provisioned and parallelized as a managed capability, the strategic question becomes simple: why are we spending scarce engineering time building and maintaining a grid.

This does not mean execution is solved. It means it is being commoditized. The differentiation moves up-stack toward:

• Test selection and coverage statements

• Failure classification and root cause acceleration

• Decision-grade evidence packs, not dashboards

• Traceability from change to coverage to outcome

Proof points (2025)

• Microsoft Playwright Workspaces overview: https://learn.microsoft.com/en-us/azure/app-testing/playwright-workspaces/overview-what-is-microsoft-playwright-workspaces

• Azure App Testing and Playwright Workspaces: https://techcommunity.microsoft.com/blog/appsonazureblog/azure-app-testing-playwright-workspaces-for-local-to-cloud-test-runs/4442711

• AWS Device Farm managed Appium endpoint: https://aws.amazon.com/about-aws/whats-new/2025/11/aws-device-farm-managed-appium-endpoint/

Trend 3: Testing moved upstream into pre-merge quality gates

As code output accelerates, quality has two choices: move earlier, or become an expensive lagging indicator.

In 2025, more attention moved to pre-merge controls: AI-assisted code review, automated PR checks, and policy-based gates that aim to reduce defect injection before runtime testing is even involved.

This is not a “testing replaces review” story. It is a “review becomes programmable” story. If you can codify what good looks like at the change level, you avoid paying for avoidable defects downstream.

This expands the definition of testing. Your quality system is no longer only a pipeline stage. It increasingly includes the controls that shape what gets merged in the first place.

Proof points (2025)

• GitHub Copilot coding agent: https://github.com/newsroom/press-releases/coding-agent-for-github-copilot

• CodeRabbit Series B, “quality gates for AI-powered coding”: https://www.coderabbit.ai/blog/coderabbit-series-b-60-million-quality-gates-for-code-reviews

Trend 4: Evaluation engineering emerged as a real testing discipline

A decade ago, test automation changed the discipline by shifting effort from manual execution to scripted verification.

In 2025, evaluation engineering began shifting effort again, from scripted verification to measured behavior.

Evaluation engineering mirrors familiar patterns:

• Test case design becomes scenario design

• Oracles become rubrics

• Pass or fail becomes scored outcomes against criteria

• Regression becomes dataset replay

• Reliability becomes monitoring plus drift detection

This is where 2026 maturity will be decided. Teams that build evaluation assets with the same rigor as test assets will ship agents with less surprise.

There is also a new complication: your judge can be wrong. If you use an LLM as a judge, you need calibration, consistency checks, and rubric hardening. That will become as normal as test flake management.

Proof points (2025)

• Copilot Studio Agent Evaluation: https://www.microsoft.com/en-us/microsoft-copilot/blog/copilot-studio/build-smarter-test-smarter-agent-evaluation-in-microsoft-copilot-studio/

• Vertex AI agent evaluation: https://cloud.google.com/blog/products/ai-machine-learning/introducing-agent-evaluation-in-vertex-ai-gen-ai-evaluation-service

• Bedrock AgentCore evaluations: https://aws.amazon.com/about-aws/whats-new/2025/12/amazon-bedrock-agentcore-policy-evaluations-preview/

Trend 5: Simulation became mainstream for testing agents

Classic failures often live at the edges of UI and API contracts. Agent failures often live in the middle: intent interpretation, tool choice, step sequencing, and safety behavior under ambiguity.

That is why simulation is becoming standard. A simulated user lets you test the behavior envelope:

• Does the agent get stuck or loop

• Does it call the wrong tool

• Does it take unsafe actions

• Does it fail gracefully and escalate when it should

Simulation is becoming the agent equivalent of performance testing. You are not only testing one path. You are testing stability under variance.

Proof points (2025)

• Coval simulation framing (TechCrunch): https://techcrunch.com/2025/01/23/coval-evaluates-ai-voice-and-chat-agents-like-self-driving-cars/

• Bluejay seed round coverage: https://www.businesswire.com/news/home/20250828002083/en/Bluejay-Raises-4M-Seed-to-Help-Build-Reliable-AI-Agents

Trend 6: Observability converged with testing

For classic systems, testing is a gate.

For agentic systems, testing becomes a control loop. You need pre-release evaluation, but you also need post-release evidence that behavior remains stable as prompts, tools, models, and context sources evolve.

In practice, the loop becomes:

1. Define scenarios and rubrics

2. Run evaluations before release

3. Sample real production interactions

4. Score them with the same graders

5. Detect drift and regressions

6. Feed failures back into the dataset

This changes the QA org’s responsibilities. Quality is no longer only readiness. It becomes ongoing behavioral reliability.

Proof points (2025)

• HoneyHive launch and funding: https://www.prnewswire.com/news-releases/honeyhive-a-leadingai-agent-observability-and-evaluation-platform-announces-launch-and-7-4m-in-total-funding-led-by-insight-partners-302419249.html

• Cekura funding post: https://www.cekura.ai/blogs/fundraise

Trend 7: Multi-agent “mission control” created a new testing problem

As platforms moved from single agents to multi-agent orchestration, new failure modes emerged that look more like distributed systems problems.

The defect is often not within one agent. It is in the handoff:

• The wrong agent gets the task

• Context gets lost between steps

• Two agents produce conflicting outputs

• Retry loops create runaway behavior

• Fallback paths do not trigger when needed

This introduces a new test category:

• Collaboration tests that validate correct delegation

• Contract tests between agents that validate artifact formats and assumptions

• Orchestration policy tests that validate routing rules, priorities, and escalation paths

• Trace-based debugging as a default operating mode

If you are adopting multi-agent architectures, orchestration becomes part of the system under test.

Proof points (2025)

• Copilot Studio multi-agent orchestration (Build 2025): https://www.microsoft.com/en-us/microsoft-copilot/blog/copilot-studio/multi-agent-orchestration-maker-controls-and-more-microsoft-copilot-studio-announcements-at-microsoft-build-2025/

• GitHub Agent HQ: https://github.blog/news-insights/company-news/welcome-home-agents/

Market signals: where money flowed in 2025

This is not a directory. It is a signal that the market is funding both speed and control.

Track A: Agentic AI to speed up existing testing methods

These bets assume the enterprise problem is still classic QA, but the manual work and maintenance costs do not scale with AI-driven change volume.

• Functionize, autonomous QA: https://www.functionize.com/blog/functionize-raises-41m-series-b-to-accelerate-the-future-of-autonomous-quality-assurance

• Testkube, continuous testing infra: https://testkube.io/blog/announcing-testkube-8m-series-a

• Momentic, verification layer framing: https://momentic.ai/blog/series-a

• CodeRabbit, PR quality gates: https://www.coderabbit.ai/blog/coderabbit-series-b-60-million-quality-gates-for-code-reviews

Track B: New methods to test agentic workflows

These bets assume agents are becoming production-critical, and testing must become evaluation plus monitoring.

• HoneyHive, evals plus observability: https://www.prnewswire.com/news-releases/honeyhive-a-leadingai-agent-observability-and-evaluation-platform-announces-launch-and-7-4m-in-total-funding-led-by-insight-partners-302419249.html

• Coval, simulation for agents: https://techcrunch.com/2025/01/23/coval-evaluates-ai-voice-and-chat-agents-like-self-driving-cars/

• Cekura, QA plus observability: https://www.cekura.ai/blogs/fundraise

The takeaway: the market is funding both speed and control. Speed is Track A. Control is Track B. Your 2026 testing strategy needs both.

What QE leaders should do next

1) Adopt a two-track testing strategy

Run Track A and Track B as separate programs, with separate artifacts and KPIs. Track A reduces toil. Track B reduces behavioral surprise.

If you treat them as one initiative, you will measure the wrong outcomes. You will optimize for test count and execution speed when the real risk is behavior quality and drift.

2) Build evaluation assets like you build test assets

Start building durable assets that survive tool changes: datasets, rubrics, scenario libraries, and trace schemas. Treat them like IP. Version them. Review them. Reuse them.

3) Assume execution is commoditizing

Use managed execution where it makes sense. Then invest differentiation budget into intelligence: test selection, evidence packaging, triage correctness, and decision automation.

4) Add coordination testing to your scope

If your organization is building agent teams, add collaboration, handoff, and orchestration testing as first-class scope.

5) Treat production as part of the quality loop

For agents, quality is monitored behavior. Build a practical control loop that uses the same rubrics before and after release.

The takeaway

2025 did not make testing irrelevant.

It raised the bar.

The winners will not be the teams with the biggest suite. They will be the teams with the best verification system: the ability to produce a clear verdict, with evidence, at the speed that change arrives.

A simple self-check for 2026:

Are you scaling test execution, or are you scaling confidence production.

Most QA organizations can run tests faster than ever. Under load, that does not translate into faster release decisions.

Under load means this: more change lands in parallel than your organization can confidently interpret inside the release window. The pipeline can execute. The organization cannot reliably produce a Verdict + Evidence Pack before the train needs to leave.

A Verdict + Evidence Pack is simple:

• Verdict: ship or do not ship, plus the risk you are accepting.

• Evidence Pack: the coverage statement, failure classification (product vs test vs environment vs data), environment and data state, rerun rationale, and traceable links from change to tests to results.

One line to remember: Execution scales. Manual confidence production does not.

If you only skim one section, skim this

When the assurance loop is mostly manual, it breaks in five predictable failure modes:

1. Understanding breaks
   You cannot answer “what changed and what could this break,” so planning turns defensive.

2. Readiness breaks
   Environments and test data become the choke point, so failures stop meaning what they should mean.

3. Asset trust breaks
   Automation and suites decay faster than they can be maintained, so run volume rises while confidence falls.

4. Signal breaks
   Triage becomes the job, noise becomes normalized, and senior people become the quality system.

5. Decision breaks
   Evidence is late or fragmented, so sign-off becomes negotiation instead of a decision supported by proof.

The rest of this post is the diagnostic, written in the words your teams already use.

The manual assurance loop, in testing terms

Most enterprises already run the same loop. They just do not call it an assurance system. They call it QA delivery, release readiness, or the test lifecycle.

Intake and change understanding
Requirements review, grooming, acceptance criteria clarification, and “what changed” analysis across code, config, feature flags, entitlements, data contracts, and dependencies.
Output: what is in scope, what changed, what could be impacted.

Test planning and risk-based coverage
Impact analysis, deciding what gets Smoke, Sanity, Regression, SIT, E2E, UAT support, plus any performance and security gates. Also the explicit risk call on what is covered vs what is being accepted.
Output: a test plan tied to risk, plus an explicit coverage statement.

Test design and test asset readiness
Writing and updating test cases, maintaining scripts, updating page objects, fixing brittle locators, keeping suites aligned with product and APIs.
Output: runnable, relevant test assets.

Environment management and release readiness
Provisioning, deployment coordination, version alignment across services, integration availability, feature flag configuration, confirming the environment is in the right state to test.
Output: an environment stable enough that failures mean something.

Test data management
Data creation and seeding, masking constraints, account and entitlement setup, synthetic data generation, keeping golden records stable across runs.
Output: reproducible data and accounts.

Test execution
Pipelines plus human execution where needed. Smoke, BVT, regression, integration checks, E2E, exploratory passes, targeted verification of risky areas.
Output: raw results.

Defect triage and failure analysis
Separating product defects from test defects, environment issues, and data issues. Reruns, stabilization, escalation, root-cause hints.
Output: cleaned signal, not a pile of failures.

Reporting, evidence, and sign-off
Test summary, readiness report, defect narrative, coverage statement, plus the rationale behind go or no-go.
Output: a decision recommendation leaders can defend.

Post-release learning
Leakage analysis, suite tuning, data and environment hardening, feeding fixes back into standards and assets.
Output: less noise and better coverage next time.

That is the loop. If it is mostly manual, you can still run it. You just cannot run it fast enough when change volume rises and work lands in parallel.

Why it breaks under load

Under load, specifically the high-frequency parallel change volume driven by AI assistants, the loop does not break because teams forget how to test. It breaks because the parts that produce confidence are the parts that scale poorly.

You do not need all of these to feel pain. A handful is enough to turn release readiness into a recurring debate.

1) Understanding breaks, so planning turns defensive

Symptom: you cannot confidently answer “what changed and what could this break,” so you compensate by running more, not smarter.

This shows up as:

• Thin or shifting stories and acceptance criteria

• Hidden blast radius from config, feature flags, permissions, entitlements, routing, caching, or infrastructure changes

• Dependency drift where updates change behavior outside the touched component

• Parallel change collisions with no consolidated impact view

• Release candidate mismatch between what was tested and what is shipping

• Weak mapping from changes to affected features, tests, and risks

• Broken traceability from ticket to code to test intent to evidence

Result: regression widens as insurance. Duration and noise increase. Uncertainty does not drop.

2) Readiness breaks, so failures stop meaning what they should mean

Symptom: the suite is fine. The state is not.

This shows up as environment issues:

• Shared environment contention as trains collide

• Version skew across services that creates non-product failures

• Deploy coordination delays that compress test windows

• Non-prod realism gaps where test differs materially from production

• Infrastructure flakiness from throttling, network variability, and unstable dependencies

• Manual resets that are slow and unreliable

• Environment scarcity that forces serialization of work

And it shows up as data issues:

• Account and entitlement drift that breaks flows unexpectedly

• Rotting golden datasets that stop representing reality

• Ticket-based data creation that is slow and knowledge-driven

• Masking constraints that reduce usable data and force brittle synthetic setups

• Data collisions across runs that create intermittent failures

• Non-deterministic setup that makes reruns meaningless

• External data dependencies that break reproducibility

Result: teams spend more time preparing to test than testing, and then spend more time arguing about whether failures are real.

3) Asset trust breaks, so automation rises while confidence falls

Symptom: maintenance grows faster than feature delivery. This is not a moral failure. It is math.

This shows up as:

• Brittle selectors and timing that create false failures

• UI churn that breaks tests even when behavior is logically unchanged

• API contract drift that creates silent gaps or noisy suites

• Framework and runner drift that reduces execution consistency

• Test debt accumulation because new coverage crowds out maintenance

• A shrinking trusted subset inside an expanding suite

Result: teams start saying “automation is high but confidence is low,” then treat the suite like a best-effort signal instead of a decision system.

4) Signal breaks, so triage becomes the job

Symptom: triage becomes full-time operating mode, and confidence becomes dependent on your most senior people.

This shows up as:

• Too many failures to interpret inside the release window

• Normalized flakiness that hides true signals

• Cross-team root cause hunts without enough visibility

• Inconsistent classification across test, environment, data, and product defects

• Repeat investigations because learning does not compound

• Manual fix verification that requires another broad run

Result: senior people spend their time cleaning noise instead of raising capability, and the system’s throughput becomes a people constraint.

5) Decision breaks, so sign-off becomes negotiation

Symptom: release readiness turns into meetings because evidence is not decision-grade.

This shows up as:

• Pass rates without meaning because nobody trusts what they imply

• Green dashboards with red risk because failures cluster in critical paths

• “One more run” syndrome because evidence is not conclusive

• Weak risk mapping from coverage to business impact

• Social sign-off based on credibility and fatigue

• Undocumented exceptions where risk is accepted silently

It also shows up in evidence capture:

• After-the-fact summaries assembled under deadline pressure

• Scattered evidence across tools, screenshots, and chat threads

• Incomplete repros that drag investigations

• No decision replay weeks later when incidents happen

• Slow retrospectives because the system cannot answer “why we shipped”

Result: the bottleneck is not testing. It is decision-making with evidence.

The pattern

Notice the theme. The bottlenecks are not in execution. They are in interpretation, state, and evidence.

You can scale execution by adding compute. You cannot scale interpretation by adding humans without increasing delay, inconsistency, handoffs, and cost.

The multiplier effect: Agentic workflows amplify every failure mode

Agentic workflows do not replace the classic assurance problem. They multiply it.

If the manual loop is already struggling to produce a Verdict + Evidence Pack for deterministic systems, it will fail faster when the system under test includes non-deterministic behavior that requires evaluation, baselines, and replay, not just pass/fail assertions.

This shows up as:

• Unstable expected outputs where classic assertions are insufficient

• No behavioral baseline, so drift becomes invisible

• Tool invocation gaps where constraint violations are not captured

• Prompt and policy changes not treated as release-impacting change

• Missing reference sets for evaluation and regression

• Weak observability, so teams cannot explain decisions

• Low reproducibility without replay artifacts

Result: you need new methods, but you still have the old manual bottlenecks. Under load, both fail at the same time.

The punchline

The manual assurance loop breaks under load because confidence depends on human throughput across understanding, readiness, triage, and evidence.

You can run a million tests. If the meaning of those tests requires hours of human interpretation to decide whether the product is safe, you are still too slow.

What it looks like when you are already overloaded

If you recognize several of these, you are operating beyond manual capacity:

• Readiness calls feel like negotiations

• Reruns happen to feel better, not to learn

• Flaky failures are tolerated, then normalized

• Stabilization windows grow even as execution gets faster

• Hotfix culture grows because surprises show up late

• Evidence is assembled after the fact

• Senior people spend their time triaging noise

• Teams stop trusting dashboards and start trusting people

These are not signs you need more testing. They are signs you need a different loop.

Next

If you are feeling the symptoms above, the answer is not “run more tests.” The answer is a different loop: one that produces a Verdict + Evidence Pack as a computed output, not a manual scramble.

That different loop is the Agentic Assurance Loop. In the next post, we will look at how to move from manual confidence to computed confidence, where impact analysis, readiness checks, noise suppression, and evidence assembly are treated as automation problems, not meeting problems.

Executive Summary

AI is showing up in two places: inside software delivery, and inside the software itself. QA feels both shifts at once.

First, AI-augmented coding makes shipping change cheaper and faster. Release cadence tightens, and volume increases. If you do nothing, the current QA operating model relies heavily on human analysis between test runs. It cannot keep up with the pace of production.

Second, modern architectures increasingly include agentic workflows. These components plan steps and adapt behavior at runtime. Traditional deterministic testing is still necessary, but it cannot be your only signal anymore.

The response is to evolve QA from a testing function into an Assurance System. Execution is the easy part. The hard part is producing a defensible decision. The modern system must automate the analysis and evidence packaging that sits around the test run.

One line to remember: Quality is not just “more testing.” Quality is the system that produces a release decision with evidence, at the pace of delivery.

What Changed

For years, software delivery had a practical limit: delivering change was expensive. Even with good automation, volume was constrained by human throughput across design, coding, and reviews.

That limit is moving.

AI-augmented development reduces the cost of getting changes into production. Teams can generate more variants, refactors, fixes, and features with less friction. The business pushes for smaller, more frequent deployments because it feels safer.

At the same time, the software itself is changing. Products now include agentic capabilities. Instead of following a fixed script, these systems choose actions based on context, tools, and policies. Output can vary even when inputs look similar.

This is why Quality Engineering needs a new point of view.

The Moment the Old Model Breaks

It is Thursday afternoon. The release cadence used to be bi-weekly. Now it is weekly. Product wants twice a week because “we can do smaller changes safely.”

You have more automation than ever. The pipeline runs fast. The dashboards look busy.

And yet, confidence is low.

The readiness call starts, and the questions are predictable but difficult to answer: What actually changed since the last safe build? What is the blast radius? Do we have evidence this specific risk is covered?

Someone shares a pass rate. It’s green overall, but nobody can explain what the red failures mean for this release. Someone says they are flaky. Someone wants one more run. The decision turns into a negotiation based on intuition rather than data.

At that point, it becomes clear: The bottleneck is not test execution. The bottleneck is confidence production.

You can run a million tests, but if the results require hours of human interpretation to understand if the product is safe, you are still too slow.

The Key Reframing: The QA Org is an Assurance System

Most organizations talk about QA as a department or a set of activities. A more useful framing is that QA is an Assurance System.

It is the set of people, partners, practices, and tools that exist to answer one question repeatedly: Is this change safe enough to ship, and what is the evidence?

The system already exists today. It just has two gaps that are exposed by this new era:

1. Speed: It relies on too much manual work (analysis, data prep, triage) to produce readiness decisions at the speed of modern delivery.

2. Coverage: It lacks standardized methods to evaluate non-deterministic (agentic) workflows.

The mandate is not to “optimize testing.” It is to evolve the system so it can produce decisions as fast as developers ship code.

The Solution: Two Lanes, One System

This is not about splitting QA into two disconnected worlds. It is about extending the Assurance System to cover a broader surface.

Lane 1: Deterministic Quality
This is the discipline we know. UI, API, mobile, and data flows where expected results are stable. The goal here is efficiency.

We must apply AI to the “human middleware” steps. Automate change impact analysis, test data generation, and failure triage so that a “Pass” result actually equals “Ready to Ship” without a manual interpretation phase.

Lane 2: Agentic Quality
This is the new discipline. For agentic workflows, the question is not “did it match the expected value?” but “did it behave within acceptable boundaries?”

This requires new methods:

• Scored Evaluations: Grading outputs against policies and reference sets rather than exact matches.

• Constraint Checks: Verifying that the agent did not attempt prohibited actions or tools.

• Drift Monitoring: Detecting if behavior is shifting over time compared to a baseline.

One readiness decision needs both signals. One Assurance System owns both.

What “Good” Looks Like

A modern Assurance System behaves like a closed loop. Every meaningful change triggers the same process:

1. Ingest: It understands what changed in the code and behavior.

2. Plan: It determines the blast radius and what coverage is required.

3. Execute: It verifies across both Deterministic and Agentic lanes.

4. Decide: It produces a clear Verdict + Evidence Pack that explains why.

Humans stay in the loop, but their role shifts up the stack. Less time is spent on repetitive execution or arguing about flaky tests. More time is spent defining policies, risk boundaries, and coverage intent.

What Comes Next

This series will double-click on the practical implementation of this view.

• The Assessment: A candid walkthrough of today’s assurance lifecycle to identify where it breaks under load.

• The Mechanics: What the modern loop looks like in practice, including the capabilities that matter most to automate the high-friction steps (like data and environments).

• The Migration: How to move from “testing” to “assurance” without boiling the ocean, starting with the highest-risk workflows.

The goal is simple: Keep shipping more change, while making release confidence faster, clearer, and defensible.

How to use this: Score each statement 0–2. Add the totals by section. Your lowest sections are your priority constraints.
0 = Not in place | 1 = Partially | 2 = Consistent at scale

If you want a lightweight sanity check, reply to this post with your section totals and your top constraint. I will respond with a few priority moves to consider.

A) Operating model and ownership (0–10)

1. Decision rights are clear (standards, quality gates, exceptions). 0 1 2

2. The engagement model is explicit (embedded, shared services, hybrid). 0 1 2

3. QE responsibilities across Dev, QA, Product, and Ops are defined and practiced. 0 1 2

4. There is a modernization backlog with an owner, funding, and cadence. 0 1 2

5. Standards are adopted through enablement, not policing. 0 1 2

Section total (0–10): ____

B) Delivery integration and feedback loops (0–10)

6. Quality signals arrive early enough to change outcomes. 0 1 2

7. Environments, test data, and dependencies are managed as first-class constraints. 0 1 2

8. Triage is repeatable and assigns ownership quickly to reduce MTTR. 0 1 2

9. Teams can distinguish product defects vs test defects vs environment defects. 0 1 2

10. Release readiness is evidence-based and repeatable, not meeting-based. 0 1 2

Section total (0–10): ____

C) Automation effectiveness and economics (0–10)

11. Automation is prioritized by risk and critical user journeys, not test counts. 0 1 2

12. Flaky tests are tracked, owned, and resolved with urgency. 0 1 2

13. Maintenance cost is visible (time spent fixing tests and false failures). 0 1 2

14. Low-value automation has a normal retirement path (kill, rebuild, replace). 0 1 2

15. Execution scales without linear effort (stability and orchestration are addressed). 0 1 2

Section total (0–10): ____

D) Governance, controls, and auditability (0–10)

16. Quality gates are explicit and tuned by risk level, not one-size-fits-all. 0 1 2

17. Evidence for release decisions is captured consistently and easy to retrieve. 0 1 2

18. Exceptions are managed with a defined process, owner, and expiry. 0 1 2

19. Practices align to application criticality (crown jewels vs non-critical apps). 0 1 2

20. Reporting and controls match delivery reality, not an idealized process. 0 1 2

Section total (0–10): ____

E) Quality signals leaders trust (0–10)

21. Leadership reporting focuses on outcomes (risk, stability, confidence), not activity. 0 1 2

22. Trends are visible (leakage, reliability, change failure, incident risk). 0 1 2

23. Teams can explain what changed and why confidence changed since the last release. 0 1 2

24. Signals connect to customer impact (critical journeys, severity, incident history). 0 1 2

25. Decisions move faster because leaders trust the signals and the process behind them. 0 1 2

Section total (0–10): ____

Scoring and what to do next

Total score (0–50): ____

0–15: Foundations missing
Start with Operating Model + Feedback Loops. Without these, everything else becomes chaos.

16–30: Capability exists but does not scale
Focus on Governance + Automation Economics. This is where most orgs get stuck.

31–40: Mature pockets, inconsistent execution
Standardize the operating model, build reusable patterns, and fix signal credibility.

41–50: Strong baseline
Shift from building to optimizing. Tighten metrics, reduce drag, and adopt AI with control.

Turn this into an action plan (5 minutes)

1. Circle your lowest two sections.

2. Write the top two constraints you see behind those scores.

3. Pick one 30-day move that reduces friction immediately.

4. Pick one 90-day move that changes the operating model, not just symptoms.

If you want help translating the score into a modernization plan, reply with your totals and context.

Subscribe now

If you lead Quality Engineering in a growing mid-market company or a large enterprise, you can feel it.

The pace of change is rising. AI is accelerating how software gets built. The old assumptions about testing, automation, and release confidence are being stressed. Some patterns still hold. Others are breaking.

I am not going to pretend there is a finished playbook.

Quality Reimagined is a set of field notes and practical frameworks for QE leaders modernizing their organizations while the landscape shifts. The goal is to help you make good decisions with imperfect information, and build an operating model that stays resilient as capabilities evolve.

Start here: the QE Discovery Framework

Most QE teams are about to invest in AI tooling without knowing where the effort actually goes. The Discovery Framework fixes that.

It is a structured walkthrough that maps your testing operation across all ten lifecycle stages, every test type, and every release type — so you can see exactly where AI would deliver the most impact before you spend a dollar.

The framework includes:

• A routing section so you only complete what is relevant

• Full and lightweight lifecycle assessment templates

• An “art of possible” comparison for every lifecycle stage

• A priority matrix to sequence where to invest first

• A results summary template you can take to your CTO

Subscribe and I will send you the framework directly. It is free.

Subscribe now

What else you will find here

• Decision frameworks for modernization when the path is not obvious

• Operating model patterns that scale across teams and portfolios

• Governance and controls that preserve confidence as change volume increases

• Quality signals leaders can trust for real release decisions

• Practical experiments and lessons learned translated into enterprise action

Conversations with industry leaders

I will also be speaking with QE and engineering leaders, SI partners, and testing tool builders about how they are navigating this transition inside real organizations.

No hype. No vendor theatre. The focus is what is changing, what is not, where teams are getting stuck, and what patterns are emerging that actually scale.

Other starting points

• To ground the conversation, start with the QE Modernization Diagnostic.

• If you prefer practical assets you can use immediately, browse the QE Leader Toolkit.

If you want a lightweight sanity check, reply after you review the diagnostic with your section totals (A–E) and your top constraint (one sentence). I will respond with a few priority moves to consider.

What I mean by modernization

Modernization is not predicting the future. It is building a QE system that can adapt: faster feedback loops, clearer ownership and fewer handoffs, automation that stays worth maintaining, governance that matches delivery reality, and reporting that supports release decisions under uncertainty.

Transparency

This publication is vendor-neutral by default. If I ever include any financial relationship tied to a recommendation, it will be disclosed clearly in the post.

We bought testing platforms based on the elegance of their recorders and the layout of their dashboards. We built entire organizations optimized for humans who log in, click buttons, and visually interpret results.

That agreement is expiring.

We are entering the Agentic Era. AI coding agents like GitHub Copilot Workspace, Devin, and Cursor are moving from autocomplete to autonomy. They are planning architecture, writing code, and generating pull requests at machine speed.

For leaders in BFSI and other regulated industries, this creates a specific crisis. Releases are gated by risk, compliance, and audit reviews. Agentic delivery does not remove these constraints. It amplifies them. A human-in-the-loop model cannot scale when the volume of change increases by orders of magnitude.

This paper argues that the future of Quality Leadership is not about managing a Testing Department, but about building Quality Infrastructure. It describes the shift from buying tools as destinations for humans to building a grid as infrastructure for machines.

Clarifying the Terminology Before proceeding, we must distinguish between two systems that are often conflated.

• The Crown Jewel (Your App): The critical software your organization builds and operates. For example, a loan decision engine, fraud detection service, or pricing platform.

• The Grid (Your Platform): The quality infrastructure used to evaluate, verify, and approve changes to the Crown Jewel.

• The Argument: As decision-making inside Crown Jewels becomes increasingly automated and AI-assisted, the Grid must stop being a human-driven dashboard and become a machine-driven grading engine.

Part 1: The Invisible Crisis

Walk into any modern software delivery organization and you will see the same ritual.

A release candidate is built. Thousands of automated tests run. And then… the pause.

The results sit in a dashboard. A senior release manager logs in, scans the red tests, applies context (e.g., “known environment issue”), and makes a judgment call.

This “human middleware” layer was acceptable when software moved at human speed. It breaks down when systems can be refactored, regenerated, or reconfigured overnight. The volume of change overwhelms the human capacity to interpret dashboards.

In effect, many organizations are telling their CIOs: “We can use AI to write code far faster than before, but we can only ship it as fast as a senior engineer can read a Jenkins report.”

That is not a tooling limitation. It is an architectural bottleneck.

Headless Does Not Mean Blind

For many leaders, the phrase “headless quality” triggers an immediate concern about loss of visibility and control. It means the opposite.

Headless means machine-addressable first, human-readable second.

• Today: Critical risk data is trapped inside proprietary user interfaces. To understand release safety, a human must log in.

• Tomorrow: Risk is an API query. Visualization still exists, but it is optional, not mandatory.

In this environment, a vendor selling a better dashboard is solving a 2015 problem. You do not need a better place to visit. You need a better answer delivered directly to where decisions are made.

The Visual Metaphor: The Stark Reality

To visualize this shift, consider how Tony Stark builds Iron Man armor.

When Stark wants to test a new design, he does not put on the suit and jump off a roof to see if the stabilizers work. That is the old, manual testing model.

Instead, he instructs J.A.R.V.I.S. to run thousands of simulated flight scenarios under extreme conditions. The agent executes the tests. Stark defines the Success Criteria.

“If structural integrity drops below 98 percent at supersonic speed, mark the design as a failure.”

In the Agentic Era, your quality organization is no longer jumping off the roof. It defines what “Good” means. Agents do the execution.

Part 2: The Google Signal

In December 2025, Google provided a real-world signal of this approach with the release of Gemini Deep Research, an autonomous research agent. The headline was the AI. The deeper story was how such a system must be evaluated.

Gemini Deep Research is not validated through UI scripts or interaction checks. Its outputs are long-form, multi-step research reports that must be assessed on reasoning quality, evidence use, and outcome integrity, not whether a button was clicked.

The implication is clear. In an autonomous world, you cannot test a system by watching its cursor. You must evaluate the quality of its decisions and conclusions against defined expectations.

The Lesson for Legacy Systems

This logic does not apply only to AI agents. The same principle applies to loan engines, fraud detection systems, pricing logic, and eligibility rules.

These systems have always required grading rather than simple UI validation. Did the engine offer the correct interest rate? Did it flag the transaction as fraud for the right reason?

A UI script cannot answer those questions. Only a rigorous exam—a curated set of ground-truth scenarios with clear evaluation logic—can. AI simply makes this shift unavoidable. Whether logic is written by a human or an LLM, quality must validate outcomes, not clicks.

Part 3: The New Quality Architecture

To run this exam at scale, the testing platform itself must be re-architected. Most legacy tools are dangerously overweight in the wrong layers.

Layer 1: The Quality System of Record (”The Truth”)

The Grid must be the canonical system of record for quality state. It cannot be a bucket of screenshots and logs. Test outcomes must be structured, queryable signals.

• Litmus Test: Can your platform answer this via API? “Is this release statistically safer than the previous one, and based on what coverage model?”

• The Fail State: If the answer requires logging into a UI and visually comparing charts, the system is failing.

• What This Replaces: Screenshot archives, unstructured log buckets, ad-hoc spreadsheets.

Layer 2: The Verification Engine (”The Grader”)

This is where the exam lives. The verification engine selects relevant scenarios, executes evaluations, and grades outcomes without human intervention.

• Core Logic: Given this code change, which scenarios are relevant? Did the system’s decision align with policy? Can this release be auto-approved?

• What This Replaces: Brittle regression packs, manual test selection, release-manager bottlenecks.

Layer 3: The Consumption Layer (”The View”)

This is where most budgets are currently over-invested. In an agentic world, we do not need permanent dashboards for temporary problems. We need Views on Demand.

• The Shift: If a release fails, generate a concise summary and attach it to the pull request. If an incident occurs, compile diagnostics and deliver them to the right channel.

• What This Replaces: Permanent dashboards and “single-pane-of-glass” portals checked only when it is already too late.

Part 4: The Vendor Conversation

Most testing vendors are currently selling GenAI features such as automated test writing. These features are not strategy. When renewing contracts, audit architecture, not demos.

Tell your vendor: “Our strategy is shifting to agentic workflows. I need to know if your platform is agent-ready.”

Then ask these three questions.

Question 1: The Headless Verdict Test Can an external agent trigger tests and retrieve a definitive go or no-go verdict via API, without a human logging in?

• Bad answer: “Trigger Jenkins and parse XML.”

• Good answer: “We expose a verdict API with confidence scoring.”

Question 2: The Deep Link Test When a test fails, is every artifact accessible via authenticated APIs and URLs? If the UI is mandatory, the agentic loop is broken.

Question 3: The System of Record Test Can your platform act as a canonical system of record for quality decisions, with immutable verdicts, evidence lineage, and audit traceability? If not, it is still just a dashboard.

Part 5: The Organizational Pivot

This shift is not just technical. It changes what your team produces.

The New Asset: The Golden Dataset

Stop measuring success by the number of scripts written. Measure the depth of ground truth. Do you have hundreds of validated examples of correct outcomes? Do you know which edge cases define unacceptable risk?

This dataset is intellectual property. It is your enterprise exam.

The Role Shift: From Scripting to Stewardship

GenAI will commoditize script writing. Stewardship cannot be automated. Modern quality roles focus on curating truth, designing grading logic, and maintaining audit-ready integrity.

Example: A 12-Person QE Team in a Bank

• 4 steward golden datasets for lending, payments, and fraud.

• 3 maintain grading logic aligned to policy and regulation.

• 3 maintain execution infrastructure and data integrity.

• 2 arbitrate release exceptions only.

• 0 are measured on scripts. Everyone is measured on decision confidence produced.

The New KPI: Time to Verdict

Stop measuring test execution time. That is an engineering metric. Measure Time to Verdict.

This is the elapsed time between a code commit and a trusted “safe-or-not-safe” decision. Time to verdict matters because delayed decisions increase both delivery risk and audit exposure.

Conclusion: The Choice

We are at a bifurcation point in software delivery.

One path leads to a Legacy Bottleneck. Humans drown in dashboards, maintaining brittle scripts that verify clicks.

The other leads to the Quality Grid. An organization that acts as the exam board for the enterprise, defining success, grading outcomes, and delivering trusted verdicts at machine speed.

The interface is disposable. The dashboard is dying. Long live the verdict.

The narrative is seductive because it is partially true. We have tools that can scaffold a React app in seconds, explain complex regex, and migrate SQL queries between dialects instantly. The friction of syntax has vanished. For investors and founders, this looks like the Holy Grail: the decoupling of software output from human headcount.

But amongst senior engineering leaders, a different, quieter conversation is happening. It is not about how fast we can move. It is about a growing uneasiness regarding what we are leaving behind.

We are not just generating code faster. We are generating complexity faster. And crucially, we are generating it faster than our current governance, testing, and verification structures can absorb.

We are entering the era of the Verification Gap.

This is not a luddite’s screed against AI. AI is a powerful lever. But Archimedes taught us that a lever needs a fulcrum to work. In software, that fulcrum is Verification. If you lengthen the lever (generation) without strengthening the fulcrum (robust testing, types, and specs), the system does not lift more weight. It snaps.

Here is the reality check on why the next decade of software will not be defined by who can generate the most code, but by who can verify it.

I. The Facade: The Efficiency Illusion

To understand the crisis, we have to look past the “sugar rush” of the demo.

When an AI agent “fixes” a bug or implements a feature, it feels like magic because the typing is instantaneous. The “Time to Pull Request” (TTR) has collapsed. But software engineering was never limited by typing speed. It was limited by mental modeling: the ability to hold the state of a system in your head and predict how a change in Module A affects Module B.

The “Uncanny Valley” of Code

The danger of modern LLMs in 2025 is not that they write bad code. It is that they write plausible code.

In 2022, AI code was often obviously broken. Today, AI code looks like a senior engineer wrote it. It follows patterns, uses clear variable names, and comments profusely. It is often indistinguishable from human code, until it fails.

When a human writes a complex system, they build a mental map of the “why,” the intent behind the constraints. When an AI writes that same system, it is performing high-dimensional pattern matching. It creates code that is syntactically polished but often semantically hollow.

It is the difference between a deeply researched history book and a historical novel. They both look like history. One is grounded in truth. The other is grounded in vibes.

II. The Entropy Engine: How the Debt Piles Up

The central friction of AI development is not “stupidity.” It is entropy.

In physics, entropy is the measure of disorder. In software, entropy is technical debt, fragmentation, and cognitive load. Historically, the friction of writing code acted as a natural throttle on entropy. Because it was hard to write code, we thought twice before adding complexity.

AI removes that throttle.

The “Horizontal Sprawl”

While elite engineering organizations such as Meta or Stripe have the tooling and discipline to absorb this complexity, most teams do not. The result is a rise in “append-only” development.

It is cognitively easier for an AI (and the human guiding it) to duplicate a function and modify it than it is to understand the abstract hierarchy of a shared class and refactor it safely. Modern models can use abstract syntax trees to navigate code, but they still struggle with multi-file, multi-layer architectural refactoring. They excel at the local and struggle with the global.

The result is a shallow sprawl. We are building codebases that grow wider and shallower, filled with near-duplicates. This works fine for a month. But six months later, when you need to change a core business rule, you discover that rule is hard-coded in fifteen slightly different ways across forty files.

The “Instant Legacy” Problem

We are used to thinking of “legacy code” as old code written by people who have left the company. Today, we are creating instant legacy code.

If an AI generates a complex microservice for a critical business path, and a developer merges it after a cursory “looks good to me” review, that code is effectively legacy the moment it lands. The human does not have the mental model of how it works. They did not struggle through the edge cases. They did not build the neural pathways that associate “that variable” with “that specific business risk.”

The knowledge resides in the weights of the model, not the mind of the maintainer. When the model updates or the context window shifts, that knowledge is gone.

III. The Tautology Trap: Why AI Cannot Grade Its Own Homework

“But wait,” the optimist argues. “We will just use AI to write the tests. We will have agents checking agents.”

To be fair, AI already helps with important parts of verification. It is effective at generating fuzz cases, spotting simple security issues, and pointing out localized logic errors. Combined with static analysis, it can be a powerful lens over a codebase.

But when it comes to verifying intent, it suffers from a fundamental flaw: the lack of ground truth.

The Specification Bottleneck

The hardest part of software engineering has never been writing the code. It has been defining the specification. Ambiguity is the enemy of correctness.

When you prompt an AI, you are providing a fuzzy, natural-language specification. If that specification is ambiguous (and it almost always is), the AI must hallucinate the intent.

If you ask an AI to “write code to calculate tax” and then “write a test to verify the tax calculation,” you are often creating a tautology.

• Ambiguous intent: “Round half up.”

• AI code: rounds half down, based on a pattern it has seen.

• AI test: assert(round(2.5) == 2)

The test passes. The green checkmark appears. But the system is wrong.

We have decoupled the mechanics of testing from the value of testing. The value of a test is that it confronts the code with an adversarial truth. If the code and the test share the same blind spots, because they were generated by the same probability distribution, the test is theater.

IV. The Organizational Trap: Output vs Risk

Why are we falling for this? Because the incentives are misaligned.

In most organizations, developers are rewarded for visible output. Shipping features, closing tickets, and merging pull requests are visible activities. AI acts as a supercharger for visible output.

However, risk is invisible. Technical debt, security vulnerabilities, and architectural brittleness accumulate quietly.

AI allows us to maximize visible output while hiding the accumulation of invisible risk. Managers see velocity charts going up and celebrate. Underneath, the Verification Gap is widening. We are borrowing time from our future selves at predatory interest rates.

V. The Solution: From Builders to Auditors

So, is the sky falling? No. But the job description is changing.

If code generation is becoming a commodity, abundant and cheap, then verification is becoming the scarcity, rare and expensive.

The economic value of a software engineer is shifting. We are moving from being construction workers who lay bricks to building inspectors who ensure the structure will not collapse.

1. The Specification Is the Deliverable

In an AI-augmented world, the implementation is a disposable artifact. You might delete and regenerate the implementation five times a day.

The test suite and the formal or semi-formal specification, however, are the assets.

Senior engineers must stop viewing testing as a chore to be outsourced to AI and start viewing it as the codification of reality. The bottleneck is no longer writing the function. The bottleneck is articulating the ground-truth behavior in a verifiable form.

2. The Rise of “Adversarial Engineering”

We need to adopt an adversarial relationship with our tools. We cannot be prompt engineers who coax the AI to do the right thing. We must be audit engineers who assume the AI has done the subtle wrong thing.

This means investing in:

• Property-based testing: defining invariants such as “the result must always be positive” rather than checking a single example.

• Mutation testing: intentionally breaking the code to ensure the tests fail when behavior changes.

• Formal verification: for critical paths, returning to mathematical proofs and model checking, tools that once felt “too academic” but are now necessary guardrails against non-deterministic generation.

Conclusion: The Only Way Out Is Through Rigor

The Verification Gap is the defining challenge of the next era of engineering.

AI will not replace engineers, but it will ruthlessly punish teams that have weak verification cultures. If your process relies on “looking at the code and seeing if it feels right,” you will be buried under a mountain of subtle, hallucinated technical debt.

But if you pivot your culture, if you treat verification as the highest form of engineering, and if you treat AI as a talented but unreliable junior partner, you can ride the wave without drowning.

The code is free. The truth is expensive. Pay for the truth.