They’re going to pick a tool. They’re going to run a pilot on their UI regression suite. They’re going to report early wins. And eighteen months from now, they’re going to wonder why their testing operation still feels the same.

I’ve seen this pattern play out enough times to know why it happens.

The problem isn’t the tool. It’s the target.

When most organizations say “we’re adopting AI in QE,” what they mean is: we’re going to use AI to speed up test execution. Maybe auto-generate some Selenium scripts. Maybe add a copilot for writing test cases.

That’s not wrong. But it’s optimizing a stage that typically represents 15-20% of total QE effort.

The other 80% — coverage design, data preparation, environment setup, results analysis, defect triage, reporting, regression management — stays untouched. Manual. Expensive. Invisible.

A team can report 70% automation and still spend the majority of its budget on manual work. The metric everyone watches measures the wrong thing.

The real question isn’t “which AI tool should we buy?”

It’s: where in our testing operation would AI actually move the needle?

And you can’t answer that if you don’t know where the effort actually goes.

I’ve worked with QE organizations that were convinced their bottleneck was test execution speed. When we actually mapped the operation — every lifecycle stage, every test type, every release type — we found the real constraint was somewhere else entirely. Data preparation eating two days per release. Environment contention blocking three teams simultaneously. Results analysis where a senior engineer spent half their week manually triaging false failures.

These aren’t glamorous problems. They don’t make for exciting vendor demos. But they’re where the money is.

Why discovery has to come first

Here’s what I mean by discovery: before you evaluate a single AI tool, map your current testing operation from end to end. Not the process diagram on the wiki — what actually happens.

For every test type your organization performs, trace it through all ten stages of the testing lifecycle:

1. Coverage design — how do you decide what to test?

2. Test case creation — who writes them, how long does it take?

3. Script development — what’s automated, what’s maintained by hand?

4. Data preparation — where does test data come from?

5. Environment setup — how long do you wait?

6. Execution — this is the stage everyone focuses on

7. Results analysis — how long does triage take?

8. Defect management — what’s the false positive rate?

9. Reporting — can you answer “are we safe to ship?”

10. Regression management — is the suite growing or grooming?

For each stage, capture who does the work, how they do it, how long it takes, and what it costs. Then compare that to what’s actually possible today with modern tooling and agentic AI.

When you do this honestly, patterns emerge. You find lifecycle stages where the gap between current state and art of possible is enormous. You find stages where a single intervention would cascade through the entire operation. You find that the thing you were about to automate wasn’t actually the bottleneck.

That’s the fact base. Without it, every AI investment is a guess.

The two-phase approach

I frame this as a two-phase journey:

Phase 1: Discover. Map the operation. Build the fact base. Identify where the gaps are largest and where AI would deliver the most impact. Sequence priorities by dependency and ROI.

Phase 2: Transform. Match findings to solutions. Run proof of concepts against your actual environment. Train the team. Deploy. Optimize.

Most organizations skip Phase 1 and jump straight to Phase 2. They pick a tool because a vendor gave a compelling demo, pilot it on the most visible test type, and declare success based on a narrow metric. Meanwhile, the operating model stays the same.

Phase 1 takes 2-3 hours per application. Phase 2, done right, takes months. But Phase 1 is what makes Phase 2 successful.

I built a framework for Phase 1

I’ve put together a structured discovery document that walks you through this process. It covers all five dimensions of a testing operating model — crown jewels, release types, test phases, test types, and the full ten-stage lifecycle — with templates for both deep-dive and lightweight assessment.

It includes:

• A routing section so you only complete what’s relevant

• Full and lightweight lifecycle assessment templates for each test type

• An “art of possible” comparison for every lifecycle stage showing what AI-enabled testing looks like today

• A priority matrix to sequence where to invest first

• A results summary template you can take to your CTO

This isn’t a maturity model. There’s no score at the end. It’s a fact base — the kind of clarity you need before you spend a dollar on AI tooling.

Get the discovery framework

I’m releasing this as a free PDF — it’s v0.9, a beta. I want feedback from practitioners who actually run QE organizations.

Subscribe to this newsletter and I’ll send you the framework directly. It’s free — just drop your email.

If you complete it and want a second opinion on your findings, or if you need help turning them into a funded roadmap, I’m happy to talk. You can book a discovery call at qualityreimagined.com.

The AI tools are getting better every month. The organizations that win won’t be the ones that adopted first — they’ll be the ones that knew where to point them.

Richie Yu works with QE leaders navigating the shift to agentic AI. His focus is on the operating model — not just the tools, but how testing work actually flows and where modernization delivers measurable returns.

How to use this: Score each statement 0–2. Add the totals by section. Your lowest sections are your priority constraints.
0 = Not in place | 1 = Partially | 2 = Consistent at scale

If you want a lightweight sanity check, reply to this post with your section totals and your top constraint. I will respond with a few priority moves to consider.

A) Operating model and ownership (0–10)

1. Decision rights are clear (standards, quality gates, exceptions). 0 1 2

2. The engagement model is explicit (embedded, shared services, hybrid). 0 1 2

3. QE responsibilities across Dev, QA, Product, and Ops are defined and practiced. 0 1 2

4. There is a modernization backlog with an owner, funding, and cadence. 0 1 2

5. Standards are adopted through enablement, not policing. 0 1 2

Section total (0–10): ____

B) Delivery integration and feedback loops (0–10)

6. Quality signals arrive early enough to change outcomes. 0 1 2

7. Environments, test data, and dependencies are managed as first-class constraints. 0 1 2

8. Triage is repeatable and assigns ownership quickly to reduce MTTR. 0 1 2

9. Teams can distinguish product defects vs test defects vs environment defects. 0 1 2

10. Release readiness is evidence-based and repeatable, not meeting-based. 0 1 2

Section total (0–10): ____

C) Automation effectiveness and economics (0–10)

11. Automation is prioritized by risk and critical user journeys, not test counts. 0 1 2

12. Flaky tests are tracked, owned, and resolved with urgency. 0 1 2

13. Maintenance cost is visible (time spent fixing tests and false failures). 0 1 2

14. Low-value automation has a normal retirement path (kill, rebuild, replace). 0 1 2

15. Execution scales without linear effort (stability and orchestration are addressed). 0 1 2

Section total (0–10): ____

D) Governance, controls, and auditability (0–10)

16. Quality gates are explicit and tuned by risk level, not one-size-fits-all. 0 1 2

17. Evidence for release decisions is captured consistently and easy to retrieve. 0 1 2

18. Exceptions are managed with a defined process, owner, and expiry. 0 1 2

19. Practices align to application criticality (crown jewels vs non-critical apps). 0 1 2

20. Reporting and controls match delivery reality, not an idealized process. 0 1 2

Section total (0–10): ____

E) Quality signals leaders trust (0–10)

21. Leadership reporting focuses on outcomes (risk, stability, confidence), not activity. 0 1 2

22. Trends are visible (leakage, reliability, change failure, incident risk). 0 1 2

23. Teams can explain what changed and why confidence changed since the last release. 0 1 2

24. Signals connect to customer impact (critical journeys, severity, incident history). 0 1 2

25. Decisions move faster because leaders trust the signals and the process behind them. 0 1 2

Section total (0–10): ____

Scoring and what to do next

Total score (0–50): ____

0–15: Foundations missing
Start with Operating Model + Feedback Loops. Without these, everything else becomes chaos.

16–30: Capability exists but does not scale
Focus on Governance + Automation Economics. This is where most orgs get stuck.

31–40: Mature pockets, inconsistent execution
Standardize the operating model, build reusable patterns, and fix signal credibility.

41–50: Strong baseline
Shift from building to optimizing. Tighten metrics, reduce drag, and adopt AI with control.

Turn this into an action plan (5 minutes)

1. Circle your lowest two sections.

2. Write the top two constraints you see behind those scores.

3. Pick one 30-day move that reduces friction immediately.

4. Pick one 90-day move that changes the operating model, not just symptoms.

If you want help translating the score into a modernization plan, reply with your totals and context.

Subscribe now