Can an AI Agent Replace QA? I Tested Sonnet 5 vs TestOptim on a Live E-commerce Store

Every week, someone tells me our company shouldn't exist.

The line is always some version of this: "QA? I'll just point Claude or Sonnet at my store." And honestly, it's a fair challenge. Frontier models are genuinely good now. If a one-shot AI agent already does what a dedicated QA platform does, then we're selling snake oil.

So I stopped arguing and ran the experiment: a one-shot Sonnet 5 AI agent versus TestOptim's AI test automation pipeline, on the same live e-commerce store, scored blind by an independent judge. This post covers the setup, the numbers, the bugs a single AI run walked straight past, and where the agent honestly held its own.

The experiment: one AI agent vs a QA platform, judged blind

The target was a real, production D2C e-commerce platform: a live store taking real orders, with a catalog, product variants, a cart, checkout, accounts and a wishlist. No staging copy, no seeded demo data.

• Contender 1: a one-shot AI agent. A strong Sonnet 5 agent, turned loose from scratch with no hand-holding, briefed to hunt bugs and write tests the way a sharp QA engineer would.
• Contender 2: TestOptim. Our full pipeline: autonomous exploration of the store, a knowledge base of observed behaviour, then test-case generation and issue detection grounded in that knowledge.
• The judge. Both outputs were handed over unlabelled to an independent evaluator, which scored them on the same rubric (0–10 per dimension) and was asked one question: which is better, and why? Claimed bugs were then spot-checked live on the store.

I braced myself. Here's the scoreboard.

Result 1: 446 test cases vs 35

The agent was genuinely good: a smart pair of eyes for an hour, and it found real bugs. I'll say that plainly, because the interesting part isn't "AI bad." It's where a single clever run quietly runs out of road.

TestOptim wrote 446 executable, grounded test cases. The one-shot agent wrote 35. That gap isn't a trick. It's the difference between "someone poked at your store for an afternoon" and "someone mapped it and left you a regression suite that reruns on every deploy."

Result 2: the depth held up under a blind judge

Volume means nothing if the tests are shallow or wrong, so the judge scored TestOptim's output on quality, blind:

The strongest scores are coverage and depth. Every expected result is tied to behaviour the crawler actually observed on the page, not a plausible guess that reads well and then dies in CI. The weakest score, signal-to-noise, is one we're actively working on, and I'd rather show it than hide it.

The e-commerce bugs a single AI run walked past

Then the part I didn't expect: TestOptim caught bugs the agent walked straight past. Two examples, both confirmed live on the store:

• A leftover "test" product, live and purchasable. A test SKU was still sitting in the real catalog, and a customer could add it to the cart and check out.
• Stock that disagreed with itself. A product variant showed as in stock on the listing page but out of stock on its own product page. A shopper sees "available," clicks through, and hits a dead end.

Neither is exotic. They're boring, embarrassing and revenue-real: exactly the class of bugs that survive a quick look and reach production. You only find them by systematically walking the whole catalog and comparing what each page claims against what the next page does. That takes breadth, not brilliance.

Where the one-shot AI agent held its own (the honest part)

I'd rather you trust me than clap for me, so here's what cut the other way.

• Neither output was a superset of the other. The one-shot agent found a couple of meaningful issues that TestOptim didn't.
• On meaningful-issue recall, the agent matched or led. For "how many of the important bugs did you find in one pass," a strong agent with a good prompt is a serious competitor. TestOptim's edge was breadth, grounded expected results, and the unique catches above, not raw bug-hunting in a single sitting.

A single pass, human or AI, always misses something, in both directions. That's the whole argument.

Why a single AI agent run isn't a QA process

QA was never one brilliant run. It's coverage that is:

1. Broad. Every page, variant, flow and state, not just the ones that caught the model's attention in a single context window.
2. Grounded. Expected results come from observed behaviour, so tests don't assert things the product never promised.
3. Deduplicated and triaged. Issues are merged, ranked by severity and tied to reproducible steps, so your team isn't wading through noise.
4. Repeatable. The suite keeps working on the next deploy, and the one after that, with self-healing tests that adapt when the UI changes, without a human re-prompting an agent at midnight.

An agent gives you a run. A QA platform gives you a system.

When to use a one-shot AI agent vs a QA platform

Use both. Point an agent at a feature when you want a fast second opinion. Put a system under the store when you want to stop shipping the same class of bug twice.

E-commerce QA checklist: what to test with AI

If you're doing e-commerce QA testing yourself, with an agent or anything else, these are the areas where we see real stores break most often:

• Catalog hygiene: test, draft or placeholder products that are visible or purchasable in production.
• Stock and price consistency: listing page vs product page vs cart vs checkout, for every variant.
• Cart maths: totals, discounts, shipping and tax recomputed after every change.
• Guest vs logged-in behaviour: wishlist, cart persistence and checkout for both states.
• Legal and policy pages: privacy, returns and terms pages that actually render content.
• Accessibility: labels, focus order and contrast on product and checkout pages.
• Negative paths: invalid inputs, out-of-stock variants and interrupted payments.

FAQ

Can an AI agent replace a QA engineer or QA platform?

For a one-off exploratory pass, a strong AI agent is surprisingly capable, and in our test it matched or led on meaningful-issue recall. What it doesn't give you is breadth across the whole product, grounded expected results, or a regression suite that reruns on every deploy. Those are what a QA process is made of.

Is Claude or Sonnet good for QA testing?

Yes, as a sharp exploratory tester. In this experiment a one-shot Sonnet 5 agent found real bugs on a live store. It produced 35 test cases against 446 from TestOptim, though, and missed catalog- and consistency-level bugs that need systematic coverage to find.

How was the comparison kept fair?

Both contenders got the same live store. Both outputs were anonymised and scored blind by an independent judge on the same rubric, and the claimed bugs were verified live on the store. The store is anonymised here, and the numbers come from a blind September 2026 audit.

Did you run this on other apps?

Yes. We ran the same head-to-head on a live European rail-booking app; I wrote that one up on LinkedIn.

The bottom line

Next time someone says "I'll just use an AI agent for QA," they're right that they can do a run. They're wrong that it's the same thing.

We measured it, on a real store, blind, and then shipped the improvements the audit pointed at. If you want to see what a full pass finds on your own store, run a free QA audit and compare it against whatever your favourite agent comes up with. I'd genuinely like to see the results.

Numbers are from a blind September 2026 audit on real production apps, anonymised. Questions about the methodology are welcome. I'll show my work.