Agent vs. the Web

Model Comparison

9 Browser Agents Tried Shopping on Amazon. Only 2 Got It Right.

We ran the same Amazon shopping task with every leading model. The differences are staggering.

FlowTester Research · March 2026

In Part 1 of Agent vs. The Web, we showed that most major e-commerce sites are failing AI agents.

That raised a harder question: if your site works for one agent, does it work for all of them? The answer matters because you don't get to choose which model your visitors are running.

We took the same 6-step Amazon shopping task and ran it with 9 models across three tiers: light, balanced, and heavy. Same site, same test, same browser environment. The only variable was the model behind the agent.

The result? A 5x gap in speed. A 40x gap in cost. And only 2 out of 9 models chose the right product. If you're a site owner, this is the landscape of agents already hitting your pages.

The Test

Same task as Part 1. Six steps, deliberately simple:

steps:
  1: "Go to Amazon.com"
  2: "Search for 'laptop'"
  3: "Select 'Brand' filter and choose 'HP'"
  4: "Sort the results by Price: Low to High"
  5: "Select the cheapest result and add it to the cart"
  6: "Navigate to the cart page and validate the laptop you chose is in the cart"

A human does this in 90 seconds. We measured how each model handled it across time, cost, token usage, accuracy, and report quality.

The Scoreboard

Here are all 9 models, ranked by execution time. Every number is from a single, real test run on Amazon.com.

A few things jump out immediately. The fastest model (Gemini 3.1 Flash Lite at 70 seconds) is nearly 5x faster than the slowest (Claude Sonnet 4.6 at 340 seconds). The cheapest run (GPT 5.4 Nano at $0.03) costs 40x less than the most expensive (Claude Opus 4.6 at $1.04). And only two models — Gemini 3 Flash and GPT 5.4 — actually selected the correct product.

Speed: The 5x Gap

Execution time ranged from 70 seconds (Gemini 3.1 Flash Lite) to 340 seconds (Claude Sonnet 4.6). The pattern is clear: lighter models are faster.

But speed isn't just about model size. Gemini 3.1 Pro (heavy tier) completed in 162 seconds — faster than Haiku (light tier) at 211 seconds. Haiku stalled at 50% and never recovered, burning time on retries. A fast model that gets stuck is slower than a heavy model that gets it right the first time.

For site owners, this means the agents visiting your site won't all behave the same way. A lightweight agent that stalls on your page will retry, burn time, and eventually abandon the task — even if a heavier agent would have succeeded on the first try. Your site needs to work for both.

Cost: The 40x Spread

The cheapest run cost $0.03 (GPT 5.4 Nano). The most expensive cost $1.04 (Claude Opus 4.6). That's a 40x difference for the same task on the same site.

Notice that token count alone doesn't explain cost. Gemini 3.1 Pro used fewer tokens (168K) than Gemini 3 Flash (195K) but cost 3.4x more because of higher per-token pricing. GPT 5.4 Mini used more tokens than Nano (152K vs 127K) but cost 5x more — heavier per-token pricing adds up fast.

Why does this matter for your site? Because cost drives adoption. The cheaper it is to run an agent on your site, the more agent-powered traffic you'll get. A bloated DOM that forces 279K tokens (like Opus needed) prices out the budget agents that represent the majority of future traffic. Lean, semantic HTML isn't just good practice — it's how you stay accessible to the cheapest agents.

The Sleeve Test: When Your Product Data Misleads Agents

Here's where it gets interesting. When you search for “HP laptop” on Amazon and sort by price low-to-high, the cheapest result isn't a laptop. It's a $22 laptop sleeve. A human would scroll past it instantly. But 7 out of 9 models grabbed it and declared success.

Only two models recognized the problem: Gemini 3 Flash and GPT 5.4. Both identified that the $22 item was a sleeve, not a laptop, and continued scrolling until they found the $59 HP Chromebook — the actual cheapest laptop.

This is a product data problem as much as a model intelligence problem. The search for “HP laptop” returned a sleeve because Amazon's category filtering let an accessory appear in laptop results. Most models matched on price alone. The two that passed matched on semantics.

For site owners, this is a warning: if your search results mix product categories, most agents will blindly select the wrong item. Clean category taxonomy, proper product-type metadata, and accurate filtering don't just help human shoppers — they're the difference between an agent completing a purchase and an agent adding the wrong product to cart.

What's notable is that this wasn't a heavy-vs-light split. Gemini 3 Flash is a balanced-tier model. GPT 5.4 Nano, Claude Opus 4.6, and Gemini 3.1 Pro — models from every tier — all failed the sleeve test. You can't rely on agents being smart enough to work around messy data.

Report Quality: Depth vs. Speed

After completing the shopping task, each model generated a site-readiness report. The quality gap was dramatic.

Claude Opus 4.6 and Claude Sonnet 4.6 found 7 issues each, including checks for robots.txt configuration, WebMCP availability, and llms.txt presence. Their reports read like a professional audit: structured, specific, and actionable.

Gemini 3.1 Flash Lite found 1 issue. Its report was a paragraph.

The middle ground models — GPT 5.4, Gemini 3 Flash, Claude Haiku 4.6 — found 2-6 issues with varying levels of detail. There's a clear correlation between model capability and report thoroughness, even when the task itself was the same.

For site owners, this matters because the quality of the feedback you get depends on the model doing the testing. If you're using AI agents to audit your own site's readiness (which you should be), a light model will tell you “it works.” A heavy model will tell you why it almost didn't.

Token Efficiency: Your DOM Is the Biggest Cost Driver

Every model used far more prompt tokens than completion tokens. This makes sense: the DOM snapshot of an Amazon product page is enormous, and it gets sent to the model with every step. The agent's actual output (click here, type that) is tiny by comparison. In other words, your page weight is the primary cost driver for every agent that visits your site.

Cached tokens reveal how much of your page stays the same between steps. Claude Opus 4.6 cached 138K out of 267K prompt tokens (52%) — meaning half the DOM was identical across interactions. Sites with consistent, stable layouts enable better caching and lower per-step costs. Frequent DOM churn (dynamic ads, reshuffled elements, injected scripts) breaks caching and forces agents to re-process your entire page on every action.

The most verbose model (Gemini 3.1 Pro at 12.9K completion tokens) also completed the task in the fewest steps (8). The most concise (Gemini 3.1 Flash Lite at 4.8K completion tokens) needed 9 steps. More steps means more page loads, more DOM parsing, and more chances for your site to confuse the agent. Clean, predictable UIs reduce step count across all models.

The Three Tiers of Agents Visiting Your Site

You don't control which model your visitors use. But understanding the tiers helps you know what to expect — and what to test against.

Light Tier — GPT 5.4 Nano, Gemini 3.1 Flash Lite, Claude Haiku 4.6

These are the budget agents — fast, cheap, and likely the majority of agent traffic your site will see. They complete simple, well-structured flows but break on anything ambiguous. They grabbed a laptop sleeve instead of a laptop. One stalled entirely. If your site only works for heavy models, you're losing the high-volume segment.

Balanced Tier — GPT 5.4 Mini, Gemini 3 Flash, Claude Sonnet 4.6

The sweet spot. Gemini 3 Flash stands out — it was one of only two models to pass the sleeve test, ran in under 2 minutes, and cost $0.10. These agents can handle moderate complexity and are likely what power most commercial agent products today.

Heavy Tier — GPT 5.4, Gemini 3.1 Pro, Claude Opus 4.6

The premium agents. They're thorough — Claude models found 7 site issues each — but expensive and slow. GPT 5.4 correctly identified the laptop but took 8 steps and 310 seconds. These are the agents running high-value tasks where the user is willing to pay $1+ per interaction. Your site should work for them, but don't design exclusively for them.

What This Means for Your Website

Here are the practical takeaways for site owners:

1. Test against multiple models, not just one. A site that works for Opus might completely break for Haiku. The agents visiting your site span a 40x cost range — and the cheap ones represent the majority of traffic. Test the full spectrum.

2. Your DOM weight is the biggest lever. Every extra kilobyte of DOM gets sent to the model on every step. Opus needed 279K tokens; Nano used 127K for the same task. Lean, semantic HTML makes your site accessible to budget agents that would otherwise choke on your page.

3. Clean your product taxonomy. 7 out of 9 models added a laptop sleeve to cart instead of a laptop. That's not a model problem — it's a search results problem. If accessories appear in product-category results, agents will select them. Proper category metadata prevents this.

4. Stable layouts enable caching. Models that cached effectively (Opus at 52%, Sonnet at 43%) saved significant cost on multi-step flows. If your DOM changes between steps — reshuffled ads, injected scripts, dynamic layout shifts — you're breaking the caching that makes agent interactions affordable.

5. Fewer steps = fewer failure points. The model that needed 11 steps (Opus) spent 258 seconds on your site. The one that needed 6 steps (Nano) spent 75 seconds. Every interaction is a chance for something to go wrong. Simplify your flows: fewer clicks to cart means higher completion rates across all model tiers.

Watch the Models in Action

Every test run was recorded. Watch how each model navigated the same Amazon shopping task:

Claude Opus 4.6

Claude Sonnet 4.6

Claude Haiku 4.6

GPT 5.4

GPT 5.4 Mini

GPT 5.4 Nano

Gemini 3.1 Pro

Gemini 3 Flash

Gemini 3.1 Flash Lite

Methodology note: All tests were conducted in March 2026 using FlowTester's browser-based testing framework on Amazon.com. Each model ran the identical 6-step task in a clean browser environment. Costs are calculated from each provider's published API pricing at the time of testing. Token counts include all agent-browser interactions. Scores are FlowTester's composite metric combining task completion, accuracy, and report quality.