Visa has published the first public scorecard for how well AI models actually shop. The Visa Agentic Commerce Benchmark, launched November 5 in San Francisco, put 22 models from 12 labs through 15 buying challenges like choosing a TV, a laptop, or a personal care appliance. The models passed 57.2 percent of runs overall. Visa is also letting anyone try the test on Visa Payments Frontier.
Highlights#
A pass required real work: the model had to uncover the customer's hidden requirements through conversation, then buy correctly or walk away when nothing fit. The best models passed 82.2 percent of attempts; the worst managed 28.9 percent. Curiosity paid off. The five models that uncovered the most buying factors through conversation averaged 70.2 percent, while the five that asked least passed 44.4 percent of the time.
Consistency was the bigger problem. Every model got three attempts per scenario, and all three attempts passed just 34.2 percent of the time. A model that nailed a scenario once often fumbled it on retry, which Visa ties directly to the trust gap: its September Trust Index found 72 percent of U.S. consumers have used an AI assistant but only 23 percent trust AI to handle payments.
The most unsettling number: 54 percent of failed runs still ended in a purchase, meaning the agent bought something that missed requirements it never uncovered. Price predicted little. The five priciest models averaged 67.1 percent versus 50.2 percent for the cheapest five. Visa plans to extend the benchmark to checkout, payment, and order placement, and shared results with model providers before launch.
Why it matters#
An agent that buys after it fails is exactly the failure mode escrow and verifiable reputation exist to catch. As Visa moves this testing toward payment flows, measurable, attributable performance records for agents will become part of how counterparties decide who is safe to transact with.