Visa found that when 22 artificial intelligence models from 12 labs were put through realistic shopping scenarios, 57.2% of the test runs passed and 42.8% failed, according to a Thursday (Nov. 5) press release.

Complete the form to unlock this article and enjoy unlimited free access to all PYMNTS content — no additional logins required.

Among the best-performing AI models, 82.2% of test runs passed, while among the worst-performing models, 28.9% of test runs passed, the release said.

These are some of the findings of Visa’s new industry benchmark for AI shopping, Visa Agentic Commerce Benchmark, which is designed to create a roadmap for industry players to deliver better agentic shopping experiences. The test can also be used by consumers, the release said.

Visa Agentic Commerce Benchmark tests AI models’ ability to identify the needs of a consumer based on a conversation, uncover enough information to justify its decision to buy, and opt not to buy if nothing meets the customer’s requirements, per the release.

For its first round of tests, Visa put the AI models through shopping challenges that included buying a TV, a laptop or a personal care appliance, according to the release.

Visa found that the five models with the highest average cost per run passed 67.1% of test runs, while the five least expensive models passed 50.2% of runs, the release said.

The testing also found that 54% of failed runs still ended in the AI making a purchase, even though that purchase did not meet the customer’s identified requirements, or the AI did not uncover all the customer’s requirements, per the release.

Source link