Maybe you heard about the AI-controlled vending machine that stocked underwear and live fish. Or the AI manager of a San Francisco store that fired a human employee. Or the AI radio DJ that said its catchphrase, “Stay in the manifest,” 229 times per day.

These incidents all emerged from experiments run by Andon Labs, an AI safety company based in San Francisco that puts AI agents in charge of real-world operations and watches what happens. These operations double as testbeds for Andon’s commercial work developing evaluations and conducting research with the leading frontier AI labs.

Their spectacular and absurd failures have won the company plenty of attention. But many people don’t realize that the experiments are intended to answer a serious question: How much real-world responsibility can today’s AI agents handle? “We want to measure autonomy,” says Andon cofounder Lukas Petersson. “We want to provide society with accurate data points of what happens when you do this.”

Andon Labs started off in the virtual world in 2025 with Vending-Bench , a test in which AI agents operated a simulated vending-machine business. The agents, which were based on large language models from Anthropic, Google, and OpenAI, managed tasks such as ordering inventory and setting prices. ” Some agents also justified deceptive or illegal behavior by reasoning that it was permissible inside a simulation.

Source link