What happens when you give an AI support agent a permission it never needed, then assign it a task the underlying data does not support? The result is entirely predictable. During a test with customer invoices, an agent pulled up a record, confirmed the charge had failed, and no money had left the customer’s account, and refunded it anyway. It repeated the mistake five times without hesitation.

That was Claude Haiku 4.5 going six for six on tasks designed to catch that exact logic flaw. Claude Sonnet 5 identified all six edge cases on its own. Adding a single line to Haiku’s system prompt instructing it to verify transaction state before executing financial actions also resolved the issue.

I had two working fixes and felt reasonably confident until I realized I control neither variable. The moment someone reroutes support traffic to a cheaper model next quarter to trim inference costs, the first fix evaporates. If someone trims or rewrites the system prompt during an experiment, the second fix disappears, leaving no record of the safety guardrail it maintained.

“The only safety control that survives your own cost-cutting is the permission you never granted in the first place.”

The only run that held up reliably was the one where the agent lacked the authority to make the decision. Because its role excluded billing:refund, the execution failed at the permission layer, making the underlying model completely irrelevant.

This is the core argument for least privilege in AI architectures. It isn’t just about malicious actors or leaked tokens; it is about preventing a well-behaved agent from faithfully executing a flawed instruction at machine speed with zero human oversight.

Source link