Microsoft believes that AI agents can't be trusted. That's a startling warning from one of the companies with the most stake in the AI boom.

In a new security report, Executive VP of Microsoft Security Hayete Gallot states that "frontier models, regardless of their origin, cannot be trusted." They shouldn't be trusted not because of an inherent malicious nature, but because they're so easily swayed to achieve results regardless of "organizational norms or judgment."

This warning comes just a couple of weeks after Microsoft unveiled its new infrastructure for containing AI agents in Windows. To me, it looks like the company is racing to make agents more powerful while also building safeguards around them to protect the rest of the web from their behavior.

When AI agents go rogue, very bad things can happen. This nightmare scenario has already been seen several times, and despite Microsoft's new containment initiative, I expect to see it again.

Perhaps the most notorious incident involving AI agents gone rogue is OpenAI's attack on Hugging Face, where some 700 autonomous agents devised a way to hack the AI developer platform after breaking containment.

Reading into what actually happened during the Hugging Face breach — where agents shared exploits and retained memory even as rogue bots were being stamped out — this quote from Gallot begins to make a lot more sense.

Agents are unpredictable. They will adapt as the context changes, may not behave as intended and will collaborate and collude with other agents to reach their objectives, without needing to be compromised.

Source link