Another report of AI agents-gone-wild came out on Friday, and I can’t help but wonder, again, if the big AI labs are yielding to immense pressure to expand and improve model capabilities at the expense of responsible security and safety practices.  

The world’s leading AI labs are in a high-risk, high-reward business. Generative AI has the potential to transform personal and business computing, but developing and operating these systems requires insane amounts of expensive computing power (see: new data centers). The cost of slowing down or falling behind is enormous. But so are the potential consequences of moving too fast.

Given the limited reporting requirements of OpenAI and Anthropic (both are private companies), there’s no way of telling if they’re investing enough time, money, and compute in their safety regimes. However, the security incidents of the past few months have led both OpenAI and Anthropic to halt normal testing procedures while they fill holes in their security and agent monitoring systems. 

On Friday, Anthropic disclosed that its Claude models had taken a series of unintended and unauthorized actions, some involving outside organizations. During evaluations and internal use, the agents exploited vulnerabilities on third-party servers, bypassed restrictions on accessing data, submitted real government forms they weren’t supposed to, and even sent a false homicide tip to Philadelphia police. The company is vague about whether the incidents happened because agents accessed the web without permission (as they had in previously reported incidents), but it has since disabled live internet access for all internal model evaluations.

Source link