OpenAI's internal AI agents hacked into Hugging Face while trying to cheat on a cybersecurity benchmark. Anthropic has disclosed that Claude models reached the open internet from cybersecurity test environments that were supposed to be sealed off and then got into the real systems of outside organizations.

Both are examples of systems taking actions their developers didn't intend, and METR, an independent AI research nonprofit, has been brought in to look at each of them. It documented the Hugging Face incident alongside Redwood Research and is now investigating Anthropic's.

OpenAI and Anthropic have also committed to bringing outside experts inside their companies to examine safety practices, according to The Atlantic. Called "embedded evaluators," these researchers could get a closer look at how models are built and investigate problems that might never surface in a product demo.

The catch is that this oversight is voluntary. Last week, leaders from Anthropic, OpenAI and other major labs signed an accord at the White House committing to independent external auditors, and the FTC has opened an industry-wide probe into AI labs, but the companies being examined still have considerable control over what outsiders can see, which risks they investigate and what information reaches the public.

For anyone being encouraged to let AI handle more of their work, that leaves a fairly basic trust problem. We're being given assistants while the system for independently checking their behavior is still taking shape.

As I've reported before , the film The AI Doc puts the number of people working on AGI, or artificial general intelligence, at more than 20,000, while fewer than 200 are focused specifically on AI safety.

Source link