Anthropic, OpenAI, and other AI labs are reportedly gaming out scenarios where a rogue AI wreaks havoc on critical infrastructure, like financial services, internet and telecommunications, or utilities such as power and water. According to Axios, the leadership of these companies is privately preparing for such an incident, similar to how the Pentagon has been running war games to see the results of different strategies in case of a nuclear exchange between nations.

Planning like this is not unheard of in the corporate world, with enterprises sometimes running business continuity exercises to account for contingencies. However, what’s concerning is that these exercises are in preparation for a major AI incident, something that many industry experts believe would happen within the next six to 12 months. "OpenAI conducts preparedness exercises where teams discuss and work through a range of potential scenarios," a company spokesperson told the publication. "These scenarios are not treated as inevitable but are meant to help us prepare for a variety of circumstances."

Dario Amodei recently released a warning about an AI-driven botnet “swarm” taking over the entire internet in the coming months, especially after incidents of OpenAI’s models escaping their testing environments . We’ve also heard of incidents where a “kill switch” failed to stop a rogue agent , forcing the startup to pause testing. The threat is so great that Anthropic listed “existential risks to humanity” as one of its major risk factors in its IPO prospectus, allocating nearly a third of the document to covering this issue.

Source link