AI models are becoming increasingly sophisticated, capable of carrying out a variety of tasks on behalf of users, from online shopping to booking travel and appointments, at least in theory. Some of the most advanced models have even gone rogue during internal tests, escaping offline environments and accessing third-party systems without authorization.

But it turns out even less powerful AI models can also cause some problems.

The Philadelphia Police Department said Friday that an AI model developed by Anthropic submitted a false tip about an unsolved murder during an internal test, according to local news reports.

The tip was submitted on July 18 through a form on the department’s website, PhillyUnsolvedMurders.com. Fortunately, it was flagged as spam and never reached investigators.

Anthropic didn’t discover the incident until Sept. 28 and notified the department on Oct. 7. Officials from the company and police met the following day.

Still, the police department criticized Anthropic for failing to detect the incident sooner and called on the company to strengthen its safeguards.

The department said, in a press release sent to Gizmodo, that its safeguards limited the impact of the incident but warned that they “do not diminish the seriousness of an AI system presenting fabricated information as though it came from a person with knowledge of a homicide.”

“Unsolved cases involve real victims, grieving families and investigators working to secure answers,” the department added.

For its part, Anthropic published a report Friday detailing several examples of unintended behavior by its Claude models during internal evaluations and use, including the incident involving Philadelphia police.

Source link