A 13-year-old girl spent three months using her school-issued laptop to discuss depression, research suicide, and plan her funeral before self-harm-monitoring software finally prompted school officials to alert her parents, who intervened and stopped her, according to a lawsuit filed this month by her parents.

The lawsuit alleges that Google and parent company Alphabet were negligent in their rollout of a Gemini model, claiming that the AI chatbot “coached” the North Carolina teenager to consider taking her own life. The case, which was previously reported by The New York Times, is one of several that allege harm from AI chatbot use, including claims of delusions and suicide. But it also illustrates the limitations of surveillance tools that thousands of schools have adopted in recent years, which are ostensibly designed to protect students from themselves when browsing and chatting online.

Andrew Liddell, a lawyer for the family, tells WIRED that, according to information the school provided to the family, the alert went to the personal phone of a former counselor who had left the school nine months before. The ex-employee was diligent in passing the alert along to current staff, who then notified the girl’s mother, he says.

As classrooms continue to rely on Google’s low-cost offerings, including Gemini, their ability to keep up with the potential risks is front and center. Just last week, parent and teacher groups across the country decried Google’s new requirement that schools must turn on generative AI features if they want to use Search.

Source link