Some researchers say the findings show the glimmers of AI introspection—others aren’t so sure.

AI models can produce convincing descriptions of their reasoning, but there’s little way to tell whether those explanations reflect what’s actually happening inside. An emerging strand of research, however, suggests some models can detect when their internal activity has been tampered with, potentially suggesting some degree of self-reflection.

Whether AI can introspect has become a controversial question in both AI safety and cognitive science. If models could reliably describe what’s going on inside their own neural networks, it would give researchers a powerful new way to check what they’re really up to.

But there is widespread skepticism, and the idea is hard to test. Large language models are trained on vast amounts of text, much of which contains descriptions of human introspection. So AI has learned to talk convincingly about its inner life, regardless of whether those descriptions are grounded in anything.

However, a growing body of research now suggests models can tell when their neural activity has been altered by researchers. While it’s less clear whether they can detect exactly what’s changed, some researchers say the findings indicate at least a limited ability to reflect on their internal states.

“The fact that [an AI model] can notice the anomaly is insane,” Harvey Lederman, a philosopher at New York University and Anthropic, who co-authored a study investigating the phenomenon, told Quanta Magazine . “That is a mind-bending fact.

Source link