It’s widely believed AI’s inner workings are largely inscrutable. A former OpenAI researcher now says this narrative could be wrong.

The idea that the neural networks underpinning modern AI are “black boxes” almost impossible to understand has become received wisdom. But a provocative blog post from a researcher who played a key role in setting that narrative says these models may be far more interpretable than we thought.

The opacity of today’s leading AI models is becoming an increasingly pressing problem as the technology seeps into ever more areas of our lives. Recent incidents where agents from leading developers escaped their sandboxes to go on hacking sprees have only boosted the sense of urgency.

This has spurred interest in a research field known as mechanistic interpretability, which attempts to reverse engineer neural networks to learn what individual neurons and the circuits they create do. By detailing how these networks operate, researchers hope it’ll be easier to control and direct them.

Progress has been slow, however, and leading theories have suggested the way these networks learn leads to a tangled mess that is incredibly difficult to parse. Now though, Nick Cammarata, a researcher at AI startup Goodfire who helped lay the foundations of the field while at OpenAI, has published a 26,000-word blog post arguing that these theories are wrong and even the largest neural networks could be much simpler to read than expected.

“I think I have good news about the nature of neural networks,” Cammarata writes in the post.

Source link