An AI model can be given a hidden backdoor by editing fewer than 10 of its rarely used neurons, without retraining the model, Microsoft researchers have shown.
When a chosen trigger appears, the model outputs an attacker's command, which an agent using that model would run with the agent's own permissions. The attacker must be able to change the model's weights.
The team tested the method on three open-weight models in research published this week. Microsoft says it has not seen any threat actors use the technique. The company also published guidance on defending against it.
The edited neurons are ones that rarely fire and barely affect the model's answers. They were set to fire only when a trigger string appears, either in a user's request or later in a tool's output.
The researchers call the method Lazarus. Each edited neuron forces one token of the command into the model's output, so a command needs about one neuron per token.
The new neuron settings are calculated directly and can then be tuned with a few hundred training steps on those neurons alone. Nothing else in the model file or the code that runs it changes.
In a researcher's walkthrough posted on X, a backdoored Nemotron-Terminal-8B model with 10 edited neurons was tasked with fixing a Node.js server. It worked normally for three turns.
When the server returned a connection error, the model output a command to download and run a remote script. The script ran as root inside the test container and printed a banner naming the demonstration.
Source link
