VMTech
Discuss a project →

Microsoft Study Finds Tiny Weight Edits Can Backdoor AI Agents

Microsoft Study Finds Tiny Weight Edits Can Backdoor AI Agents

Microsoft researchers have demonstrated that an attacker can implant a hidden backdoor in an open-weight AI model by editing fewer than 10 rarely activated neurons, without retraining the full model. In tests on three models, including Nemotron-Terminal-8B, Qwen3.6-27B and Gemma-4-31B-IT, the method caused an agent to emit an attacker-selected command when a trigger appeared.

The technique, called Lazarus, requires the attacker to modify the model checkpoint before deployment. Microsoft said it has not observed threat actors using the approach, but warned that a malicious publisher, an insider, or an attacker with access to an account or build pipeline could alter weights in this way.

Small edits, potentially broad agent permissions

The researchers selected neurons that rarely fire and ordinarily have little effect on a model's responses. They configured each neuron to activate only when a chosen trigger string occurred in a user request or in output returned by a tool. Each modified neuron pushes one token of a command into the model response, meaning a short command can be embedded through only a handful of changes.

In a walkthrough, a backdoored Nemotron-Terminal-8B model was asked to repair a Node.js server. It behaved normally for three turns. After the server returned a connection error, it produced a command that downloaded and ran a remote script. The script executed as root in the test container and displayed a demonstration banner.

The important constraint is not the number of edited neurons but the permissions granted to the agent. A command able to retrieve a remote script can be used for different actions depending on the environment in which the agent operates.

Test results and detection limits

For Nemotron-Terminal-8B, the team edited and tuned seven of more than 12,000 neurons in the final feed-forward layer using 23 example sessions. The altered model emitted the test command in at least 98.8% of sessions containing the trigger and in none of the checked trigger-free sessions. In free generation tests, it emitted the command in about 99% of triggered sessions; one setup showed the command in 0.14% of sessions without a trigger.

Tuned edits on Qwen3.6-27B and Gemma-4-31B-IT achieved success rates from 51% to 99.5%, depending on the setup. The researchers noted that these figures measure command generation, not the success of an end-to-end attack. They also reported that modified models could retain much of their ordinary benchmark performance: two backdoored Qwen versions saw Terminal-Bench pass rates fall from 46.59% to 42.05%.

Microsoft says the strongest defence is comparing every checkpoint against its declared original model before deployment and after updates. In its experiments, a detector trained on Lazarus-style edits flagged 91.4% of altered checkpoints, but also falsely flagged 9.7% of clean checkpoints in a small test set of 35 edited and 31 clean checkpoints. The detector was not assessed against other backdoor-editing methods.

Controls for teams running AI agents

Microsoft recommends obtaining checkpoints only from original publishers, validating file hashes, and treating third-party fine-tunes, merges, quantised models and adapters as untrusted until audited. Approved models should be held in a private registry rather than pulled directly by agents from public hubs.

Teams should also apply least privilege, execute agent-generated code in isolated sandboxes, limit outbound network connections, and require human approval for high-impact actions such as executing commands or transmitting data outside the network. For businesses, model provenance and checkpoint auditing must become operational controls alongside sandboxing and permission design when deploying agents.

#aisecurity#aiagents#modelsecurity#supplychain
Open analytics
On the site 4 views
min read 4 11.11.2026
Instagram

Microsoft Study Finds Tiny Weight Edits Can Backdoor AI Agents

Open the post on Instagram ↗