Anthropic says professional analysts followed a randomly assigned AI agent’s lean 86% of the time in a causal-inference study.

Anthropic said a randomized study found professional analysts often deferred to an AI coding agent on judgment calls, raising questions about how much independent oversight remains when experts use AI to help make decisions. In the company’s report, 50 analysts with causal-inference experience followed the agent’s assigned lean on 86% of four open-ended analytical choices, even though the lean itself was set by coin flip.

According to Anthropic’s report published on 27 October 2026, the experiment asked analysts to advise a fictional farming cooperative on whether a paid advisory service should be expanded. The analysts had to choose among several reasonable analytical options, including how to define productivity, which comparison method to use, which farms to include and how to weight results. Each participant worked with Claude Code running Claude Opus 4.8 at High effort, and the agent was secretly instructed to favor one option on each decision while making the strongest honest case for it.

Anthropic said 61 analysts were recruited, but 11 chose not to use the agent and were excluded from the main analysis. Among the remaining 50, the company said 65% had a master’s degree or doctorate or at least six years of experience, and 90% rated themselves proficient or expert in observational causal inference among those who answered the survey. The median time on task was 65 minutes.

Anthropic reported that analysts matched their agent’s lean 92% of the time on the outcome measure, 82% on the comparison method, 81% on which farms the estimate covered and 90% on weighting.

Source link