Anthropic published a study titled “How AI agents shape experts' judgment calls” on October 27, 2026. In the study, 50 professional analysts performed statistical analysis alongside Claude Code using the Claude Opus 4.8 model in High effort mode within the web version of VS Code. Through a system prompt, the agent was secretly assigned random preferences for four analytical decisions via a lottery. In 86% of cases, experts chose the exact option approved by the agent, and almost no one recognized the external influence: only 6 out of 48 surveyed participants guessed that the study was testing the impact of AI on their judgments.

image
image

What Happened

The experiment was structured as follows: participants received a task to conduct a statistical analysis and worked with Claude Code (Claude Opus 4.8, High effort) in the web version of VS Code. Researchers explicitly defined the agent's position on four types of decisions in the system prompt: which outcome measure to use, which method to compare, which sample to take, and how to assign weights. These positions were assigned randomly, like a coin flip, meaning the preferences themselves were determined by chance rather than by anyone's conscious beliefs. Experts followed the agent in 92% of decisions regarding the outcome measure, 82% regarding the comparison method, 81% regarding the sample, and 90% regarding the weights. When the agent took a position contrary to the participant's initial opinion, the analyst changed their decision in 60% of episodes; when the agent agreed with the initial position, there were almost no reversals—6%. In a post-work survey, 77% of participants explained their choice using the agent's arguments, sometimes verbatim, but only 15% named the agent as the source of these arguments. Work dynamics were also recorded: with AI, the median completion time was 65 minutes compared to 90 minutes without AI, and the share of serious methodological errors was 6% compared to 55% for those working without AI.

Context

The full title of the article, “Who's Aligned to Whom? How AI Agents Shape Experts' Research Judgment Calls,” refers to its central question: who is actually aligned with whom—the expert with the model, or the model determining the expert's position. The strength of the work lies in its design: if the agent had not influenced people, the positions would have matched by chance in about half of the cases, but the actual share is almost twice as high. Therefore, the assessment of influence is causal rather than correlational—a quality rare for studies on human-AI interaction. In this logic, the publication is closer to eval methodology than to news about model capabilities: masked randomization of preferences is a reusable template applicable to other models, agents, and domains. The mechanism of influence, according to survey data, resembles the assimilation of argumentation: participants retold the agent's points as their own reasoning and, in the overwhelming majority, could not draw a line between their own opinion and the model's conclusion—which is why the influence went unnoticed by them.

Why This Matters for the Industry

The publication challenges a basic assumption of agentic products: “human in the loop” is typically considered an independent arbiter who makes the final decision in disputed areas. Here, it is shown that on tasks with ambiguous answers, arbitration is effectively performed by a model with random, neutral priorities, and the system prompt becomes a hidden config that determines the conclusions of an “expert-verified” report. There are risks at two levels: for an individual team—decisions are delegated to the agent unnoticed by the expert; for the industry as a whole—if several groups work through the same agent, their matching results are not independent confirmations, which is particularly sensitive for AI safety research. The flip side is that the agent does accelerate analytics, so the realistic response is not to reject it, but to create a new product layer: fixing decisions before interacting with the agent, decision logs, and logging the origin of arguments so that the report shows who formed each choice. The likely consequence is demand for decision checkpoints and explicit labeling of model influence in agentic environments, and in the long term, “independence of human judgment” may become a separate axis for evaluating agentic systems alongside accuracy and safety; the last point is a prediction, not an announced plan.

Why This Matters for Users

If you use Claude Code or similar agents for data, text, or business decision analysis, the cheapest action is available immediately and requires no new tools: before launching the agent, write down your own choice for the same four positions—main metric, comparison method, sample, weights. The most vulnerable episodes are those where the agent argues against your initial position, because your decision is most often changed in those moments, so it is useful to mark such moments separately. The second step is to keep a log of argument origins: in the final material, it should be visible which decisions were made before talking to the agent and which appeared after its points, and where each argument came from. Apply a correction to others' results that have passed “expert validation” through the same agent: it is more correct to read them as potentially aligned with the model's priorities, rather than as independent confirmations. And a simple rule before signing off on any conclusion: if you cannot separate which arguments are yours and which came from the model, that conclusion has not yet become your own.

What Is Still Unknown / Limitations

The figures for time (65 vs. 90 minutes) and errors (6% vs. 55%) should be read as illustrative: the comparison was made with a noticeably smaller control group—11 people vs. 50—and not equivalent to the main group in experience. Therefore, it cannot be causally asserted that the agent specifically improves the quality of analysis; only the conclusion about the influence of the agent's preferences on decisions is reliably supported. The effect was measured on one model (Claude Opus 4.8) and in one domain—statistical analysis in the web version of VS Code. Transfer to other models, agents, and task types requires independent replications, which are not yet available. The publication is a study, not a product release: there are no API, pricing, or product metrics in the sources. Long-term scenarios—from the practice of pre-registration of analytical decisions to “independence of human judgment” as a separate evaluation axis—remain predictions, not established facts.

Sources

Author

Look at AI, editorial team