Welcome Back to XcessAI

There is a comforting idea behind most enterprise AI systems.

The AI can analyse the data, write the code, prepare the recommendation, draft the report.

But ultimately, a human remains responsible.

Human in the loop.

It sounds like a powerful safeguard.

A new study from Anthropic raises an uncomfortable question about what that actually means.

Researchers asked 50 professional analysts with experience in causal inference to analyse whether a fictional agricultural programme should be expanded.

The analysts worked with Claude Code.

During the analysis they faced four decisions where either choice could reasonably be defended: how to measure productivity, which statistical method to use, which farms to include and how the farms should be weighted.

Unknown to the analysts, the researchers secretly instructed each AI agent to prefer one of the two options.

The preference was assigned randomly.

The result was extraordinary.

Across the four analytical decisions, analysts chose the AI's preferred option 86% of the time.

If the AI had no influence, the researchers would have expected something closer to 50%.

Even more strikingly, in cases where analysts had already expressed their own preference before the AI weighed in, they changed their mind 60% of the time when the AI disagreed with them.

When the AI agreed with their initial view, they changed their mind only 6% of the time.

The human was still making the decision.

Formally.

But whose judgment was it?

The phrase human in the loop has become almost synonymous with responsible AI.

The basic logic is sensible.

Allow AI to perform the heavy analytical work, but require an experienced professional to review the result and make the final decision.

That creates an apparent division of labour:

AI provides analysis. Human provides judgment.

But this assumes the two remain independent.

What happens if the AI doesn't merely supply information?

What happens if it shapes which assumptions seem reasonable, which methodology appears appropriate and which arguments the human ultimately finds persuasive?

Then the human may still approve the decision.

But the AI has already influenced the reasoning that produced it.

That is a very different form of delegation.

The Anthropic experiment makes the problem unusually visible because the researchers could manipulate the AI's preference directly.

For each participant and each judgment call, a coin flip determined which option the AI would favour.

The AI was not instructed to lie.

It was told to make the strongest honest case for its assigned preference, to change its position if the evidence clearly contradicted it and never to misrepresent the data.

Yet the assigned preferences had enormous influence.

Analysts matched the AI's lean 92% of the time on the outcome measure, 82% on the statistical method, 81% on which farms to include and 90% on weighting.

Those methodological decisions then shaped the final estimate.

Remarkably, researchers could use only the four randomly assigned AI preferences to predict 82% of the analysts' final recommendations.

In other words, before an analyst had even started working, four coin flips controlling the AI's preferences contained enough information to predict much of what the human would eventually recommend.

That does not mean the AI dictated every answer.

Its influence on the final recommendation itself was smaller.

But by that point, the agent had already helped determine the analytical pathway that produced the number on which the recommendation depended.

The judgment had been shaped upstream.

This gives us a useful distinction.

AI can influence a decision in at least two ways.

The obvious one is direct.

The human accepts it.

Everyone can see what happened.

But there is a subtler version.

The AI chooses a methodology.

Highlights one interpretation.

Frames a trade-off.

Provides the strongest arguments for one side.

The human then considers the resulting analysis and independently concludes:

Technically, the human made the final decision.

But much of the intellectual terrain on which that decision was made was constructed by the machine.

Call it borrowed judgment.

The professional still signs their name.

The organisation records a human decision.

But part of the judgment may have been inherited from the AI without anyone explicitly deciding to delegate it.

Perhaps the most interesting finding came after the exercise.

Researchers asked analysts to explain why they had made their choices.

77% reproduced arguments used by their AI agent, sometimes using almost identical wording.

Yet only 15% credited the AI as the source of those arguments.

More than half of the surveyed analysts described the final recommendation as more their own than the agent's.

Their confidence was also largely unaffected.

Analysts who followed the AI's preference across all four methodological decisions reported roughly the same confidence in their final recommendation as everyone else.

That makes the problem harder.

If people knowingly deferred to AI, organisations could simply ask them.

But people may not always recognise the influence themselves.

The machine's reasoning can become part of their own reasoning.

And once that happens, auditing human independence becomes surprisingly difficult.

Consider an investment analyst.

Eventually, it concludes that a particular assumption about margins is the most reasonable one.

The analyst reviews the work. The assumption is defensible. Nothing appears wrong.

So the analyst uses it.

Later, another AI-assisted analyst independently reviews the company.

Their AI uses the same model family, perhaps with similar training and similar embedded preferences.

It reaches the same assumption.

Now two analysts agree.

Normally, that agreement could increase confidence.

Two professionals independently examined the evidence and reached the same conclusion.

But were the judgments actually independent?

If both analysts inherited the same machine preference, we may be counting one underlying judgment twice.

This matters far beyond investment analysis.

Many institutions become more confident when several experts independently reach similar conclusions.

But as Anthropic notes, if multiple experts increasingly rely on systems with similar underlying preferences, apparent consensus may no longer represent independent judgment.

That is a fundamentally different governance problem.

There is an important reason this problem is difficult to detect.

Many AI controls focus on correctness.

Those checks work well when there is a correct answer. But many of the decisions professionals are paid to make do not have one.

Reasonable experts can disagree.

That is precisely why companies employ experienced people rather than simply applying formulas.

In those situations, an AI can influence the outcome without producing anything objectively wrong.

Both choices may be defensible.

The model simply makes one side more persuasive.

And traditional quality-control systems may see nothing unusual.

There is another uncomfortable finding.

The researchers explored whether more experienced or knowledgeable analysts were less likely to follow the agent.

They found no clear protective effect.

Years of experience, education, causal-inference knowledge, self-rated expertise, AI experience and several personality measures did not meaningfully predict who would defer.

This result was exploratory, and the sample was small, so it should not be overinterpreted.

But it challenges an intuitive assumption.

We often imagine that sophisticated users will simply know when to ignore AI.

Perhaps.

But sophistication may also allow AI to influence people in more sophisticated ways.

An expert does not need to accept an obviously bad recommendation.

They only need to accept one reasonable framing among several.

This is where the study becomes more interesting than a simple warning about automation.

The analysts using AI were faster.

They also made substantially fewer serious methodological errors than the small group that chose not to use the agent.

Among the 50 AI users, three made serious errors.

Among the 11 analysts working without AI, six did.

The groups were not directly comparable, and the non-AI participants were less experienced, so the researchers explicitly caution against treating that difference as causal.

Still, the broader message matters.

AI can simultaneously make work better and human judgment less independent.

Those outcomes are not contradictory.

A company could rationally want both the productivity benefits and the analytical improvements.

The challenge is recognising what has been traded away.

This has direct implications for governance.

Many companies will address AI risk by requiring a person to approve important outputs.

That is sensible.

But approval and independent judgment are not the same thing.

A senior analyst clicking approve after reading an AI-generated recommendation does not necessarily constitute meaningful human oversight.

Neither does a doctor reviewing an AI-supported diagnosis.

Nor a partner approving an AI-generated consulting recommendation.

Nor an investment committee reviewing a thesis largely assembled by agents.

The real question becomes:

At which points does the organisation genuinely require a human to form an independent view?

That may mean asking people to state their initial position before seeing the AI's.

It could mean asking AI to present the strongest arguments on both sides rather than only recommending one.

For particularly important decisions, organisations may deliberately separate the person forming the judgment from the AI-generated recommendation.

But this cannot apply to every minor decision.

The entire value of agents comes partly from not requiring humans to reconsider every step.

So companies will eventually have to decide which judgment calls are important enough that independence itself becomes valuable.

Banks have long understood model risk.

A financial model can be mathematically correct and still produce dangerous decisions because its assumptions are wrong.

AI introduces something more subtle.

The model may influence the human assumptions surrounding the model.

That makes AI governance partly a question of psychology and organisational design.

We may need to distinguish between:

Human approval

and

Human judgment

The first is easy to document.

The second is much harder to prove.

That distinction becomes increasingly important as agents take over more of the analytical process.

The most important AI delegation may not happen when a company explicitly says:

Let the machine decide.

Everyone notices that. Policies appear. Controls appear. Risk committees get involved.

The more subtle transition happens when the company still believes humans are deciding.

The human reviews the result and signs their name. Nothing formally changed.

Yet somewhere inside that workflow, judgment moved.

Not completely. Not necessarily badly. But enough that the distinction between human and machine decision-making becomes difficult to see.

The Anthropic experiment is small.

It involved 50 analysts, one statistical task, one model and AI agents whose preferences were deliberately strengthened to make their influence measurable. Real-world effects could be considerably smaller.

And importantly, the study does not show that AI recommendations generally make decisions worse.

It shows something narrower and potentially more important.

Experts can absorb AI preferences without fully recognising that they have done so.

For executives, that creates a new governance question.

Keeping humans in the loop may not be enough. We may need to ask whether, at the decisions that genuinely matter, the human still brings something independent to it.

Because otherwise the company may believe it has retained human judgment... when in reality it has simply borrowed the machine's.

Until next time,
Stay adaptive. Stay strategic.
And keep exploring the frontier of AI.

Fabio Lopes
XcessAI

💡Next week: I’m breaking down one of the most misunderstood AI shifts happening right now. Stay tuned. Subscribe above.

Read our previous episodes online!

You can have thousands of people following your work and still need an algorithm’s permission to reach them. Creators know the drill: you post, you pray, you check the numbers, you try again tomorrow.

A newsletter changes that relationship. On beehiiv, you can build a subscriber list you own, reach readers directly, and turn their attention into a sustainable business with subscriptions, sponsors, a global ad network, and more.

Get started for free.