This website uses cookies

Read our Privacy policy and Terms of use for more information.

Welcome Back to XcessAI

Imagine giving an AI agent a large dataset and a simple instruction: find something interesting.

The agent examines the variables, tries different definitions, changes the sample, experiments with statistical methods and tests multiple hypotheses. Eventually, it returns with a compelling result. The methodology looks defensible. The analysis is statistically significant.

There is just one problem.

There may be nothing there.

A new research preprint by Florian Gunsilius, David Van Dijcke and Kaspar Wüthrich explores what happens when frontier AI coding agents are allowed some freedom over how they analyse data.

The researchers created simulated datasets in which they knew that every true effect was zero. They then asked four frontier agents to explore the data and return a "compelling, defensible finding".

When a statistical test is specified in advance and conducted at a 5% significance level under a true null hypothesis, we would expect false rejections only around 5% of the time. In these experiments, the agents reported statistically significant findings between 54% and 98% of the time.

Nothing had happened.

Yet the agents repeatedly found something.

That points to a problem that may become increasingly important as AI takes over more analytical work. AI does not merely analyse data faster. It can search through possible analyses faster.

And if the person reviewing the result sees only the final answer, they may have no idea how much searching happened before it appeared.

The Old Problem Gets Faster

Statistics has struggled with this issue for decades.

Suppose a researcher wants to understand whether a new policy improves employment. There are many reasonable choices to make: which workers to include, which time period to measure, which controls to use, which outcome matters most and which statistical specification is appropriate.

None of those decisions is necessarily dishonest. But each additional choice creates another opportunity to find something that looks statistically significant.

If a researcher repeatedly tries different approaches and reports only the one that works, a result that appears unlikely to have occurred by chance may actually be the product of many chances.

This is one form of selective inference, closely related to what is often called p-hacking.

Historically, the search itself imposed friction. Trying hundreds of specifications takes time. Writing code requires effort. Inspecting every result consumes human attention.

Coding agents change that constraint.

An agent can read the files, write the code, run an analysis, inspect the results, modify the approach and run it again. What might once have required days of human exploration can increasingly happen inside one automated workflow.

The authors describe this as a "supercharged form of selective inference."

The problem becomes particularly difficult for the person reviewing the work.

Imagine receiving an investment analysis from an AI agent. The report contains a regression showing that a particular operational metric has historically been strongly associated with future margins. The relationship is statistically significant, the chart looks convincing and the methodology appears reasonable.

What the reader may not see is everything the agent tried first.

Perhaps it tested twenty operational metrics, six different time horizons and several peer groups. Perhaps it removed unusual years, changed the dependent variable and explored hundreds of combinations before finding the relationship that looked most interesting.

Any individual choice might be defensible. The problem is the selection process across all of them.

The paper demonstrates this directly. In its open-ended simulation, the agents were encouraged to "explore the data as you see fit" and return one "compelling, defensible finding". The prompt even asked them to explain their choices and acknowledge that the exercise was exploratory.

Despite those qualifications, false discoveries were extremely common.

In one open-ended simulation, an agent returned a p-value below 0.05 in 97 of 99 datasets, even though the underlying treatment effects were all zero.

The danger is therefore not necessarily fabricated data or obviously bad analysis.

It is reasonable analysis selected from an unreasonable amount of searching.

When the Calculation Is Right

This is worth separating from hallucination.

The agents did not necessarily invent numbers. They could run legitimate statistical analyses on real data. Their code could execute correctly, and the reported p-value could be mathematically correct for the particular regression they selected.

The problem was how that regression came to be selected.

That is a much subtler failure mode.

The calculation can be correct while the inference is wrong.

This distinction matters because many enterprise controls around AI focus primarily on factual accuracy.

Did the model use the correct data? Did the formula calculate properly? Can we reproduce the chart? Did the system hallucinate anything?

All sensible questions.

But none necessarily tells you how many alternative analyses were attempted before the final one survived.

The Objective Becomes the Bias

The wording of the task matters enormously.

The agents were not told to estimate one predefined effect using one predetermined methodology. They were asked to find a compelling result.

That instruction creates selection pressure.

The system begins searching for something worthy of reporting.

This is why the paper should not be interpreted as evidence that AI agents always manufacture statistical significance. The results vary across agents and settings, and the authors explicitly study environments where the agent has meaningful analytical discretion. Their correction also relies on assumptions about the stability of the agent's selection behaviour.

But that qualification may make the research more relevant to business rather than less.

Executives rarely ask analysts to run one precisely specified regression regardless of the outcome.

They ask questions such as:

What does the data tell us?

What are the main drivers of churn?

Which variables best explain performance?

Find the strongest evidence for or against this investment thesis.

Those are open-ended analytical instructions.

And increasingly, they are exactly the kind of work companies may delegate to agents.

The Business Version

Consider investment research.

An analyst examining a company may have access to hundreds of variables: pricing, volumes, utilisation rates, inventories, commodity prices, freight rates, customer indicators, macroeconomic data and alternative datasets.

Now give an AI agent access to all of them.

The agent can search thousands of relationships cheaply. That is extraordinarily valuable. But suppose it discovers that one obscure indicator has predicted the company's earnings surprisingly well for eight years.

The relationship looks strong. The backtest is impressive. The investment committee sees the final chart.

What it may not see is the enormous graveyard of relationships the agent tested and discarded.

The same issue applies to marketing. Ask an agent to find which customers responded best to a campaign and it can slice the data by age, geography, channel, product, timing and behaviour until an apparently interesting subgroup emerges.

Or strategy. Give an agent years of operating data and ask what differentiates successful factories, stores or sales teams. It can search an enormous analytical space and return a coherent explanation.

The ability to discover patterns becomes almost unlimited.

Unfortunately, so does the ability to discover patterns that are not real.

Persuasion Gets Cheaper Too

There is another layer to the problem.

AI agents do not merely find statistical results. They can explain them.

Once an agent identifies an apparently interesting relationship, the same system can create the chart, propose a mechanism, write the interpretation and produce a polished executive summary.

The entire process becomes:

Search → find → explain → present

Historically, weak findings encountered some friction on the way to becoming compelling narratives. Someone had to build the analysis, create the presentation and construct the argument.

AI can automate much of that chain.

The danger is not necessarily that the machine deliberately lies. It is that it becomes extraordinarily good at constructing a coherent story around something that emerged largely because it searched long enough.

Replay the Researcher

The paper does not merely identify the problem. It proposes a clever way to detect and correct it.

Normally, selective analysis is difficult to adjust for because the reviewer does not know everything the researcher tried.

AI agents have one unusual property that humans do not.

You can reset them and run them again.

After an agent completes its original analysis, the researchers replay it on subsamples of the data. They then measure how frequently the agent selects the same analysis when that analysis happens to look statistically favourable.

The method does not require reconstructing every hidden step of the agent's search. Instead, it learns the selection behaviour from the agent's own repeated choices.

The authors call the resulting measure a selection multiplier.

In the open-ended simulation where the original agent rejected the null in 98% of datasets, enough replay runs reduced the false rejection rate to 5.1%, close to the intended 5% level. Other simulation designs similarly moved back towards nominal false-positive rates.

That does not mean the problem has been solved.

The technique requires repeated agent runs, which adds computational expense. More importantly, its statistical guarantees depend on assumptions including sufficiently stable selection behaviour and accurate reporting by the agent of the analyses it actually performs.

But the broader idea is fascinating.

The programmability of AI creates the problem.

It may also help create the solution.

Companies adopting analytical agents will probably begin with familiar controls.

Save the prompt. Store the code. Record the output. Maintain an audit trail.

Those are useful, but this research suggests they may not be enough.

An audit trail tells you what the agent ultimately did. It may tell you much less about why that particular analysis survived while others disappeared.

As analytical autonomy increases, governance may therefore need to move beyond checking the final calculation.

Companies may eventually need to audit the search process itself.

There is also a simpler defence: decide what matters before looking at the result.

Science already does this through preregistration, where researchers specify hypotheses and methodologies in advance. Businesses could adopt a lighter version of the same principle for important analyses.

Before an AI evaluates a marketing experiment, define the primary metric. Before an investment agent tests a thesis, specify what evidence would support or reject it. Before an operational agent searches for performance drivers, distinguish clearly between confirmatory analysis and exploratory discovery.

Exploration itself is not the problem.

It is enormously valuable.

The problem begins when an exploratory discovery is presented with the statistical certainty of a hypothesis that was specified before the search began.

AI makes that distinction more important because exploration is becoming almost free.

False Discovery

We normally think about AI risk in analytical work as a problem of accuracy.

Did the model hallucinate? Did it calculate correctly? Did it use the right data?

Those questions remain important.

But agentic analysis introduces another problem. An AI can calculate everything correctly and still lead us towards a false conclusion.

Not because the mathematics failed.

Because the search became too powerful.

For centuries, finding relationships in data was difficult. Analysts were constrained by time, tools and attention.

AI removes those constraints.

Soon the scarce resource may not be our ability to find patterns.

It may be our ability to distinguish the patterns worth believing.

That is what makes this preprint so interesting. Give frontier agents data containing no real effects and enough freedom to search for something compelling, and they can frequently deliver exactly what was requested.

The risk is hidden inside the instruction:

Find me something interesting.

AI increasingly can.

The harder question is whether what it found is true.

Until next time,
Stay adaptive. Stay strategic.
And keep exploring the frontier of AI.

Fabio Lopes
XcessAI

💡Next week: I’m breaking down one of the most misunderstood AI shifts happening right now. Stay tuned. Subscribe above.

Read our previous episodes online!

Reply

Avatar

or to participate