Blind Variation, Selective Retention

The paper in under a minute

Can an AI scientist propose creative ideas?

AI agents now beat the best human-designed solutions on mathematical discovery, optimization and a growing list of scientific problems. But strong performance alone does not make a method creative.

Prior work has repeatedly observed that AI-generated proposals tend to be incremental: modifications or extensions of approaches that already exist. The usual remedies each run into a wall. Prompting a model to "be novel" yields diminishing returns on frontier models. Retrieved literature is a valuable reference, but the papers an agent retrieves for itself suffer from the same lack of diversity as its own proposals. Heuristic search relies on a fine-grained signal, while most scientific problems offer only a coarse score for a finished proposal.

We take a different route, borrowed from a sixty-year-old theory of human creativity. Donald Campbell described creative thought as blind variation and selective retention: scientists first generate diverse ideas without foreknowledge of which will succeed, often drawing on experience unrelated to the problem at hand, and then use what they have learned to discard the unreasonable ones and keep the plausible ones for further investigation.

The analogy to AI scientists turns out to be surprisingly literal. Giving a model too much task information while it generates proposals makes it anchor on familiar approaches. So we split the roles: limit task information during exploration, and exploit accumulated empirical experience during selection.

What counts as creative?

Psychologists have studied how to judge creativity since the 1950s, and identify three defining properties of a creative idea: it is novel, it is effective, and its success was not obvious in advance. Simonton turned this into a quantitative definition, which we adopt throughout:

The product matters. An idea that everyone proposes scores zero no matter how well it works; so does a wild idea that fails, and so does a rare idea whose success the literature already guarantees. Try it:

To use this definition on an AI scientist, each candidate method (an idea, a supporting reference paper, and the code that implements it) is scored on the three terms:

Limited information induces more creativity

During ideation, the model reads a problem context \(I\) and returns one idea together with a reference paper. The context can be the full problem description \(S\), or only part of it. The question is what happens to creativity as \(I\) reveals more and more of \(S\).

Why would less information help?

Extra task information makes ideas more relevant, but it does something else too: it gives every independent generation the same strong signal to follow. We model this as shared-signal generation. Each idea class \(z_i\) has a task-association score \(a_i\), and a generation picks the idea maximizing \(\lambda(b)\,a_i\) plus its own exploratory noise, where \(b\) is the amount of task information and \(\lambda(b)\) grows with it. The resulting proposal distribution is a softmax that sharpens as information increases:

The figure below plays this model forward for one batch of 112 proposals, the batch size used in our experiments. Turn up the task signal and watch the proposals pile onto the most task-associated ideas.

Testing it on 12 scientific tasks

To test this empirically, we use Claude Opus 5 to summarize each full problem statement into contexts that reveal different amounts of information about the same task, and measure how much each context reveals with \(\mathcal{V}\)-usable information, a model-dependent analogue of mutual information:

Then five proposer models, all with reasoning enabled and a literature-search tool, generate 112 proposals for every task at every information level. Each proposal is implemented and run by the benchmark's evaluator.

The figure below is the main result. Slide the amount of task information from nothing to the full description and follow all four measurements at once.

Because AI-scientist tasks place particular value on the best methods, the first two panels look at the top 1% of proposals within each benchmark. Models produce a larger share of these top-ranked proposals at lower information levels: their relative frequency falls by 42% for creativity and 25% for utility as information grows from 0 to 140 bits. Mean originality drops by 15%, and mean non-obviousness by 67%.

We call this way of improving model creativity, simply giving the model less, Blind Variation (BV).

Against prompting-based baselines

Prior work improves AI-scientist creativity by changing the prompt. We compare BV with three such baselines (analogical reasoning, cross-domain search, and no-domain search), plus one control: the same BV prompt but with the full task description, which separates the effect of limited information from the effect of our prompt. All methods share the proposer models, candidate budget and reasoning configuration.

Learning from scientific experience

Blind variation deliberately reduces task-specific guidance during idea generation. But discovery should not ignore what is learned by working on the task. Scientists develop judgment about a problem by testing ideas on it, and verification is the expensive part: wet-lab experiments can take months, and machine-learning experiments can burn substantial GPU time.

So we treat every proposal already verified on a task as a unit of experience, a pair of a candidate and its empirical outcome, and learn from those pairs a scoring function that decides which new candidates should be retained for execution.

Screening is easier than picking a winner

A selector does not need to predict every score, or even rank candidates precisely. Its job is narrower: keep high-value candidates above the discard line. That is a much easier problem than finding the single best one.

A pairwise objective for keeping the best

We implement the selector as a language model with a scalar scoring head: given the task specification \(S\) and a candidate \(x\), it outputs \(s = f_\theta(S, x)\). For each training run, \(\mathcal{T}\) holds the top-5% candidates by utility and \(\mathcal{B}\) is a uniform sample of the rest. The loss compares only across those two groups:

Each term is one small lesson in scientific judgment: of two earlier proposals, which was more worth the experiment? The selector, a fine-tuned Qwen3.5-9B, is trained on 40,733 verified proposals and evaluated on 5,783 held-out proposals from the same 12 tasks. To test it, we rank the held-out candidates, discard the lowest-ranked fraction, and count how many of the truly top proposals survive.

Learned selection preserves top candidates far better than prompting. The in-context-learning judge, despite seeing verified outcomes of similar proposals, does no significantly better than zero-shot judges. Supervised fine-tuning helps, but trails SR: predicting scores accurately across the whole distribution is a different objective from keeping exceptional candidates out of the discard set.

At a discard ratio of 50%, SR preserves 79% of the top-5% proposals by creativity and 82% by utility. We use this operating point in everything that follows: the selector throws away the lowest-ranked half of all candidates before any of them is run.

BVSR inside an AI scientist

Do the two pieces add up to better end-to-end discovery? We plug BVSR into SimpleTES, a widely adopted AI-scientist workflow, and compare it with the original on five tasks: gene expression analysis, drug response prediction, anomaly detection, scaling-law discovery and brain activity prediction. Both configurations use Kimi-K3, with four parallel search trajectories of 40 proposals per iteration, for 10 iterations.

Which half is doing the work?

Both. We run four configurations on two tasks: the original SimpleTES, SimpleTES with BV alone, with SR alone, and with full BVSR. Toggle the configurations to compare them.

What it discovered

The most interesting outputs are the programs themselves. Because blind variation frees the model from the task's home field, BVSR's best programs borrow mechanisms from far away: transition-state theory, superconductivity, matching markets. Each of the three below beats the best program found by SimpleTES under the same budget, and the strongest existing baseline for its task.

Where do good ideas come from?

During generation, models search the literature for a paper to draw on. We sampled 20,000 proposals, grouped them by the field of their reference paper, and asked which fields are over-represented among each task's top 1%. Mathematics is the best source of top-creativity proposals, while biology / medicine and mathematics lead for top utility.

Do human experts agree?

Two of our three creativity terms rest on LLM judgments, so we checked both against PhD-level researchers, under blinded conditions.

The information measure is also robust to the choice of scoring model: across ten open-weight models, pairwise estimates agree with a median Pearson correlation of 0.91, and the three-model average we use correlates with the ten-model average at 0.99.

BibTeX

@misc{xu2026BVSR,
  title={Blind Variation, Selective Retention: Improving Creative Idea Generation in AI Scientists},
  author={Guowei Xu and Jiaru Zou and Mete Erdogan and Haotian Ye and Pan Lu and James Zou},
  year={2026},
  eprint={2611.12184},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2611.12184},
}