The Proposer: Candidates Come From Failures

In one line: a proposer that has seen what went wrong searches a different space from one that has only seen the current artifact, and the difference shows up as the loop either converging or wandering.

The naive proposer asks a model to write a better version of the current prompt. It produces fluent, plausible variants that score about the same, because nothing in the request told it what is broken. The optimiser then spends its evaluation budget discovering that.

Grounding a proposal

The inputs that make a proposal informed are: a sample of failing traces, a summary of what they have in common, the current artifact, and a note of what has already been tried and rejected. The last one is routinely omitted, and without it the loop re-proposes the same idea for weeks.

Turning traces into targets

Failures arrive as a pile. They are useful as clusters.

Proposing against one cluster at a time is the part that matters. A candidate aimed at a single named defect either fixes it or does not, and both outcomes teach you something. A candidate aimed at "the failures" generally moves several things slightly and leaves you unable to say why the score changed.

The named cluster also gives you the targeted eval slice — the cases in that cluster become the set on which this candidate must improve, while the full set is what it must not regress.

Where the trap is

A cluster is a description of failures the current system produces. Optimising against it improves the current system's weak spots and tells you nothing about the failures it never gets the chance to make. That is the same blind spot a recommender has about items it never shows, arriving here through a different door.

The consequence is that a purely failure-driven proposer converges to a local optimum and stays there, improving the fourth decimal place of a shape that was wrong. Two things keep it moving:

• A share of the budget spent on candidates that are not derived from failures at all — a different framing, a restructured output contract, a stage removed. Most will lose. The ones that win tend to win by more than the incremental ones.
• Proposals grounded in inputs the current system handles well, checking that a fix for one cluster has not quietly traded away another.

A reasonable split is most of the budget on targeted candidates and a fixed minority on unrelated ones, held constant rather than tuned — because the loop will happily optimise the exploration share to zero, which is locally correct and eventually fatal.

How many candidates

More candidates is not free, and not only in compute. Each one is a draw on a noisy metric, and the winner of a large field is the candidate that got the luckiest draw at least as often as it is the best candidate. That effect is strong enough to have its own lesson later; the practical form here is that a field of a few dozen candidates evaluated properly beats a field of hundreds evaluated cheaply.

The standard implementation squares this circle by scoring candidates on small batches first and re-scoring only the survivors on the full set. That is a two-stage design, and it is the right shape — provided the second stage uses data the first stage did not.

The version everyone skips

Record the rejected candidates and why they were rejected. It costs a table and it buys three things: the proposer stops repeating itself, the rejection history is the clearest description of what the metric actually rewards, and a candidate that lost narrowly on one cluster is often the best starting point when that cluster grows.

Next: the evaluation the loop treats as truth.