Attribution and Evaluation

In one line: the system produced a bad answer, five components each reported success, and something has to determine where it actually went wrong.

Why this is hard

In a single agent, a failure is one trajectory and the trace is linear. Read it and the wrong step is usually visible.

In a multi-agent system, the failure is distributed. Each worker did what it was asked, the critic passed each output, and the assembled result is wrong. There is no defective component — which is exactly the framing failure from the error-propagation lesson.

So attribution has to distinguish three cases, and they need completely different fixes.

The worker was wrong. It was given a clear sub-task and produced a bad result. Fix the worker — its prompt, its tools, its model.

The worker was badly briefed. The sub-task was ambiguous, underspecified, or missing context the supervisor had. The worker did well given what it received. Fix the decomposition.

The decomposition was wrong. Every sub-task was clear and well executed, and they did not add up to the goal. Fix the supervisor.

The second is the most common and the most often misdiagnosed. Teams look at a bad worker output and improve the worker, when the sub-task it received could not have produced a good one. That is consistent with specification being the largest failure category in the published taxonomy — the errors surface at the workers and originate at the supervisor.

The test that separates them

Cheap and effective: give the sub-task, exactly as the worker received it, to a fresh instance with no other context.

If the fresh instance produces something similarly wrong, the brief was bad. If it produces something good, the worker was.

That works because it removes the reviewer's own knowledge from the loop. A supervisor reading a worker's output supplies the missing context automatically and concludes the worker underperformed — which is the implicit-context failure from the interface lesson, appearing during debugging rather than during execution.

What to log

Attribution depends entirely on what was recorded, and this system needs more than a single agent does.

Per agent, per invocation: the exact input, the exact output, the model and prompt version, the cost, and the elapsed time.

Per handoff: the full structured message, and the validation result — including the ones that passed, because a validation that passed on a wrong value tells you the check was too weak.

The decomposition itself, with the reasoning that produced it. This is the artifact most often missing and the one needed most, since the largest failure category originates here.

The mapping from final output to sub-results. Which parts of the answer came from which worker. Without it, tracing a wrong sentence back to its source is guesswork.

That last one is genuinely hard when the assembler is a model — it synthesises rather than concatenates, so the provenance is not automatic. The practical answer is to require the assembler to cite which sub-result each claim came from, exactly as a RAG system cites passages, and to verify those citations mechanically.

Evaluating the system

Four levels, and measuring only the last is the common failure.

Per agent. Given a sub-task, does the worker do it well? Testable in isolation, with a fixed set of sub-tasks and expected outputs. Cheap, and it is the only level where a regression can be localised.

The decomposition. Given a goal, are the sub-tasks correct and complete? Requires human judgement and it is the highest-value thing to review, because it is the largest failure source.

The handoffs. Do the messages carry what the receiver needs? Testable by the isolation replay above, run systematically.

End to end. Does the system produce the right answer? What the user cares about, and useless for diagnosis on its own.

The counterfactual question

The one worth asking of any multi-agent system in production, and the one that most often changes a design.

Would a single agent have done this?

Run the same evaluation set through one agent with all the tools, and compare on quality, cost and latency. The result is frequently uncomfortable — comparable quality, lower cost, simpler system — and it is far better to discover that from a measurement than from an incident.

That comparison should be a standing part of the evaluation rather than a one-off, because as models improve the case for decomposition weakens. A split that was necessary a year ago because one context could not hold the whole problem may no longer be.

Metrics worth watching

Five, and two are specific to this shape.

Task success, end to end, verified against the goal.

Attribution rate — of the failures, how many could be localised to a component. A low rate means the logging is insufficient, and it should be treated as a bug in the system rather than a property of the problem.

Coordination share — what fraction of total cost went to decomposition, dispatch and assembly rather than to work. A rising share means the tax is growing, usually because someone added a layer.

Critic rejection rate. Near zero means the critic is not doing anything; very high means the producers or the specifications are bad. Both are actionable and the number is free.

Handoff validation failure rate, per interface. This localises specification drift to a boundary, and it moves before end-to-end success does — the same early-warning property as verification failure rate in the browsing chapter.

Next: the whole thing, as an interview.