Telemetry, Metrics, and the Perception Gap

In one line: the chapter is unusually honest that this product's benefit is disputed by measurement. That honesty makes the metric design the most important part of the system, and the chapter gets half of it right.

The telemetry path

While the inference path is latency-critical, the telemetry path is designed to be fully asynchronous, ensuring that it never impacts the developer experience.

Full separation of the two paths is the design's cleanest structural decision

Two paths, two contracts, and no coupling between them:

INFERENCE PATH:  synchronous · 300 ms budget · every millisecond visible
TELEMETRY PATH:  asynchronous · seconds to minutes · nobody waiting

The queue is what enforces the separation. submitFeedback returns immediately; a consumer drains at its own pace; aggregation runs on a schedule. A telemetry outage cannot slow down a keystroke, because there is no synchronous dependency to slow.

This is the same principle as the pub-sub tiers in the deployment, payment, and support-bot chapters, and it lands with unusual clarity here because the latency budget is so tight that any synchronous coupling would be visible.

The columnar store is the right target for the same reason it was in that building block: the write pattern is append-only at high volume and the read pattern is analytical scans over a few columns. "Acceptance rate by language, by model version, over the last week" is exactly what columnar formats are built for.

Take everything not required for correctness off the critical path — and here the critical path is 300 ms, so the rule is not a preference.

Acceptance rate is gameable

Aggregation jobs compute acceptance rates per model version, latency distributions, and performance across languages or repositories.

Shorter suggestions are accepted more often, so acceptance rate rewards suggesting less

Acceptance rate is the obvious metric and it has a structural bias that makes it dangerous as an optimization target.

A one-token completion  ("  " -> ")")     -> accepted ~always, worth ~nothing
A twenty-line function                    -> accepted rarely, worth a great deal

The probability a suggestion is accepted falls with its length, because a longer suggestion has more surface on which to be wrong. So a model tuned to maximize acceptance rate learns to suggest less — shorter, safer, more obvious completions — and the metric improves while the product gets worse.

This is Goodhart's law in its cleanest form, and it is the same failure the support bot chapter had with thumbs-up: the easy signal measures something adjacent to value, and optimizing it moves the system away from value.

SUPPORT BOT:    thumbs-up rewards fluency and agreeableness, not accuracy
CODE ASSISTANT: acceptance rate rewards brevity and obviousness, not usefulness

The composite that fixes it is straightforward and the design has half of it:

Value ≈ acceptance rate x characters retained

A single-token completion accepted every time contributes almost nothing; a twenty-line function accepted a third of the time contributes a great deal. Measure characters of accepted code surviving in the file, not the count of accepted suggestions.

Any acceptance metric must be weighted by the size of what was accepted, or it rewards suggesting nothing.

Edit distance is the chapter's best metric, and it is the one that survives Goodhart

Monitoring services collect metrics such as suggestion acceptance rate, edit distance between the suggested code and the final code, time-to-first-token latency, and model performance by language or repository type.

That second item is genuinely good and it is rare to see it named.

Acceptance is a moment; edit distance is an outcome. A developer accepts a suggestion, then rewrites half of it. Acceptance rate counts that as a success. Edit distance shows what actually survived:

Accepted, kept verbatim         -> edit distance 0     -> real value
Accepted, then heavily rewritten -> edit distance high  -> the suggestion was a
                                                          starting point at best
Accepted, then deleted           -> everything removed  -> negative value:
                                                          it cost time

And it is much harder to game. Optimizing for low edit distance means optimizing for code the developer keeps, which is the actual product goal. Short trivial completions score well on acceptance and contribute nothing to retained characters.

The refinement worth adding: measure it after a delay, not at the moment of acceptance.

Edit distance at t+0s:     ~0 by definition
Edit distance at t+5min:   did it survive the developer's next thought?
Survival at commit time:   the strongest signal available

Retention over time is the honest version of acceptance, and the design has the identifiers to compute it — completion_id correlates the suggestion with the file region it landed in.

The perception gap

Research into the perception gap suggests that developers often believe AI assistance makes them faster, yet empirical data sometimes shows no measurable improvement. This reinforces why the monitoring service must track actual developer productivity metrics rather than relying solely on raw suggestion throughput.

A chapter that admits its product's benefit is contested is telling you what to measure

Almost no system design source says this about its own subject, and it changes the design's centre of gravity.

If perceived and measured productivity diverge, then every metric derived from developer behaviour is suspect — acceptance rate most of all, because accepting a suggestion is exactly the behaviour that feels productive.

FEELS productive:  the suggestion appeared, it looked right, I pressed Tab
IS productive:     the code shipped, and I did not spend the saved time
                   debugging it

The metrics that survive that distinction are further downstream and harder to attribute:

Characters retained at commit          <- did the code survive?
Time from first keystroke to merged PR <- did the task get faster?
Defect rate in AI-assisted regions     <- did quality hold?
Review cycles per PR                   <- did it create work downstream?

The last is the one the chapter's own note points at, and it is the sharpest observation in the design:

AWS research identifies a review and QA bottleneck in which AI-generated code increases output volume faster than teams can review and validate it.

A system that increases production without increasing review capacity moves the constraint rather than removing it. If review was already the slower half of delivery, generating more code makes the queue longer, and local speedup produces global slowdown.

That is a genuinely systems-level observation about a productivity tool — optimizing one stage of a pipeline is only a win if that stage was the bottleneck — and it is why the monitoring section belongs in the architecture rather than in an appendix.

When the benefit is contested, measurement is not reporting; it is the mechanism by which you find out whether to keep building.

The fine-tuning loop

These aggregated signals feed into fine-tuning pipelines, evaluation systems, and monitoring dashboards.

Training on accepted completions has a bias the design does not name

The loop is: suggest, observe acceptance, fine-tune on what was accepted. It is intuitive and it has a self-reinforcing failure.

Accepted suggestions are the ones the model already gets right. Training on them strengthens existing behaviour and teaches nothing about the cases it gets wrong — which are exactly the cases that were rejected and therefore excluded from the training signal.

Model is good at Python boilerplate  -> those suggestions get accepted
                                     -> training set fills with Python boilerplate
                                     -> the model gets better at Python boilerplate
Model is weak at Rust lifetimes      -> those get rejected
                                     -> excluded from training
                                     -> it stays weak, permanently

A feedback loop trained on successes narrows rather than broadens. It is the same shape as the recommendation-system filter bubble, and it will show up in the design's own "performance by language" metric as a widening gap between the languages it started strong in and everything else.

Two corrections, both of which the design's telemetry can already support:

Train on the corrected version, not the accepted one. The strongest signal is (prompt, what the developer actually ended up with) — which is what edit distance already measures. A rejected suggestion followed by hand-written code is a labelled example of the right answer, and it is the most valuable data the system produces. Discarding rejections throws away the only signal about failures.

Sample deliberately for coverage. Weight the training set toward languages, repository types, and constructs where acceptance is low, rather than letting it be dominated by whatever traffic is most common.

And the caveat Lesson 7 raised applies with full force here: fine-tuning on customer code means customer code enters model weights, from which it cannot be deleted. That is a stronger commitment than retention, and it needs the same tenant-policy gate — enforced before the training set is assembled, not after.

Next: the evaluation.