APIs and Storage Schema

In one line: the API design is the best in the AI module — every endpoint states its response type and its latency contract. The storage schema is where the privacy problem from Lesson 2 becomes concrete.

The APIs

Specifying the response type per endpoint is what makes this API design good

Most API tables in this module list parameters. This one lists how the response arrives — streaming, async, or synchronous — and that is the more consequential fact.

Each choice matches a real constraint:

getCompletion streams, because Lesson 3 showed the budget is on the first token. A blocking response would have to wait for the whole completion, and the whole completion is not what the deadline is about.

submitFeedback is fire-and-forget through pub-sub. The source's own quiz asks why, and the answer is right: to avoid blocking the IDE. Feedback is valuable and it is not worth a millisecond of a developer's typing. This is the clearest possible statement of a principle this module keeps returning to — take everything that is not required for correctness off the critical path.

getStatus is synchronous and exists so the plugin can check availability before sending inference requests, enabling client-side fallback rather than a failed request the developer waits on.

An API contract that specifies delivery semantics per endpoint is documenting its latency budget, and that is more useful than a parameter list.

Two endpoints stream and their latency contracts are completely different

getCompletion and explainCode share a transport and share nothing else.

getCompletion:  fires on TYPING            · TTFT < 300 ms · output is short
                the user did not ask       · silence on failure is correct
                a wrong answer costs a keystroke

explainCode:    fires on a CLICK           · seconds are fine · output is long
                the user DID ask           · silence on failure is a BUG
                a wrong answer wastes real attention

The design applies one degradation policy to both — "if inference fails, return no suggestion." That is exactly right for completion and wrong for explanation: a developer who selected code and clicked "explain" and received nothing has hit a broken button.

getCompletion fails -> show nothing. The developer never knew they asked.
explainCode   fails -> show an error. The developer is WAITING.

The same split applies to the context budget (Lesson 4) and to caching: an explanation is a far better cache candidate than a completion, because the input is a selected block that does not change on every keystroke.

One transport does not imply one contract — and a degradation policy has to follow whether the user initiated the request.

The debounce tip is the same lesson as the typeahead chapter, arriving from a different direction

Practical tip: Adding a client-side debounce of 150–200 ms before calling getCompletion prevents excessive requests on every keystroke and reduces backend load.

This is the largest capacity lever in the system and it is offered as a tip.

At 40–80 words per minute, keystrokes arrive every ~150–200 ms. A debounce at that threshold means a request fires only when the developer pauses — which suppresses a large fraction of would-be requests, and the suppressed ones are precisely those whose context would be stale by the time the response arrived.

Two things follow, and both are consistent with Lesson 2's numbers:

It is what makes "50 requests per active hour" plausible. One request every 72 seconds is not one per keystroke; it is one per meaningful pause. The debounce is why the assumption holds.

It is the cheapest GPU saving available. Every suppressed request is inference that never happens — better than caching it, better than routing it to a smaller model.

The cheapest request is the one that never leaves the editor — the same conclusion as the typeahead chapter, where debouncing at the stated inter-keystroke interval suppressed roughly half of all requests.

One refinement the design omits: cancel in-flight requests when the context changes. If the developer keeps typing while a request is outstanding, that response is already answering a stale prefix. Cancelling frees GPU capacity immediately and avoids rendering a suggestion for code that no longer exists.

The storage schema

Four stores, four access patterns — and the model registry is the one that earns its place

The split is well judged: two latency-critical stores on the request path, two analytical stores off it.

The model registry is the entry worth noticing, because it is doing more than the description suggests. Tracking model_version, quantization level, and deployment_status is what makes two things possible:

Safe rollout and rollback. A model version can be promoted regionally and reverted by changing a pointer — which is the same build-beside-and-swap pattern as the trie publication in that building block and blue-green deployment in that building block.

Attribution. Telemetry records model_version, so acceptance rates are comparable across versions. Without the registry you can measure quality; with it you can measure whether a change improved quality.

And per Lesson 4, the registry should record one more thing the design omits: the prompt format, including FIM markers. Those are model-family-specific, so a model swap that ignores them produces silently degraded completions rather than an error.

A model registry is what turns "we deployed a new model" into a measurable, reversible operation.

The completion cache is well specified and its key is the problem

Everything about this entry is right except the thing being hashed.

TTL for automatic expiry — correct, and the design's warning is exactly right: "too low significantly reduces the hit rate, too high results in outdated suggestions." A completion cached against a file that has since changed is not stale, it is wrong.

Consistent hashing across Redis nodes — correct, and it prevents hotspots.

Storing the model version alongside — correct and often forgotten. A completion generated by model v1 must not be served after v2 ships, or you cannot attribute quality changes to the model.

The problem is that the key is hash(assembled prompt), and Lesson 3 showed why that nearly never hits for inline completion: the assembled prompt contains the file content and cursor position, which change on every keystroke.

This cache helps:  boilerplate that many developers type identically
This cache misses: essentially every interactive completion

The mechanism that does work — prefix KV cache reuse — lives in the GPU tier, not in Redis, and the schema has no entry for it. That is not an omission in the schema so much as a sign that the design under-weights the more important of its two caches.

When your cache key contains something that changes every request, you are caching the wrong thing.

The telemetry schema says metadata and the volume says source code

Two statements, one section apart.

Telemetry events capture each feedback signal with fields for completion_id, session_id, timestamp, action, language, and latency metrics.

This store ingests approximately 4 TB per day.

Add up the stated fields:

completion_id  ~36 B   session_id  ~36 B   timestamp   ~8 B
action         ~10 B   language    ~10 B   latency    ~16 B
                                          --------
                                          ~120 bytes, generously ~200

At one billion events a day, that is 0.2 TB — not 4 TB. The 4 TB figure comes from Lesson 2's assumption of "4 KB of telemetry per request-response pair", and 4 KB is exactly the 3 KB request plus the 1 KB response.

The schema describes metadata. The estimate prices the prompt and the completion. Those differ by 20×, and the entire difference is whether you retain the customer's code.

METADATA ONLY:   0.2 TB/day    no source code retained
WITH PAYLOAD:      4 TB/day    1.46 PB/year of customer source code

Which matters because of a stated requirement: "options for enterprise customers to restrict data retention entirely." You cannot offer that and default to storing every prompt.

The resolution the design needs is tiered retention keyed on tenant policy:

ALL tenants:            metadata — always, cheap, no exposure
OPTED-IN tenants:       full prompt + completion, for fine-tuning
NO-RETENTION tenants:   metadata only, enforced at ingestion, not at query time

The enforcement point matters: drop the payload at the ingestion consumer, before it is written, rather than filtering on read. A payload written and later excluded from queries has still been retained.

When a storage estimate is exactly the size of the payload, the system is storing the payload — and here it is the thing the privacy requirement exists to protect.

What the schema is missing

Three entries the design would need in production, each covered elsewhere:

A repository embedding index. Lesson 4's retrieval layer queries a vector database of repository embeddings, and no schema entry describes it — including the scoping that matters most: per repository, per branch, with incremental re-embedding on commit. It is also, per Lesson 7, a copy of the customer's repository.

A prompt-format record in the model registry. FIM markers and template structure are model-family-specific (Lesson 4).

Tenant policy. Retention mode, data residency region, and whether public-code filtering is enabled — all of which the requirements imply and none of which has a home.

Next: the inference path, and the optimizations that make the budget close.