The Serving Layer, and the CDN Mistake

In one line: the separation of online from batch serving is well judged. The component sitting in front of it is not.

Two access patterns

Online serving is optimized for speed. Models fetch the latest feature values from the online store via low-latency APIs (gRPC or REST).

Batch serving is optimized for high throughput and offline access. Data scientists export full training datasets or run feature backfills from the offline store. This supports massive reads without impacting real-time inference latency.

Isolating the two paths is the point, and the clause that says why is easy to miss

"This supports massive reads without impacting real-time inference latency."

That is the justification, and it is the right one. If a data scientist exporting six months of features for 50 million users hit the same store as live inference, the export would consume the memory, connections, and I/O that a 10 ms budget depends on.

A single analytical query can destroy an operational latency budget, and the defence is not rate limiting or query optimization — it is physical separation. Different stores, different hardware, different failure domains.

Batch export runs long / fails / consumes everything
  -> online inference is COMPLETELY UNAFFECTED — different system

This is the same instinct as the read replica pattern, and as the fast-path/slow-path splits throughout this module. What makes it stronger here is that the two stores were already separate for access-pattern reasons (Lesson 7) — so the isolation is free, a second dividend of a decision made for other reasons.

Separating stores by access pattern buys workload isolation as a side effect, and it is worth naming both benefits when you justify the dual store.

gRPC over REST is the right default on this path, for reasons that actually apply here

The design names both and does not choose. gRPC is the better default for feature serving, and unusually the standard arguments genuinely apply:

Binary protobuf rather than JSON. A feature vector is a few hundred numbers. JSON encodes each as text with field names repeated per record; protobuf encodes them positionally in binary. For numeric payloads the size difference is large, and the parsing cost difference is larger — and at 15,000 RPS, deserialization is a real share of the latency budget.

Persistent HTTP/2 connections with multiplexing. No per-request connection setup, and many concurrent requests over one connection — which matters when a model server is making batched feature lookups (Lesson 7's array-valued serveFeatures).

A generated, typed contract. Feature schemas change; a .proto makes an incompatible change a compile error rather than a runtime surprise. That is the same argument as the schema registry in Lesson 4, applied to the serving boundary.

The caveat: REST is fine when the caller is a browser or an external partner. gRPC belongs on internal, high-frequency, numeric paths — which is exactly what this one is.

The CDN

CDN: Caches feature data geographically closer to inference models to reduce latency.

A user request travels through a CDN and load balancer to the deployed model.

Feature data is the worst possible CDN candidate, and caching it recreates the chapter's headline problem

A CDN caches content that is shared across many requesters and changes infrequently. Feature vectors are the opposite on both counts, and there is a third objection that is fatal.

They are per-entity, so the hit rate is near zero. A CDN's value comes from many users requesting the same object. Feature vectors are keyed by user or item — every request wants a different object.

CDN for an image:    1 object, 1,000,000 requests   -> ~100% hit rate
CDN for features:    1,000,000 entities, ~1 request each -> ~0% hit rate
                     -> you have added a network hop and gained nothing

They change constantly. Lesson 7 defined the online store as holding "the latest feature values," updated continuously by the stream path. Anything cached is immediately at risk of being superseded.

And a stale feature is training-serving skew. This is the fatal objection. The chapter's first challenge, its quiz question, and the entire justification for the feature store is that the values a model sees at inference must match what it was trained on. A cache serving a value from thirty seconds ago is, by definition, serving a value the model was not given:

Online store:  txn_count_7d = 13   (updated seconds ago)
CDN cache:     txn_count_7d = 12   (cached 60 s ago)
Model receives 12. The pipeline computed 13.
-> a value from a moment that no store holds
-> skew, introduced deliberately, by a caching layer

Introducing a cache into a path whose correctness requirement is freshness undoes the layer that exists to guarantee freshness.

Where a CDN does belong in an ML system:

The model artifact row is the one worth substituting in an interview: a model file is hundreds of megabytes, identical across every serving node in every region, and immutable once versioned — every property a feature vector lacks. That is a genuine use for edge distribution, and it is the same content-addressed, immutable-artifact distribution problem as the deployment chapter.

Cache what is shared and immutable; never cache what is per-entity and whose freshness is the requirement.

The model registry

Named once in the workflow and it closes the loop

The high-level workflow mentions that trained models are "versioned in the model registry", and the component then largely disappears from the design.

It matters because it is the counterpart to the feature registry, and together they make a deployed model fully described:

FEATURE registry:  what the inputs mean, and at which version
MODEL registry:    which model artifact, trained when, on which data,
                   using which feature versions

Without both, "what is running in production and what does it depend on?" is unanswerable — and it is the question every ML incident starts with.

The dependency runs one way and is easy to get wrong: a model version pins a set of feature versions. Deploying model v2 that requires a new feature means the feature must exist in both stores before the model is promoted — which is exactly the deployment ordering the design's own exercise asks about. Promote the model first and inference requests a feature the online store does not have.

Deploy the feature before the model that consumes it, and record the binding in the registry. This is the same expand-then-migrate discipline as the deployment chapter: make the dependency available before the thing that depends on it.

Next: the evaluation.