The Live Data Pipeline

In one line: this closes the loop from Lesson 9 — location data becomes traffic estimates becomes updated edge weights becomes better routes. And it contains the chapter's sharpest operational insight: not every observation should change the graph.

The pipeline

The analytics engine processes real-time location data for:

Counting vehicles and measuring speed — traffic low, medium, or high
Hotspot identification
New roads identification

A pub-sub system aggregates location streams (device, time, location) from servers. A data analytics engine (e.g. Apache Spark) consumes this data and applies machine learning to predict traffic, identify hotspots, and detect new roads, directly improving ETA calculations.

New road identification is the surprising one — the map improves itself

Traffic and hotspots are expected outputs. Detecting new roads is more interesting.

If many users are repeatedly observed travelling along a path that is not in the graph, the most likely explanation is that a road exists there and the map does not know. The system can infer the road network from movement, rather than only from the third-party data Lesson 4 assumed.

That partially closes a gap Lesson 2 scoped out. The design assumed "road data is provided by external sources" — and this pipeline means the map is also self-correcting from observed behaviour, catching new construction faster than a survey cycle would.

It also explains why the analytics engine needs machine learning rather than simple aggregation. Counting vehicles is arithmetic; distinguishing "a new road exists here" from "GPS error and a few people crossing a car park" is a classification problem.

Applying the results

The data analytics engine publishes results to a dedicated pub-sub topic. The map update service subscribes to these analytics. It updates segment graphs when new roads appear or edge weights (average speed under traffic and road conditions) change. The service identifies the relevant segment, locates its routing server via the key-value store, and applies the update.

The graph preprocessing service recalculates paths on the updated graph, ensuring background path updates reflect live data.

The map update service uses the same key-value store as request routing — and that is the right reuse

"Locates its routing server via the key-value store" is the same segID → serverID lookup Lesson 10 uses for queries.

So the directory serves both paths: reads (which server answers this route?) and writes (which server holds the graph I need to update?).

That is correct and worth noticing, because it means the key-value store is on the critical path of everything. Lesson 12 lists it under low response times; it is equally load-bearing for keeping the map current.

It also means a segment moving servers must update one place, and both readers and writers immediately follow — the same argument that building block made for ZooKeeper holding the primary-replica mapping.

The transitory-versus-persistent distinction is the chapter's best operational insight

"Many traffic conditions are transitory (like stopping at a signal), so updating the graph very often wouldn't scale well because it requires excessive processing. What could be the solution?"

The system maintains two sets of data for the routing graph's edge weights: one reflecting normal (historical or average) conditions, and another capturing transitory (temporary) events like signal stops or short-term congestion.

When calculating routes, the system overlays the transitory data on top of the normal conditions, applying only relevant and recent updates. As transitory events expire, their effects are automatically removed, ensuring only significant and persistent changes impact the underlying graph.

This solves a real problem elegantly. Every car stopping at a red light is a slowdown observation — writing that into the graph and re-running Lesson 5's precomputation would mean continuous, pointless recomputation.

The two-layer scheme separates by durability of the signal:

Base weights change rarely, so precomputed paths stay valid.
Transitory overlay changes constantly but is applied at query time, so it costs nothing when nothing is queried and expires on its own.

That is exactly the materialized-versus-computed trade seen throughout this course: precompute the stable part, compute the volatile part on demand.

And it answers the tension from Lesson 9 — how can you precompute paths when weights keep changing? You precompute against the stable weights and overlay the volatile ones.

WebSockets mean this tier is sized in connections, not requests

"This requires persistent connections between servers and user devices" with a load balancer to manage capacity limits.

An active navigation session holds a connection for the entire journey — potentially hours. So the navigation tier's capacity is concurrent connections, a completely different unit from the 18,518 requests/second Lesson 3 computed for the request tier.

That is why the load balancer is called out specifically: distributing long-lived connections is a different problem from distributing short requests. A connection assigned to a server stays there, so imbalance persists rather than averaging out, and a server restart drops every session on it.

It is the C10K profile again — the same one behind that building block's event-driven web server choice and that building block's long-polling caveat. Many concurrent, mostly-idle, long-lived connections is a recurring shape, and it always argues for event-driven servers over thread-per-connection.

Note the loop is closed — this is a control system, not a pipeline

Trace the cycle:

users navigate -> location streams -> analytics -> updated weights
   ^                                                     |
   +--------- better routes and ETAs <-- recomputation <-+

Output feeds back into input. The system observes the effect of its own routing decisions and adjusts.

That has a property worth stating: it can oscillate. Route everyone away from a congested road and the alternative congests; observe that and route them back. Without damping, a naive controller would flap.

Real systems mitigate this by distributing traffic across several good routes rather than sending everyone to the single best one, and by smoothing weight updates over time rather than reacting instantly.

The Distributed Task Scheduler chapter's aging and that building block's buffer-based bitrate control are the same instinct: react to accumulated state, not instantaneous measurements, or the controller oscillates.

Key takeaway

Location streams flow through WebSockets → pub-sub → analytics → map updates → graph preprocessing, with the analytics engine deriving traffic, hotspots, and even new roads — so the map partially corrects itself. The key insight is the dual-weight scheme: precompute against stable base weights and overlay transitory conditions at query time, so a red light never triggers a graph rebuild. And because output feeds back into input, this is a control loop that can oscillate without damping.

The invalidation step is the hard one, and it is where the chapter's two challenges collide. Precomputation is what makes routing fast; changing edge weights is what makes precomputation stale. The resolution is that distance changes almost never and travel time changes constantly, so the path stays valid while only its weight is refreshed — you re-cost the tile rather than re-solving it.

Interview signal by level

Level	What a strong answer sounds like
L4	"Collect location data through a stream and use it to update traffic estimates."
L5	Traces the pipeline: "WebSockets to navigation servers, into pub-sub, consumed by an analytics engine that derives traffic and publishes results, which a map update service applies to the right segment's graph."
Staff+	Leads with the dual-weight scheme: "you can't write every observation into the graph — a car at a red light would trigger recomputation constantly. So keep two layers: stable base weights that precomputation runs against, and a transitory overlay applied at query time that expires on its own. That resolves the tension between precomputing paths and weights that keep changing. And this is a closed loop, so it can oscillate — routing everyone off a congested road congests the alternative — which argues for spreading traffic across several good routes and smoothing weight updates rather than reacting instantly."

Next: checking the design against the requirements.