The Newsfeed Publishing Service

In one line: this service does one thing — turn <Post_ID, User_ID> tuples into something a phone can render — and that one thing is where every "store references, not values" decision in the chapter gets paid for.

What it does

At this stage, the newsfeeds are generated and stored in the form of <Post_ID, User_ID> in the newsfeed cache.

The newsfeed publishing service fetches a list of post IDs from the newsfeed cache. The data fetched is a tuple of post and user IDs. Therefore, the complete data about posts and users is retrieved from the users and posts cache to create an entirely constructed newsfeed.

In the last step, the fully constructed newsfeed is sent to the client using one of the fan-out approaches. The popular newsfeed and media content are also stored in CDN for fast retrieval.

Hydration is the price of storing references, and it is worth paying

The stored feed is a list of tuples. The rendered feed needs post text, author name, avatar, timestamps, like counts, and a media thumbnail. Hydration is the step that bridges them.

STORED:    [<P1,U9>, <P2,U4>, <P3,U9>, ...]         ~16 bytes per entry
                    |
                    v  hydrate from post cache + user cache + blob store
                    |
RENDERED:  [{text, author, avatar, counts, thumb, link}, ...]   ~KB per entry

That is a real cost — a feed of 20 items means fetching up to 20 posts and their distinct authors. But it buys three things that make the whole design work:

Storage. Lesson 3's correction: references are the difference between 56 PB and a fraction of a petabyte.

Freshness. Like and comment counts change constantly. Hydrating at read time means the counts are current; baking them into feeds would mean rewriting hundreds of millions of stored feeds every time someone clicks like.

Correctness on delete. A removed post is filtered during hydration — one check — rather than scrubbed from every feed containing it.

The cost is manageable because of batching and skew. Twenty tuples become two batched multi-gets, not forty round trips. And the same popular posts appear in millions of feeds, so the post cache hit rate is extremely high — the skew that makes fan-out expensive is what makes hydration cheap.

Storing references converts a storage problem into a read-time join, and the join is affordable precisely because the data is skewed.

'Sent to the client using one of the fan-out approaches' — the six words this chapter owes

This is the only mention of fan-out in the design, and it is both unexplained and misplaced.

Unexplained: the approaches are never named or compared. Lesson 4 exists because of this sentence.

Misplaced: fan-out is not about sending a constructed feed to a client. Fan-out is about how a post reaches many users' feeds — a write-path concern that happens long before publishing. By the time the publishing service is running, the fan-out decision was made when the post was created.

The sentence conflates two different deliveries:

FAN-OUT:   a new post  ->  many users' feeds        (write path, Lesson 4)
DELIVERY:  one user's feed  ->  their device        (read path, this service)

The second is not fan-out at all — it is a response to a request, plus optionally a push to devices with an open connection.

That second half is worth designing though, and the design does not: does the client poll on refresh, or does the server push updates over an open connection? The chapter's own Q&A says "the updated newsfeed is delivered on the next page/screen refresh event" — so pull, client-initiated. Which is right for a feed: unlike a ride offer or a chat message, nobody needs a post the instant it exists.

What the CDN caches here — and what it cannot

"The popular newsfeed and media content are also stored in CDN for fast retrieval."

Media on a CDN is unambiguously right, and Lesson 3 explains why: it is nearly all the bytes, it is immutable, and it is shared across millions of feeds. A viral video should be served from an edge, once per edge.

"Popular newsfeed" on a CDN is the questionable half. A newsfeed is personalized by construction — it is the ranked merge of what this user follows. Two users share no feed, so there is nothing to cache at a shared edge.

The Twitter chapter reached the same conclusion for the same reason. What a CDN can genuinely serve:

So the accurate statement is that the CDN caches the popular items feeds are made of, not the feeds. That is still a large win — Lesson 9's skew means a small number of posts appear in a huge share of feeds — but it is a different claim.

Personalized content cannot be cached at a shared edge; its components often can.

The update path: what happens when Bob posts

The design's Q&A:

When Bob creates a new post, it is stored in the post database and cache. Then the newsfeed generation service generates a newsfeed for Bob's friends and followers, and the newsfeed cache is updated. The updated newsfeed is delivered on the next page/screen refresh event.

Three things this confirms.

Push fan-out is the default. Generation runs when Bob posts, not when his followers read — Lesson 4's push path.

Feeds are regenerated, not appended. "Generates a newsfeed" rather than "inserts the post" — which is more expensive but keeps ranking coherent, since a ranked feed cannot simply have an item prepended.

Delivery is pull. Nothing is pushed to devices; the updated feed waits for a refresh. Which means the effective staleness a user sees is fan-out lag plus time-until-refresh, and the second term dominates. Optimizing fan-out from 10 seconds to 1 second changes nothing for a user who refreshes every few minutes.

Optimize the term that dominates. A common error is tuning the fast half of a two-part latency when the slow half is a user behaviour you do not control.

Next: the ranking service, where the hard problem lives.