Interview Walkthrough: Sizing an LLM Assistant Live

The prompt: "You're building an AI assistant inside a B2B SaaS product. Size it."

Step 0 — Get the four numbers you need

Candidate: "I need four things: how many users, how often they use it, how long a conversation runs, and whether we're using a hosted API or self-hosting. Can I assume 50,000 seats, and that this is knowledge-assistant style — questions against the customer's own documents?"

Interviewer: "Yes. Assume 50,000 seats, and let's say it's self-hosted for data-residency reasons."

Self-hosted is a significant constraint and worth acknowledging immediately, because it means the answer must be in GPUs rather than in dollars per token.

Candidate: "That changes the shape of the answer — I'll size a fleet rather than an API bill, and GPU memory will probably be the binding constraint rather than compute. Let me establish traffic first."

1. Traffic

50,000 seats
x 30% daily active                =  15,000 DAU
x 3 conversations per active day  =  45,000 conversations/day
x 6 turns each                    =  270,000 model calls/day

Spread over a working day, not a full day — this is B2B, so demand is concentrated:

270,000 / (8 x 3,600)  =  ~9.4 calls/sec average
x 3 peak factor        =  ~28 calls/sec at peak

Candidate: "I'm using a 3× peak factor because business traffic clusters — a bad assumption if there's a nightly batch, so tell me if there is."

2. Tokens

Candidate: "This is retrieval-augmented, so the prompt dominates."

System prompt                          500
Retrieved context, 6 chunks x 400    2,400
Conversation history                 varies
User question                          100
                                    -------
Turn 1 prompt                        ~3,000
Completion                             300

Now the moment that carries signal:

Candidate: "History is where this gets interesting. Each turn re-sends everything before it, so cost grows with the square of turn count, not linearly. At 300 tokens added per exchange over 6 turns, total history sent across the conversation is roughly 300 × 6 × 7 / 2, about 6,300 — so average prompt across the conversation is nearer 3,500 than 3,000. That's a 17% underestimate if I'd done it linearly, and at 20 turns it'd be closer to double."

Per conversation:  6 turns x ~3,800 tokens  =  ~23,000 tokens
Per day:           45,000 x 23,000          =  ~1.04e9 tokens/day

3. Memory — and the number that changes the design

Candidate: "Self-hosted, so I need to pick a model class. For document Q and A over retrieved context, a 7B-class model is usually sufficient — this is comprehension, not deep reasoning. Let me check whether it fits."

7B at FP16          =  14 GB weights
80 GB card - 14 - ~5 runtime  =  ~61 GB for KV cache

Context ~3,500 tokens, ~0.05 MB/token for a 7B model
  = ~0.18 GB per in-flight request
  61 / 0.18  =  ~340 concurrent requests per GPU

Candidate: "28 calls per second at maybe 8 seconds each is around 220 concurrent — so one GPU is nominally enough for peak. That's suspiciously tidy, so I want two GPUs minimum for availability, and I'd check the throughput bound separately rather than trust the memory bound alone."

Candidate: "Good thing I checked — throughput binds, not memory. Roughly 4 GPUs for peak decode, call it 6 with prefill headroom, and 8 across two zones for availability."

Interviewer: "What if they want 128K context for whole-document analysis?"

That changes the answer completely rather than incrementally. At 128K, per-request cache goes to about 6.4GB, so one card holds nine concurrent requests instead of 340 — a 35-fold reduction. At 220 concurrent I'd need roughly 25 GPUs on memory alone, and memory would become the binding constraint instead of throughput. I'd push back and ask what fraction of requests actually need it, then route: a small long-context pool for the few percent that do, and the normal pool for everything else. Sizing the whole fleet for the worst case would be several times more expensive for a feature most requests never use.

That is the highest-value exchange in the whole walkthrough, and it comes from one number.

4. Cost

8 GPUs x ~$2/hour x 730 hours  =  ~$11,700/month

Candidate: "Then compare against hosted. A billion tokens a day is 30 billion a month; at a small-model price around $0.30 per million that's roughly $9,000 — so hosted is actually competitive here, and self-hosting is being chosen for data residency rather than cost. Worth saying out loud, because it tells us the fleet only needs to be defensible, not optimal."

5. Retrieval sizing

Candidate: "One more component to size — the index."

Per customer: ~50,000 documents x 10 pages x 2 chunks  =  1,000,000 chunks
x 768 dims x 4 bytes                                   =  ~3 GB raw
x 1.75 HNSW overhead                                   =  ~5 GB
quantised int8                                         =  ~1.3 GB

Candidate: "Comfortably small per customer — and since this is B2B with strict data separation, I'd shard the index per tenant anyway. That gives isolation for free and means no single index ever gets large. Across 500 customers that's around 650GB total, spread over shards that each fit trivially in memory."

Sharding by tenant being both the compliance answer and the scaling answer is a good note to end the sizing on.

6. The levers, ranked

Candidate: "If I had to cut this, in order of impact: cap conversation history, because that's the quadratic term and it's the only lever whose benefit compounds. Then trim retrieved context from six chunks to four and measure the quality cost. Then batch aggressively, since decode is memory-bandwidth-bound and batching is close to free throughput. I would not start with the vector index — it's a few percent of the bill."

Now do it live