Memory: What to Keep

In one line: an assistant that remembers everything is as useless as one that remembers nothing, so the design question is what gets promoted and what gets dropped.

Three kinds, three lifetimes

The standard division, and it is worth using because the three genuinely behave differently.

Working memory — the current task. What was asked, what has been tried, what came back. Lives for the duration of the loop and is discarded, except for what gets promoted.

Episodic memory — time-indexed records of what happened. "On the 14th, booked a flight to Berlin, user changed the return date." Specific, dated, and useful for a while.

Semantic memory — facts and preferences, independent of when they were learned. "Prefers aisle seats." "Reports to Priya." "Does not want meetings before 10am." These are what make the assistant feel like it knows you.

Consolidation is the interesting operation

Episodes are cheap to record and expensive to keep. The useful move is to distil them: a fact that stays true without its original context moves to semantic memory and the raw episode is dropped.

Three bookings where the user changed to an aisle seat is three episodes. "Prefers aisle seats" is one fact that is smaller, more retrievable, and more useful.

Two design points about doing it well.

Consolidate on evidence, not on a single instance. One occurrence is an event; three is a pattern. Promoting a fact from one observation produces an assistant confidently wrong about your preferences, which is worse than one that has not noticed them.

Keep the provenance. A semantic fact should record roughly how it was learned and how many times it was observed, so it can be revised or shown to the user. A preference the user cannot see or correct is a preference they will eventually be annoyed by.

What not to remember

The under-designed half, and it matters more in a personal assistant than anywhere else.

Things stated as one-offs. "Book me a window seat this time" is explicitly not a preference.

Things about other people. An assistant that learns facts about your colleagues from your email is building a profile of people who never consented to it. That is a privacy decision before a technical one, and the safe default is to remember about the user, not about third parties.

Sensitive categories. Health, finances, relationships. Even where the assistant sees them, storing derived facts about them is a different commitment from processing them in the moment.

Anything the user would be surprised to learn was kept. The workable test, and the one to state in an interview: if showing the user their memory store would embarrass you, the retention policy is wrong.

Where memory lives, structurally

Not one store, because the retrieval patterns differ.

The semantic tier is worth keeping structured rather than as free text. "Prefers aisle seats" as a typed preference on a key is directly retrievable when booking a flight; the same sentence embedded in a blob competes with everything else on cosine similarity and may not surface at all.

That is the practical version of a point that recurs: if you know the shape of the fact, store it in that shape. Embeddings are for things whose shape you do not know.

Memory and the loop

The connection back, and it is a cost argument.

Memory is retrieved into the loop's context, and the loop's context is already growing with observations. So memory retrieval competes for the same budget as the agent's working state — and an assistant that injects twenty remembered facts into every step has spent its context on things that mostly do not matter for this step.

Two consequences. Retrieve memory once per task, not per step, unless the task changes topic. And retrieve few items — the published patterns land on a handful of memories under a small token budget, which is far less than instinct suggests.

The cold start

Worth naming, because it shapes the first weeks of the product.

A new user has no memory, so the assistant is generic exactly when it most needs to demonstrate value. Three responses, in order of cost.

Ask a little, once. A short onboarding capturing a handful of high-value preferences converts weeks of observation into two minutes.

Bootstrap from existing data. Calendar and mail already encode a great deal — working hours, frequent contacts, typical meeting lengths. Deriving those is cheap and needs no explicit input.

Be explicit about not knowing. "I don't know your preference here — is this right?" is better than guessing, and each answer is a memory written with real evidence rather than an inference.

Next: retrieving it, and forgetting.