The Vocabulary Problem

In one line: someone asks about PROJECT FALCON, and the embedding model has never encountered those words together in its life.

Why this bites harder here

A web search engine and its users share a public vocabulary. The words in a query appear in the documents, and an embedding model trained on the internet has seen both.

An organisation speaks a private language. Project codenames, internal system names, acronyms that mean something different outside, team names, product names that predate a rebrand, and people referred to by first name only. None of it is in any pretrained model's training data, and much of it is not even in the documents you indexed — because insiders write the acronym and never the expansion.

That conclusion inverts the usual instinct. In a consumer product the semantic side handles the conversational tail; here it is the lexical side that carries the queries that matter most, because internal terms are exact strings and exact strings are what BM25 is good at.

Four kinds of internal vocabulary

Each fails differently and needs a different fix.

Acronyms with an internal meaning. PR might be pull request, purchase requisition, or public relations depending on who asked. Expansion is genuinely ambiguous and the disambiguating signal is usually the asker's team.

Codenames. FALCON has no semantic content whatsoever. Nothing about the word relates to what the project does, so no amount of semantic modelling helps — it can only be looked up.

Renamed things. The product was called one thing for three years and something else since. Half the corpus uses the old name, and a query using either should find both.

People. "What did Sarah decide about the migration?" requires resolving a first name to a person, which requires knowing who the asker is likely to mean — usually someone on their team.

Building the lexicon

The asset that solves this, and the good news is that most of it can be mined rather than authored.

From the corpus itself. Documents frequently define their own terms on first use: "the Fleet Allocation Service (FALCON)". A pass over the corpus extracting parenthetical expansions and glossary pages produces a large fraction of the lexicon for free.

From structured systems. Project trackers, service catalogues and the directory contain authoritative name-to-thing mappings, including former names. This is usually the highest-quality source and it is already maintained by someone.

From reformulation chains. Someone searches FALCON, gets nothing useful, then searches fleet allocation. That pair is a labelled synonym, generated at no cost — the same mining technique as consumer search, and it works better here because the vocabulary is small and repetitive.

From people. A lightweight way for anyone to add a term. Small effort, high value, and it converts the people who understand the jargon into the maintainers of the lexicon.

Where to apply it

Not everywhere, and the placement matters.

Query expansion, gated. Expand an internal term to its expansion and its synonyms — but only when the original returns too little, and never when the query is already specific. That is the same gate as any expansion, and it matters more here because a codename is exactly the kind of precise term that should not be broadened when it is working.

Indexing. Store both the term and its expansion in the indexed text, so a document mentioning only the acronym also matches the full name. This is often better than query-side expansion because it is applied once rather than per query.

Disambiguation by asker. PR from an engineer and from the communications team should expand differently. The user's team is available and it resolves most ambiguity — one of the clearest uses of the org structure outside ranking.

The embedding model question

The obvious suggestion is to fine-tune embeddings on the internal corpus so the model learns the vocabulary. Worth addressing directly, because it comes up and it is usually the wrong first move.

Three reasons to be cautious.

It is a one-way door with a maintenance tail. Changing the embedding model means re-embedding the corpus, and now you own a model that must be retrained as the vocabulary evolves.

Codenames cannot be learned semantically anyway. FALCON has no relationship to fleet allocation that any amount of training can discover from the token — it is an arbitrary label, and the mapping is a lookup, not a representation.

The cheap thing usually wins. A lexicon plus a strong lexical retriever handles internal vocabulary well, and it is inspectable, instantly editable and free to maintain.

The defensible position: start with the lexicon and hybrid retrieval, measure how much residual failure is genuinely semantic rather than terminological, and only then consider fine-tuning. In most enterprise deployments the residual is small.

Acronym collisions with the outside world

One specific failure worth naming.

Some internal acronyms collide with common public terms. An embedding model will confidently interpret them in their public sense, so a query about an internal system returns documents about something unrelated that happens to share the letters — and it does so confidently, with a high similarity score.

That is why the lexicon has to take precedence rather than being one signal among many: where an internal definition exists, it is authoritative, and the pretrained model's opinion is noise.

The measurement

How to know whether this is your problem, since it is easy to assert and easy to check.

Sample queries that produced no click, no answer or an abstention. Classify each: was the failure terminological — the system did not know what the term meant — or was it something else? The proportion tells you directly how much of your quality gap is vocabulary, and in enterprise deployments it is frequently the largest single category.

That measurement is worth proposing in an interview, because it converts a plausible claim into a prioritisation decision.

Next: ranking with none of the usual signals.