Connectors Are the Product

In one line: the interesting part is retrieval and the expensive part is getting the documents out of eight systems that were never designed to be crawled.

What a connector has to do

Six jobs, and most designs assume the first and forget the rest.

Authenticate, usually as a service account or through delegated permissions, with credentials that expire and scopes that must be justified to a security team.

Enumerate everything visible, which for a system with millions of items means paginating through an API that was designed for interactive use.

Fetch content, in whatever format the system stores it.

Fetch permissions, which is a separate call in most systems and frequently a much more expensive one.

Detect change, so subsequent runs do not re-read everything.

Handle deletion, which is the one nobody plans and which matters most — a document deleted at the source and left in the index is a document you are serving that no longer exists, possibly because someone deliberately removed it.

Each of those differs per system, which is why connectors do not generalise as much as their interface suggests.

The sources are not alike

Three observations that shape the design.

The unit of retrieval is not obvious. A single chat message is usually meaningless out of context; the useful unit is a thread or a window of conversation. A ticket without its comments is half the story. Deciding the retrievable unit per source is a real design decision, not a default.

Change rates differ by orders of magnitude. Chat produces more events per hour than a document store produces per week. One sync cadence across all sources either hammers the slow systems or leaves the fast ones stale.

Email is a special case worth flagging. It is enormously valuable and enormously sensitive, and its permission model is unlike everything else — each message has its own recipient set rather than inheriting from a container. Many deployments exclude it for exactly that reason, and saying so is better than silently indexing it.

Rate limits are the binding constraint

The practical reality that surprises people.

Source systems have rate limits designed for interactive use, not for a crawler enumerating everything. Exceeding them degrades the source system for actual users — Atlassian, among others, has warned publicly about exactly this — so an aggressive initial crawl can slow down the tool everyone is trying to work in.

Four consequences:

The initial crawl takes a long time. Days to weeks for a large deployment, and that is a project-planning fact rather than a bug.

Backoff must be respectful rather than merely correct. Retrying at the limit forever still consumes the source's capacity.

Prioritise the first pass. Index the most-used spaces and most-recent content first, so the product becomes useful before the crawl finishes.

Budget the ongoing sync. Steady-state crawling has an ongoing cost in someone else's API quota, and that quota is shared with the humans using the product.

Incremental sync, and the permission trap

The sharpest detail in this lesson, and it is the kind of thing that separates someone who has built this from someone who has read about it.

Incremental sync uses a change feed — modified-since timestamps, a change token, a webhook — to fetch only what moved. Straightforward for content.

Permissions frequently do not appear in the change feed. Several major connectors process ACL changes only during a full crawl, because the source system's change API reports content modifications and not permission modifications.

The consequence is severe: someone is removed from a project, the content did not change, so the incremental sync sees nothing — and the index still says they can read it until the next full crawl, which might be weekly.

Three mitigations, and a design should name at least one.

Sync permissions on their own schedule, more often than content, since ACLs are small and change independently.

Subscribe to identity-side events — a departure, a group change, a role change — from the directory rather than from each content system. One authoritative source of membership changes covers all sources at once.

Verify at query time for the most sensitive material. Which is late binding, and it gets its own lesson.

Failure and observability

Connectors fail constantly and quietly, which is what makes them an operational problem rather than a build-once one.

Credentials expire. APIs change. A source system is upgraded. A rate limit tightens. A space is renamed. And the failure mode is usually silent: the connector keeps running, indexes nothing new, and the index simply stops being current.

Per-connector monitoring is therefore not optional, and the useful signals are specific: age of the newest document from each source, sync success and duration, item counts against expectation, and error rates by type.

The first is the one to alarm on, for the same reason as feature freshness: a connector that succeeds having fetched zero items looks perfectly healthy on every other metric.

The build-versus-buy question

Worth raising, because it is a genuine judgement call and interviewers respond to it.

Connectors are commodity work with a long tail of edge cases, and vendors sell them. The argument for building is control over the permission model and the extraction quality; the argument for buying is that fifteen connectors is a team's ongoing job, forever.

A defensible position: buy the connectors for standard systems, build for whatever is proprietary or unusually sensitive, and own the permission normalisation layer regardless — because that is where correctness lives and it is not something to delegate.

Next: the permission model itself.