Service Discovery
In one line: pull's one structural weakness is that it must know what to pull from. In a fleet that scales itself, that question has no static answer — and getting it wrong means new instances run unmonitored.
The problem
A hand-maintained target list fails in the worst possible way: new capacity is invisible. The instances added during a traffic spike — precisely the ones under most stress — are the ones nobody is watching.
The mechanism
A service discovery component tracks active instances by integrating with platforms such as EC2, Kubernetes, or Consul. The data collector queries this component to determine which targets to scrape, ensuring newly provisioned instances are monitored automatically.
What it costs
| Consideration | Detail |
|---|---|
| A new dependency | The collector now depends on the discovery service being reachable and current |
| Staleness window | An instance is unmonitored between starting and appearing in discovery — short, but non-zero |
| Churn cost | Aggressive autoscaling means a constantly changing target list, and re-resolving it on every scrape cycle |
| Deliberately-gone vs failed | A target disappearing from discovery is a scale-down; a target present but unscrapeable is a failure. Conflating them produces false alerts on every deploy |
The last row causes real alert noise
If the collector treats "target vanished from the discovery list" the same as "target stopped responding," every routine scale-down and every rolling deploy fires alerts.
The distinction matters: discovery says whether an instance is supposed to exist; the scrape says whether it is healthy. Alert on the second, not the first — otherwise your team learns to ignore the alerts, which is worse than not having them.
Key takeaway
Service discovery is what makes pull viable in a dynamic fleet. Derive the target list from the orchestrator that already owns it, and keep "should exist" separate from "is healthy."
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "We configure the list of servers to monitor." |
| L5 | Automates it: "a service discovery component integrated with Kubernetes or EC2, so autoscaled instances get monitored without anyone editing config." |
| Staff+ | Handles the ambiguity: "discovery tells me what should exist, the scrape tells me what's healthy — I'd keep those separate, or every rolling deploy pages someone. And I'd read from the orchestrator rather than keep my own registry, so there's one source of truth." |
Next: turning stored metrics into something a human acts on.