Service Discovery
Why this matters: pull's one structural weakness is that it must know what to pull from. In a fleet that scales itself, that question has no static answer — and getting it wrong means new instances run unmonitored.
Key takeaway
To pull metrics, the collector must discover the network endpoints of active service instances. Because services scale dynamically, static target lists are insufficient.
The problem
A hand-maintained target list fails in the worst possible way: new capacity is invisible. The instances added during a traffic spike — precisely the ones under most stress — are the ones nobody is watching.
The mechanism
A service discovery component tracks active instances by integrating with platforms such as EC2, Kubernetes, or Consul. The data collector queries this component to determine which targets to scrape, ensuring newly provisioned instances are monitored automatically.
What it costs
| Consideration | Detail |
|---|---|
| A new dependency | The collector now depends on the discovery service being reachable and current |
| Staleness window | An instance is unmonitored between starting and appearing in discovery — short, but non-zero |
| Churn cost | Aggressive autoscaling means a constantly changing target list, and re-resolving it on every scrape cycle |
| Deliberately-gone vs failed | A target disappearing from discovery is a scale-down; a target present but unscrapeable is a failure. Conflating them produces false alerts on every deploy |
Key takeaway
Service discovery is what makes pull viable in a dynamic fleet. Derive the target list from the orchestrator that already owns it, and keep "should exist" separate from "is healthy."
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "We configure the list of servers to monitor." |
| L5 | Automates it: "a service discovery component integrated with Kubernetes or EC2, so autoscaled instances get monitored without anyone editing config." |
| Staff+ | Handles the ambiguity: "discovery tells me what should exist, the scrape tells me what's healthy — I'd keep those separate, or every rolling deploy pages someone. And I'd read from the orchestrator rather than keep my own registry, so there's one source of truth." |
Next: turning stored metrics into something a human acts on.