The average enterprise network spent the last three years adding telemetry: gNMI streaming subscriptions, sFlow, eBPF-enriched flows, syslog from every switch, and an SNMP poller nobody remembered to turn off. Data volume rose by roughly an order of magnitude. Mean time to identify a network fault did not move. That gap is the article: acquisition got cheap, and almost nobody rebuilt the layer above it. Network observability stopped being a collection problem and became a join problem — and the joins are still done by a human with four browser tabs open at 3 a.m.

Sample rate is arithmetic, and most teams never run it

Take a mid-size estate: 2,000 devices across a campus, two data centers, and a WAN edge. Assume 48 interfaces per device, 400 counters per interface, and a 10-second streaming subscription — all conservative for a modern deployment. That's 2,000 × 48 × 400 ÷ 10 = 3.84 million samples per second, or roughly 332 billion samples a day. Compress each to 2 bytes in a columnar store and you're writing about 660 GB/day before indexes, replicas, and dashboard read amplification. A year of raw retention lands near a quarter petabyte.

Now pull your own query logs. In most estates, fewer than 3% of emitted series are read on any given day, and the overlap between "series that ever alerted" and "series retained at full resolution" is a rounding error. The storage isn't the sin. The sin is that the collector has no idea which samples will matter, so it keeps all of them and charges you for not deciding.

Cardinality is billed per series, and series come from people typing

Metric names are cheap; label combinations are not. ifInOctets across 2,000 devices × 48 ports is 96,000 series — trivially manageable. Add three labels that feel harmless — interface_description, bgp_peer_as, vlan_name — and the same counter becomes 600,000 to 1.2 million series inside a year, because description strings get edited during every recabling and old series linger in the index until retention expires. The device count never changed. The humans typing into the config did.

The rule that survives contact with reality: label values must come from closed sets — device role, site, vendor, protocol, interface type. Anything a person can free-type into a running config belongs in an exemplar, a log line, or a change event, never in a label. Enforce it in the pipeline, not in a style guide, because the pipeline is the only reviewer that never gets overruled during an outage.

"What changed?" is the only query that matters at 3 a.m.

The on-call engineer does not need to know that Gi1/0/17 is at 84% utilization. They need to know a config commit landed on that switch 11 minutes ago and touched the uplink's MTU. That answer requires a join: change events — CI/CD config pushes, firmware upgrades, BGP session transitions, link state flips, routing policy commits — joined to telemetry on a shared topology key, with timestamps on the same clock. When the join exists, the page usually resolves itself: a single correlated change event retires more pages than a quarter of threshold tuning, because most pages are symptoms of a change rather than independent faults.

Most stacks can produce the change list and the metric list. Very few produce the intersection without a human exporting CSVs.

Observability isn't the ability to see everything. It's the ability to answer "what changed" in under a minute.

Alert count is a symptom; blast radius is the unit of work

A line card fails on a distribution switch carrying 40 VLANs. Per-interface and per-VLAN thresholds do exactly what they were told: 40 alerts fire in 90 seconds, and the on-call spends the first ten minutes triaging duplicates. Group by topology — shared physical parent, shared change window, shared dependency path — and the same failure becomes one incident with a stated blast radius of 1,200 endpoints. The difference isn't a better threshold. It's a different unit of work.

Tier retention by the question each tier answers

Retention organized around age is a guess. Retention organized around questions is a design:

  1. Raw, 15-second resolution, 7 days. Question: what exactly happened during this incident?
  2. Rolled up, 1-minute resolution, 90 days. Question: is this a regression or a one-off?
  3. Downsampled, 5-minute resolution with change-event overlays, 13 months. Question: did the thing we shipped in March actually help?

Run the byte math on your own estate and you'll find roughly 90% of stored bytes sit in the tier you query least — tier three, which exists for quarterly capacity reviews and post-incident archaeology. That's a fine trade, as long as you make it deliberately instead of inheriting it from a default in a collector config file.

The network is becoming a state machine, and observability becomes a diff engine

The next two years of this market are not about capturing more. They're about stating intent — desired config, desired paths, desired policy — and continuously diffing observed state against it. Once change events, desired state, and telemetry share a topology model, an alert stops being a threshold breach and becomes a deviation with a timestamp and an owner. The teams that arrive first spend their on-call hours on the two questions that genuinely need a human: is this deviation acceptable, and what should the system do about it on its own.