Every quarter, an on-call team sits down with a dashboard showing tens of thousands of weekly alerts and concludes the fix is a filter. They write suppression rules, tighten thresholds, add a silence window for the flappy check. For a few weeks the pager is quiet, the retro goes well, and everyone moves on. Six weeks later the volume is back — not because the rules were wrong, but because filtering is a subtraction applied to a system that keeps producing. The noise was never sitting in the alert pipeline. It was sitting in the team, as a backlog of decisions nobody had assigned.

Suppression rules decay because they encode a moment, not a policy

Almost every suppression rule is a fossil of one bad night: a check that flapped during a deploy, a threshold that fired during a backfill. The rule captures that moment precisely and generalizes to nothing. Do the arithmetic on your own team: two new suppression rules a week is a modest, unremarkable pace, and it leaves you with 104 rules by year end. Ask three engineers to explain 40 of them and watch what happens. The rules that survive are rarely the ones doing the most work — they are the ones nobody dared delete. Reduction by accretion has a half-life measured in weeks, and every rule you cannot explain is a rule you cannot safely remove.

The unit of noise is the undecided alert, not the alert

Take a 60-engineer platform team generating 9,000 alerts in a week. Break that number apart by what actually happens next. Perhaps 6,300 auto-resolve before anyone looks. Another 1,800 are duplicates of an incident already open. Roughly 900 require a human decision. The 9,000 is what the dashboard shows and what the meeting is about; the 900 is the actual workload. Everything else is transport. This reframing matters because it changes the success criterion: good noise reduction drives the 8,100 down without losing a single one of the 900. Filters that cut volume while dropping a real signal are not reduction, they are sampling error with a changelog.

An alert that doesn't name a decision, an owner, and a deadline isn't monitoring — it's ambient anxiety.

Three questions every alert must answer before it can route

Before an alert is allowed to choose a transport, it should be able to answer all of the following. If it cannot, it is a log line with ambition.

  1. What decision does this require? Page someone, open a ticket, write to a dashboard, or drop it. Four answers, no fifth.
  2. Who owns that decision this week? A named rotation, not a team alias. "Platform" is not an owner; it is a place where decisions go to wait.
  3. What is the deadline? Is this "before the error budget burns through in 40 minutes" or "by Friday"? The deadline determines the transport, not the severity label.

Route by decision type, and the numbers fall out

Once alerts carry those three fields, routing stops being a judgment call. In most estates, the share of alerts with a genuine sub-30-minute deadline is in the single digits — the rest are next-business-day work wearing a P1 costume. Page for the first group. Ticket the second. Chart the third. Drop the fourth, but only with a counter attached.

Then collapse what shares a cause. Take last week's alerts and group them by service plus failure mode. Count the groups. Teams that do this for the first time usually find the number of distinct failure modes across a mid-sized estate is in the low hundreds — a number small enough to reason about, review, and eventually automate. That count, not the raw alert total, is your real incident surface.

Suppression without a receipt is how blind spots are born

Every suppressed alert should emit a metric that lands on a dashboard within 24 hours of the rule being written. A rule with no counter is a bet you cannot settle. The practical version is a ledger: rule, author, date written, review date, and the count of alerts it has swallowed since. Rules that swallow zero get deleted. Rules that swallow thousands get investigated, because a rule eating that much traffic is describing a system problem nobody has fixed. This is the difference between noise reduction and noise hiding, and it is the only audit trail that will survive contact with an incident review.

Alerts are becoming decision contracts

The direction of travel is obvious once you accept the framing. If the noise is undecided alerts, then the durable fix is not better filtering — it is deciding faster, and encoding the decisions that turn out to be deterministic. A restart that always works, a certificate that always renews, a queue that always drains when workers are recycled: these do not need a page, they need a handler and a record. What remains is a much smaller set of alerts where a human genuinely has to choose, and those deserve a clean page, a named owner, and a deadline. Teams that get there stop having alert-reduction projects. They have a decision inventory instead, and it shrinks on purpose.