Dario Kaufmann

Faking Signal in Noisy Metrics

· ~7 minutes · blog

For most of last year, the most-watched number on our team dashboard was a metric that could not tell you anything. It was called pipeline health, it was a weighted mash-up of four other metrics, and it had exactly one property that mattered to its audience: it turned red in a way that correlated with mornings when people were already annoyed about something.

This is the story of how we killed it and replaced it with three boring metrics that lie less. It is not a story about tooling. We changed almost nothing about the tooling.

Where the mush came from

The composite was invented, sincerely, during an incident review. The problem we had was real: four separate dashboards, no shared sense of what bad looks like, and an on-call engineer who had to eyeball all of them under stress. Someone said “we need one number that just tells us if things are okay,” and the composite was born.

The weights were chosen in that meeting. They were never revisited. They could not be revisited, because nobody could reconstruct why freshness got 0.4 and error rate got 0.3. The metric was a fossil of one afternoon's opinions, and it had been promoted to the truth.

What the composite hid

Composite metrics have a failure mode that I have not seen named often enough: compensation. When two inputs move in opposite directions, the average moves a little, or not at all, and the alarm does not fire. We had at least two mornings where a backlog was quietly doubling behind a green number, because freshness (good: recent events were fresh) was canceling out volume (bad: the recent events were the only ones moving).

The other thing it hid was why. When the number went red, the first question was always “okay, but which component?” — which meant going back to the four dashboards anyway, now in a worse mood. A metric that requires a second investigation to interpret is not a signal, it is a doorbell.

The replacement, and why it worked

We deleted pipeline health and put three plain metrics in its place, each with one job:

  1. Lag in minutes, age of oldest unprocessed event. Not a rate. An age. Ages are monotonically honest: if the oldest event is 40 minutes old, that sentence is true no matter how the fleet feels about itself.
  2. Failure ratio per stage, windowed over 15 minutes. Per stage, because a global ratio lets a healthy majority hide a dead minority. We were already collecting the inputs; the change was only refusing to sum them.
  3. Throughput as a range, not a line. We plot the p10–p90 band and alert when today's band sits outside last Tuesday's band. Business is seasonal; a hard threshold on volume was alerting every quiet Sunday.

None of these are clever. That is the point. Each one answers a question a human actually asks — is anything stuck, what is broken, is the shape wrong — and each can be interrogated alone, without dragging three others into the interpretation.

The part nobody argues with and everybody skips

When we decommissioned the composite, we also wrote down, next to each new metric, the question it answers and the action its alert triggers. That one paragraph has settled more arguments than the metrics themselves. If you cannot write the sentence “when this fires, we do X,” you have not designed an alert, you have designed a mood.

The checklist we use now, for any proposed dashboard addition:

Postscript

The composite still exists, in one place: a weekly report nobody has opened since March. Deleting it turned out to require a decision about who owns the report, which turned out to require a decision about whether the report's consumer still works here. The metrics were the easy part. They usually are.

If you have a composite metric you are fond of, do not delete it because a stranger on the internet is smug about composites. Delete it when you notice that its red mornings are the ones you had already heard about by other means. That gap between “the metric knew” and “you already knew” is the whole audit.

Next: Three Weeks of Maintaining.