The p99 Was Lying to Us

Note: This is demonstration content. The team, service names, and all figures below are fictional and created to illustrate the publishing setup.

For most of a quarter, our latency dashboard was green and our support inbox was not. The p99 for the API sat comfortably under our 400ms target, and yet a steady trickle of people told us the app felt sluggish. Both things were true at once, which is usually a sign that the number on the wall is answering a different question than the one you're asking.

Averaging percentiles does not work

The bug was in how we rolled the data up. Each service instance computed its own p99 over a one-minute window, and the dashboard averaged those p99 values across instances.

That operation is meaningless. A percentile is a position in a distribution, not a quantity you can add and divide. If nine instances are idle and one is on fire, averaging their p99s produces a comfortable number that describes no actual user.

The fix is to aggregate the underlying histogram buckets and compute the percentile once, at the end:

// Wrong: each instance reports a pre-computed p99, then we average.
const p99 = mean(instances.map((i) => i.p99));

// Right: merge the raw histograms, then read the percentile off the total.
const merged = instances.reduce(
  (acc, i) => mergeHistograms(acc, i.histogram),
  emptyHistogram(),
);
const p99 = quantile(merged, 0.99);

Once we did that, the p99 jumped by roughly a factor of three and stopped disagreeing with the support inbox.

The tail belongs to a specific group of people

The second thing we had wrong was treating the tail as bad luck — a random unlucky percent of requests. It wasn't random. When we broke the slow requests down by account, the same accounts kept appearing.

Those accounts had more data. Slow requests correlated with collection size, which meant the tail wasn't noise scattered across the user base; it was a specific group of customers having a consistently bad time. For them, the p99 wasn't a tail at all. It was the typical experience.

This reframes what a latency target even means. "99% of requests are fast" sounds like a strong guarantee until you notice the failing 1% is the same set of people every day.

What we changed

Three things, in order of how much they helped:

1. Aggregate histograms, never percentiles. This was a one-line change to the metrics pipeline and it made every dashboard downstream honest.
2. Slice by account size. A single global percentile hides the structure that matters. We now chart the tail for the largest accounts separately.
3. Alert on the p99 of the worst cohort, not the p99 of everything. If our biggest customers are slow, we want the page, even when the global number looks fine.

None of this made the service faster on its own. It made the measurement match reality, which is the part you need before performance work can be aimed at anything useful.

The general lesson

A metric can be individually correct at every step and still produce a number that means nothing. Ours was: each instance's p99 was accurate, the averaging code had no bug, the dashboard rendered what it was given. The error lived in the assumption that percentiles compose, and no test was ever going to catch that.

When a metric and your users disagree, the users are reporting the system. The metric is reporting your pipeline.