According to Marc Brooker, an engineer at Amazon Web Services, there's a statistical phenomenon called the inspection paradox that explains why your service metrics often don't match what customers experience. If your service has a mean request latency of one hundred milliseconds, but a user measures their mean wait time at one second, you're both right. Customers don't experience the raw distribution of latencies—they experience a time-weighted version, spending most of their time waiting for slow requests. The same holds for outages: if your mean time to recovery is under a minute but customers report seeing outages last around an hour, again you're both right. Brooker explains that tail latencies—those rare, long-running requests—dominate the customer experience far more than aggregate metrics suggest. In one example, if your median recovery time is thirty minutes and your ninety-ninth percentile is ten hours, your service shows a mean of just over an hour, but customers experience a mean recovery of around six hours.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton