← sources

What’s the worst production failure your monitoring didn’t catch?

source: reddit (reddit:devops)published 2026-10-21 02:49ingested 2026-10-22 02:38

retained text

Curious how other developers handle cases where everything technically reports success — HTTP 200, Lambda completes, webhook delivered — but the intended operation never actually finishes.

For example, a payment succeeds but the customer’s subscription isn’t activated, or an API call succeeds but a downstream system never gets updated.

Have you encountered anything like this in production? How did you discover it, and what tools or processes do you use to catch these failures?

I’m especially interested in cases where traditional logs, metrics, or error alerts weren’t enough.

document fetched 2026-10-22 00:26 · canonical https://reddit.com/r/devops/comments/1xbbpob/whats_the_worst_production_failure_your