One Error Log Pinged Me 400 Times: Two-Tier Log Alerting That Tells 'It Happened' Apart From 'It Is Happening'
Overview At 1:12 a.m. on a night in August, my phone started buzzing and would not stop. Checkout was broken on an e-commerce platform. A downstream connection pool behind the payment gateway had been saturated, the checkout service retried in a tight loop, and the logs filled up with PAYMENT_GATEWAY_TIMEOUT. The on-call engineer found the root cause within seven minutes, reset the pool ceiling and the retry count, and traffic was back to normal by 1:59 a....