Message Queue Monitoring in Practice: Kafka Lag, RabbitMQ Backlog, and Consumer Latency Alerting
Overview 3 AM, your phone is buzzing with alerts. The Kafka Consumer Lag for some consumer group jumped from 200 to 500,000. Downstream real-time dashboards are frozen. The data team is pinging you non-stop. You scramble onto the jump host, run kafka-consumer-groups.sh --describe, and see a column of alarming Lag numbers. Over the next half hour, you need to answer three questions: which partition is backed up? Is the consumer dead or just slow?...