Message Queue Monitoring in Practice: Kafka Lag, RabbitMQ Backlog, and Consumer Latency Alerting

Overview

3 AM, your phone is buzzing with alerts. The Kafka Consumer Lag for some consumer group jumped from 200 to 500,000. Downstream real-time dashboards are frozen. The data team is pinging you non-stop. You scramble onto the jump host, run kafka-consumer-groups.sh --describe, and see a column of alarming Lag numbers. Over the next half hour, you need to answer three questions: which partition is backed up? Is the consumer dead or just slow? Will adding more machines help?

Message queues are the “plumbing” of distributed systems—nobody thinks about them until they clog, and then the whole building stops working. Consumer Lag is the flow meter on that plumbing. It tells you the gap between how fast producers are pouring water in and how fast consumers are pumping it out. If the gap keeps growing, something is wrong. If the gap suddenly drops to zero, that might also be a problem (the consumer died, offsets stopped committing, and Lag looks “normal” by accident).

This article covers monitoring for three mainstream message queues—Kafka, RabbitMQ, and Redis—from metrics collection, Prometheus rules, alert thresholds, to troubleshooting approaches. All battle-tested in production. No theory, just configs.

Why Consumer Lag Is the Core Metric

Let’s clarify what Lag actually is. In Kafka, each partition has a LogEndOffset (LEO, the latest message position), and each consumer group has a CurrentOffset (the committed position). The difference is Lag:

Lag = LogEndOffset - CurrentOffset

A Lag of 0 means the consumer has fully caught up. But small fluctuations are normal in production—a per-partition Lag under 100 is usually fine. What you should really watch for are three patterns:

The third one is the most deceptive. I once saw a case where the consumer thread deadlocked, but auto.commit.enable=true kept auto-committing offsets. Messages were marked “consumed” but never actually processed. Lag showed 0, the business team thought everything was fine, until downstream discovered three days of missing data.

Takeaway: Lag alone is not enough. You must look at it alongside the Lag growth rate and consumption rate. A queue with Lag=10000 but dropping at 500/sec is far healthier than one with Lag=200 but climbing at 10/sec.

Full Kafka Consumer Lag Monitoring

Option 1: kafka-exporter (Lightweight, Recommended Starting Point)

The most popular community solution is danielqsj/kafka_exporter. Deploy one Exporter container, and it automatically collects Lag metrics for all topics and consumer groups in Prometheus format.

# docker-compose.yml
services:
  kafka-exporter:
    image: danielqsj/kafka_exporter:v1.9.0
    command:
      - --kafka.server=kafka1:9092
      - --kafka.server=kafka2:9092
      - --kafka.server=kafka3:9092
      - --topic.filter=^.*$          # Collect all topics
      - --group.filter=^.*$          # Collect all consumer groups
      - --concurrent.enable          # Concurrent collection, must-enable for large clusters
      - --concurrent.max=200
      - --log.enable-sarama=false   # Turn off sarama log spam
    ports:
      - "9308:9308"
    restart: unless-stopped

After deployment, access http://exporter:9308/metrics. Key metrics look like this:

# Current committed offset for consumer group
kafka_consumergroup_current_offset{consumergroup="order-consumer",topic="orders",partition="0"} 12345678
# Partition's latest offset (LEO)
kafka_topic_partition_leader{topic="orders",partition="0"} 12355678
# Computed Lag (exporter calculates it automatically)
kafka_consumergroup_lag{consumergroup="order-consumer",topic="orders",partition="0"} 10000

Pitfall: kafka_exporter’s Lag calculation depends on committed offsets from consumer groups. If the consumer uses manual commits with a long interval (e.g., every 60 seconds), the Lag the exporter sees can be delayed by tens of seconds to minutes. This is not a bug in the exporter—it’s inherent to the offset commit mechanism. If your business requires high real-time accuracy, see Option 3 below.

Option 2: JMX Exporter (Collecting Broker Internal Metrics)

kafka_exporter only watches the consumer side. The Broker’s own health requires JMX metrics. Use jmx_exporter as a Java Agent attached to the Kafka process:

# Add one line to Kafka startup params
KAFKA_OPTS="$KAFKA_OPTS -javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent-0.20.0.jar=9404:/opt/jmx_exporter/kafka.yml"

Key JMX metrics:

Option 3: Directly Consuming __consumer_offsets (Highest Real-Time Accuracy)

The first two options both have latency—kafka_exporter depends on offset commits, JMX watches Broker state. If you need millisecond-level Lag monitoring, the most thorough approach is to directly consume Kafka’s internal __consumer_offsets topic.

This topic records all offset commit events for consumer groups. Write your own consumer to subscribe to it, parse out the latest offset for each group+topic+partition, then compare with LEO to compute the real processing position. Flink 2.0 does exactly this, pushing Lag monitoring latency from minutes to seconds (see Flink 2.0’s Kafka Source optimization).

The tradeoff is high development cost. For normal business use, kafka_exporter with 30-second polling is more than enough. Only ultra-latency-sensitive scenarios like risk control or real-time recommendations need Option 3.

PromQL Alert Rules

Three layers of alerts, from mild to severe:

# Alert rule file: kafka-lag-alerts.yml
groups:
  - name: kafka_consumer_lag
    interval: 30s
    rules:
      # P2: Per-partition Lag exceeds 5000 for 5 minutes
      - alert: KafkaConsumerLagHigh
        expr: |
          sum by (consumergroup, topic) (
            kafka_consumergroup_lag
          ) > 5000          
        for: 5m
        labels:
          severity: P2
        annotations:
          summary: "Kafka consumer group {{ $labels.consumergroup }} has high Lag on {{ $labels.topic }}"
          description: "Current total Lag: {{ $value }}, exceeds threshold 5000"

      # P1: Lag keeps growing for 10 minutes without recovery
      - alert: KafkaConsumerLagIncreasing
        expr: |
          deriv(
            sum by (consumergroup, topic) (
              kafka_consumergroup_lag
            )[10m:1m]
          ) > 0          
        for: 10m
        labels:
          severity: P1
        annotations:
          summary: "Kafka Lag keeps growing: {{ $labels.consumergroup }} / {{ $labels.topic }}"
          description: "Lag has been rising for the past 10 minutes, consumer may have stopped or is underpowered"

      # P0: Lag suddenly dropped to zero (suspected consumer crash with stopped commits)
      - alert: KafkaConsumerLagDroppedToZero
        expr: |
          (sum by (consumergroup, topic) (kafka_consumergroup_lag)) == 0
          and
          (sum by (consumergroup, topic) (kafka_consumergroup_lag offset 5m)) > 10000          
        for: 2m
        labels:
          severity: P0
        annotations:
          summary: "Kafka Lag suddenly zeroed: {{ $labels.consumergroup }}"
          description: "Lag was >10000 five minutes ago, now zero. Suspected consumer crash or offset anomaly"

The third rule is one many people overlook. Lag dropping to zero isn’t necessarily good—it likely means the consumer process died, offsets stopped updating, and coincidentally the producer had no new messages either. In this case LEO - CurrentOffset = 0, but messages were never consumed.

Tiered Threshold Reference

Different businesses have different tolerances. Below is a threshold table validated in production, tiered by business criticality:

RabbitMQ Queue Depth Monitoring

Kafka watches offset differences. RabbitMQ directly watches the number of messages in the queue—officially called queue depth. The meaning is more intuitive than Kafka’s Lag: how many unconsumed messages are piled up in the queue.

Built-in Prometheus Plugin

RabbitMQ 3.8+ ships with the rabbitmq_prometheus plugin. Enable it with one command:

# Run on each RabbitMQ node
rabbitmq-plugins enable rabbitmq_prometheus
# Exposed on port 15692 by default

Key metrics:

PromQL Alert Rules

groups:
  - name: rabbitmq_queue_depth
    interval: 30s
    rules:
      # P1: All consumers gone
      - alert: RabbitMQNoConsumers
        expr: rabbitmq_consumers == 0
        for: 1m
        labels:
          severity: P1
        annotations:
          summary: "RabbitMQ queue {{ $labels.queue }} has no consumers"
          description: "Queue {{ $labels.queue }} under vhost {{ $labels.vhost }} has no consumers, messages will pile up indefinitely"

      # P2: Queue depth persistently high
      - alert: RabbitMQQueueBacklog
        expr: rabbitmq_queue_messages > 10000
        for: 5m
        labels:
          severity: P2
        annotations:
          summary: "RabbitMQ queue {{ $labels.queue }} is backed up"
          description: "Queue {{ $labels.queue }} message count: {{ $value }}"

      # P1: Too many unacked messages (consumer is slow or stuck)
      - alert: RabbitMQUnackedHigh
        expr: rabbitmq_queue_messages_unacknowledged > 5000
        for: 3m
        labels:
          severity: P1
        annotations:
          summary: "RabbitMQ unacked messages too high"
          description: "Queue {{ $labels.queue }} has {{ $value }} unacked messages, consumer may be stuck"

Key difference between RabbitMQ and Kafka: RabbitMQ is push-based—the Broker actively pushes messages to consumers. Kafka is pull-based—consumers fetch on demand. This means RabbitMQ’s unacked metric is especially important: the consumer received the message but is slow to ack, meaning it’s stuck during processing (waiting for a database lock, waiting for a downstream HTTP response). This kind of stall won’t show up in the ready message count, only in unacked.

Dead Letter Queue Monitoring

Messages that fail consumption in RabbitMQ go to a Dead Letter Queue (DLX). The DLX message count is also an important signal:

      # P3: Dead letter queue has messages (business anomaly, not urgent but needs attention)
      - alert: RabbitMQDeadLetterQueue
        expr: |
          rabbitmq_queue_messages{queue=~".*\\.dlq$|.*dead_letter.*"} > 0          
        for: 10m
        labels:
          severity: P3
        annotations:
          summary: "Dead letter queue {{ $labels.queue }} has messages"
          description: "Dead letter queue has {{ $value }} messages, check upstream consumption failure reasons"

Redis Queue Monitoring

Many small-to-mid teams use Redis List as a lightweight message queue (LPUSH to enqueue, BRPOP to dequeue). Redis isn’t a professional message queue, but it’s simple and has low latency. The monitoring approach differs from Kafka/RabbitMQ—Redis has no built-in “queue Lag” metric, so you have to compute it yourself.

Collecting with Redis Exporter

# prometheus-redis-exporter
services:
  redis-exporter:
    image: oliver006/redis_exporter:v1.66.0
    command:
      - --redis.addr=redis:6379
      - --redis.password=${REDIS_PASSWORD}
      # Check List lengths for specified keys
      - --check-key-groups=true
      - --check-keys=queue:*,task:*,*:pending
    ports:
      - "9121:9121"

The check-keys parameter makes the exporter periodically run LLEN to get List lengths. The corresponding metrics:

redis_key_size{key="queue:order_tasks"} 1532
redis_key_size{key="queue:email_tasks"} 0

Alert Rules

groups:
  - name: redis_queue_depth
    interval: 30s
    rules:
      - alert: RedisQueueBacklog
        expr: redis_key_size{key=~"queue:.*"} > 5000
        for: 5m
        labels:
          severity: P2
        annotations:
          summary: "Redis queue {{ $labels.key }} is backed up"
          description: "Queue length: {{ $value }}"

      # Delayed queue (Sorted Set) overdue detection
      - alert: RedisDelayedTaskOverdue
        expr: |
          redis_db_keys{db="0"}  # Combined with custom script
          # Better approach: run a script to periodically ZCOUNT expired-but-unprocessed tasks          
        for: 5m
        labels:
          severity: P2
        annotations:
          summary: "Redis delayed queue has overdue unprocessed tasks"

The pain point with Redis queue monitoring is that the exporter can only do simple LLEN/ZCARD. If you use delayed queues (Sorted Set + timestamp), you need a custom script to count “overdue but unprocessed” tasks:

#!/usr/bin/env python3
"""redis_delayed_queue_exporter.py
Periodically scans Redis Sorted Set delayed queues,
counting overdue tasks that haven't been consumed yet.
"""
import time
import redis
from prometheus_client import CollectorRegistry, Gauge, write_to_textfile

r = redis.Redis(host='redis', port=6379, decode_responses=True)
registry = CollectorRegistry()

overdue_tasks = Gauge(
    'redis_delayed_queue_overdue',
    'Number of overdue tasks in delayed queue',
    ['queue_name'],
    registry=registry
)

now = time.time()
# Scan all delayed queue keys (convention: key prefix is delayed:)
for key in r.scan_iter(match='delayed:*'):
    # ZCOUNT key min max: count elements with score <= now
    count = r.zcount(key, 0, now)
    overdue_tasks.labels(queue_name=key).set(count)

write_to_textfile('/var/lib/node_exporter/textfile/redis_delayed.prom', registry)

Drop it into crontab to run every minute, pair it with node_exporter’s textfile collector, and Prometheus can scrape it.

Root Cause Analysis of Consumer Backlog

Monitoring and alerting are just the first step. After an alert fires, how do you locate the root cause? Below is a four-quadrant troubleshooting method I’ve summarized, split along two dimensions: “is the problem with the producer or consumer” and “is it a burst or sustained”.

Four-Quadrant Localization

Burst + Producer: Most common. Promotions, scheduled batch jobs, upstream compensatory resends. This backlog usually self-resolves when traffic subsides. The key indicator is the Lag growth rate—if the growth rate is declining, production is ramping down, so just wait. But bake this judgment into alerts instead of guessing manually each time.

Sustained + Producer: Production rate chronically exceeds consumption rate. Maybe a new data source was added, or the upstream scaled up but the consumer side didn’t keep up. Adding consumer instances can help, but the real fix is either rate-limiting (producer side) or scaling (consumer side), depending on the business.

Burst + Consumer: Consumer died. Check the consumer process status, logs, and GC. In Kafka, kafka-consumer-groups.sh --describe --group <group> shows each consumer’s host and assigned partitions. If the consumer list is empty, they’re all dead.

Sustained + Consumer: Slow consumption logic. This is the most complex. First use jstack or pprof to see what the consumer threads are doing—waiting on the database, waiting for HTTP responses, or doing CPU-intensive computation. Common cause is a slow downstream dependency (slow database query, third-party API timeout). The consumer itself is fine but dragged down by downstream.

Kafka Troubleshooting Command Cheatsheet

# 1. View consumer group status and Lag
kafka-consumer-groups.sh --bootstrap-server kafka1:9092 \
  --describe --group order-consumer

# Example output:
# GROUP           TOPIC       PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG     CONSUMER-ID
# order-consumer  orders      0          12345678        12355678        10000   consumer-1
# order-consumer  orders      1          23456789        23456789        0       consumer-1
# order-consumer  orders      2          34567890        34567990        100     consumer-2
# order-consumer  orders      3          -               45678901        45678901 -  ← no consumer!

# 2. View consumer group members (confirm consumers are alive)
kafka-consumer-groups.sh --bootstrap-server kafka1:9092 \
  --describe --group order-consumer --members --verbose

# 3. View topic partition distribution
kafka-topics.sh --bootstrap-server kafka1:9092 \
  --describe --topic orders

# 4. Monitor Lag changes in real time (refresh every 5 seconds)
watch -n 5 'kafka-consumer-groups.sh --bootstrap-server kafka1:9092 --describe --group order-consumer | tail -n +3 | sort -k6 -rn'

Key insight: If a partition’s CONSUMER-ID is empty (shows -), that partition wasn’t assigned to any consumer. The reason could be that the number of consumer instances is less than the number of partitions, or a rebalance occurred. This kind of Lag can only be fixed by adding consumers or fixing the rebalance.

RabbitMQ Troubleshooting Commands

# View queue details (consumer count, message count, unacked count)
rabbitmqctl list_queues name messages consumers messages_unacknowledged

# View consumer connections
rabbitmqctl list_consumers | grep <queue_name>

# View connection details (troubleshoot stuck consumers)
rabbitmqctl list_connections name peer_host state channels

Auto-Scaling and Self-Healing

Once monitoring is solid, the next step is letting the system handle backlog on its own. Manually adding consumers is firefighting. Auto-scaling is the long-term fix.

KEDA + Kafka Metric-Driven Autoscaling

KEDA (Kubernetes Event-Driven Autoscaling) is a tool that automatically scales Deployments based on event metrics like Kafka Lag. The core logic: high Lag triggers adding Pods, low Lag triggers removing Pods.

# ScaledObject: Auto-scale consumers based on Kafka Lag
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kafka-order-consumer
  namespace: production
spec:
  scaleTargetRef:
    name: order-consumer-deployment
  minReplicaCount: 2        # Minimum 2 replicas
  maxReplicaCount: 20       # Maximum 20 replicas
  pollingInterval: 30       # Check Lag every 30 seconds
  cooldownPeriod: 300        # Scale-down cooldown 5 minutes, anti-flap
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka1:9092,kafka2:9092,kafka3:9092
        consumerGroup: order-consumer
        topic: orders
        lagThreshold: "1000"   # Scale up when per-partition Lag exceeds 1000
        offsetResetPolicy: latest
        partitionLimitation: "0,1,2,3"  # Optional: only watch specified partitions

What this config does: when the order-consumer group’s Lag on the orders topic exceeds 1000, KEDA automatically increases the Pod count. After Lag drops below 1000 and stabilizes for 5 minutes, it automatically scales down.

Pitfall: The number of Kafka consumers cannot exceed the number of partitions. If your topic has only 6 partitions and you scale the Deployment to 20 Pods, the extra 14 Pods will sit idle. Before scaling, confirm the partition count. If it’s not enough, do a partition expansion first (kafka-topics.sh --alter --partitions).

Consumer-Side Backpressure Protection

Scaling alone isn’t enough. The consumer side also needs self-protection—slow down consumption when overwhelmed, rather than getting crushed. In Spring Boot Kafka, use MaxPollRecords to control how many messages are fetched per poll:

# application.yml
spring:
  kafka:
    consumer:
      group-id: order-consumer
      max-poll-records: 100          # Max 100 messages per poll
      max-poll-interval-ms: 300000   # Max 5 minutes between polls
      properties:
        max.partition.fetch.bytes: 1048576  # Max 1MB per partition per fetch

max-poll-interval-ms is critical. If the consumer takes longer than this to process a batch without committing, Kafka considers the consumer “stuck” and triggers a rebalance, reassigning the partition to someone else. The default of 5 minutes might not be enough for slow tasks, but setting it too high means truly stuck consumers take too long to get evicted. A good rule of thumb is 3x your P99 processing time.

Dashboard Design

The Grafana dashboard needs to solve one problem: spot which consumer group is backing up at a glance, not sift through 50 metrics.

I recommend three panels:

Panel 1: Overview Dashboard (All Consumer Groups Lag Ranking)

# Aggregate total Lag by group, sorted descending
topk(20, sum by (consumergroup) (kafka_consumergroup_lag))

Use a Stat panel or Bar Gauge. At a glance, see which group has the worst backlog. Color coding: green <1000, yellow 1000-10000, red >10000.

Panel 2: Single Consumer Group Detail (Lag Trend + Partition Distribution)

# Lag trend (time series chart)
sum by (topic) (kafka_consumergroup_lag{consumergroup="$group"})

# Per-partition Lag (table, highlight max value)
kafka_consumergroup_lag{consumergroup="$group", topic="$topic"}

Panel 3: Consumption Rate (Production Rate vs Consumption Rate)

# Consumption rate (messages consumed per second)
rate(kafka_consumergroup_current_offset{consumergroup="$group"}[1m])

# Production rate (new messages per second)
rate(kafka_topic_partition_leader{topic="$topic"}[1m])

Put both lines on one chart. When production rate > consumption rate, Lag inevitably grows. This is the fundamental signal of backlog.

Production Practice Recommendations

Based on pitfalls I’ve hit, here are practical takeaways:

1. Establish baselines before setting thresholds. Don’t copy someone else’s thresholds right away. Run for a week, compute the normal Lag distribution (P50, P95, P99), then set warning lines based on P99. Lag baselines vary wildly between businesses—log consumers naturally have high Lag, transaction consumers should be near zero.

2. Per-partition monitoring is more important than aggregate monitoring. Normal total Lag doesn’t mean no problem. Six partitions with five at Lag=0 and one at Lag=60000 looks “okay” when aggregated, but that partition’s consumer may be dead. Alerts must be set at the consumergroup + topic + partition dimension, at minimum consumergroup + topic.

3. Alerts should carry context. A bare “Lag 10000” alert is useless. The on-call person doesn’t know if it’s rising or falling, or for how long. Alert annotations should include current Lag, Lag 5 minutes ago, growth rate, and consumer count. The on-call person should tell at a glance whether to act immediately or let it self-resolve.

4. Distinguish “expected backlog” from “abnormal backlog.” Scheduled batch jobs, data migrations, message replays—these operations naturally produce backlog. Silence alerts for these expected high-Lag periods, or alert fatigue will make you ignore real incidents.

5. Consumer health checks should go beyond process liveness. A live process doesn’t mean it’s consuming. Instrument a heartbeat metric on the consumer side—update a timestamp every time a message is successfully consumed. If the heartbeat hasn’t updated in 5 minutes, the consumer is alive but actually stuck.

Summary

The core of message queue monitoring isn’t how fancy your tools are, but whether you can answer three questions within 30 seconds of an alert: where is the backlog, rising or falling, and is the consumer alive.

Kafka watches Consumer Lag, RabbitMQ watches ready + unacked, Redis computes List length yourself—the metrics differ, but the monitoring approach is the same: establish baselines first, then set tiered thresholds, then add auto-scaling. Lag dropping to zero isn’t necessarily good. Lag rising isn’t necessarily bad. What matters is the trend and growth rate.

One final reminder: no matter how good your monitoring is, it can’t save a fundamentally flawed consumption design. If your consumer synchronously calls a slow downstream interface in a single thread, all the monitoring does is let you see the backlog sooner. The real fix is async processing, batching, and rate limiting. Monitoring is the eyes, not the hands and feet.

References & Acknowledgments

The following resources were referenced during the writing of this article. Thanks to the original authors for their contributions:

1. Kafka消息消费卡住了?手把手教你用Offset Explorer监控Lag与排查积压 — CSDN Column, core Consumer Lag metrics and backlog troubleshooting in practice
2. Kafka中replica offset滞后(LAG)过大的常见原因有哪些? — CSDN Q&A, five-dimensional root cause analysis for partition replica Lag anomalies
3. SpringBoot消息积压排查:监控与扩容策略 — CSDN Blog, message backlog root causes and scaling strategies
4. Flink 2.0新特性深度解析:Kafka Lag监控与性能优化实践 — Baidu Cloud, Flink 2.0’s real-time Lag monitoring by directly consuming __consumer_offsets
5. RQMZ系统中任务队列积压导致延迟,如何优化消费速率? — CSDN Q&A, four-dimensional governance model and dynamic autoscaling strategy