Alertmanager Alert Routing and Silencing Strategies

In the Prometheus ecosystem, Prometheus generates alerts based on alerting rules, while Alertmanager manages the entire alert lifecycle: grouping, routing, inhibition, deduplication, and notification delivery. A poorly configured Alertmanager can drown on-call engineers in a flood of duplicate alerts at 3 AM, whereas a well-designed routing and inhibition strategy ensures that “the right person receives the right alert at the right time.”

Reference: Prometheus Official Documentation — Alertmanager

I. Alertmanager Architecture

Alertmanager’s processing pipeline consists of five stages:

Prometheus alert rule triggered
        │
        ▼
┌───────────────┐     ┌───────────┐     ┌───────────┐     ┌───────────┐     ┌───────────┐
│   Receive      │ ──→ │  Group    │ ──→ │  Route    │ ──→ │  Inhibit  │ ──→ │  Dedup    │
└───────────────┘     └───────────┘     └───────────┘     └───────────┘     └─────┬─────┘
                                                                                      │
                                                                                      ▼
                                                                             ┌───────────────┐
                                                                             │   Notify      │
                                                                             │ Email/WeCom/  │
                                                                             │ DingTalk       │
                                                                             └───────────────┘

Stage	Purpose	Key Config
Receive	Receives alerts from Prometheus	`receivers`
Group	Merges alerts with the same characteristics into a batch	`group_by`
Route	Determines alert destination based on label matching	`route`, `matchers`
Inhibit	Silences related lower-priority alerts when a higher-priority one fires	`inhibit_rules`
Dedup	Deduplicates alerts across multiple Alertmanager instances	HA mode + Gossip
Notify	Sends notifications through configured channels	webhook / email / etc.

II. Routing Tree Design

2.1 Routing Tree Structure

Alertmanager routing is a tree. The root node is the default route, and each child node matches labels via matchers:

route:
  receiver: default
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Critical alerts → immediate notification
    - matchers:
        - severity="critical"
      receiver: critical-pager
      group_wait: 0s
      repeat_interval: 1h

    # Warning level → delayed notification
    - matchers:
        - severity="warning"
      receiver: warning-channel
      group_wait: 5m
      repeat_interval: 4h
      routes:
        # Database-related warnings → DBA team
        - matchers:
            - team="dba"
          receiver: dba-team
          continue: false

    # Alert recovery notifications → unified delivery
    - matchers:
        - alertname=~"NodeDown|HostUnavailable"
      receiver: infra-team

2.2 matchers Syntax

Starting from Alertmanager v0.22, matchers uses a unified syntax that replaces the legacy match / match_re:

matchers:
  # Exact match
  - severity="critical"

  # Regex match
  - alertname=~"Node.*"

  # Not equal
  - severity!="info"

  # Negative regex match
  - instance!~"localhost.*"

  # Check label existence (value is empty)
  - "maintenance"

  # Check label absence
  - "!staging"

2.3 continue Keyword

By default, once an alert matches a child route, it won’t continue matching subsequent sibling routes. Setting continue: true allows an alert to be sent to multiple receivers simultaneously:

routes:
  - matchers:
      - severity="critical"
    receiver: pagerduty
    continue: true          # Continue matching the next route
  - matchers:
      - severity="critical"
    receiver: slack-critical

The configuration above sends critical alerts to both PagerDuty and Slack.

2.4 Cascading Route Example

routes:
  # First level: route by team
  - matchers:
      - team="sre"
    receiver: sre-default
    routes:
      # Second level: critical alerts within SRE
      - matchers:
          - severity="critical"
        receiver: sre-critical
      # Second level: disk alerts within SRE
      - matchers:
          - alertname=~"Disk.*"
        receiver: sre-storage

  - matchers:
      - team="dba"
    receiver: dba-default
    routes:
      - matchers:
          - severity="critical"
        receiver: dba-critical

III. Grouping Strategies

Grouping determines “which alerts will be merged into a single notification.”

3.1 Core Parameters

Parameter	Description	Typical Value
`group_by`	Labels to group by	`['alertname', 'cluster', 'service']`
`group_wait`	How long to wait after the first alert before sending (waiting for more alerts in the same group)	`30s`
`group_interval`	Interval between sending merged new alerts in the same group	`5m`
`repeat_interval`	Interval for repeating the same alert notification	`4h`

3.2 Grouping Timeline Example

Assuming group_by: ['alertname'], group_wait: 30s, group_interval: 5m:

t=0s    AlertA received (instance=node-1)
        → Start waiting 30s (group_wait)

t=10s   AlertA received (instance=node-2)  → Same group, merged

t=30s   Send notification [AlertA(node-1), AlertA(node-2)]
        → Start 5min timer (group_interval)

t=3min  AlertA received (instance=node-3)  → Same group, not sent yet

t=5min  Send notification [AlertA(node-3)]       → Only new additions

t=10min No new alerts, but unresolved alerts exist
        → repeat_interval not reached, no notification sent

t=4h    repeat_interval reached
        → Send repeat notification [AlertA(node-1), AlertA(node-2), AlertA(node-3)]

3.3 Grouping Configuration Recommendations

route:
  # Group by alert name + cluster
  # Same-type alerts from the same cluster are merged into one notification
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s       # Wait 30s to collect same-group alerts
  group_interval: 5m    # Merge alerts within 5 minutes
  repeat_interval: 4h   # Don't repeat the same alert within 4 hours

Rules of thumb:

Do not include high-cardinality fields (like instance) in group_by, as it causes over-fragmented grouping.
Set group_wait to 0s for Critical alerts to send immediately.
Set group_wait to 5m for Warning alerts to reduce noise through merging.

IV. Inhibition Rules

The core scenario for inhibition: automatically silence related lower-priority alerts when a higher-priority alert fires.

4.1 Typical Scenarios

When a node goes down (NodeDown), inhibit all service alerts on that node
During a network partition, inhibit downstream alerts caused by network unreachability
When a database primary fails, inhibit application alerts that depend on that database

4.2 Configuration Example

inhibit_rules:
  # Scenario 1: When a node is down, inhibit all other alerts on that node
  - source_matchers:
      - alertname="NodeDown"
    target_matchers:
      - alertname!="NodeDown"
    equal: ['instance']

  # Scenario 2: Critical alerts inhibit Warning alerts for the same service
  - source_matchers:
      - severity="critical"
    target_matchers:
      - severity="warning"
    equal: ['alertname', 'service']

  # Scenario 3: When a cluster is unreachable, inhibit all alerts within that cluster
  - source_matchers:
      - alertname="ClusterUnavailable"
    target_matchers:
      - alertname!="ClusterUnavailable"
    equal: ['cluster']

  # Scenario 4: When DB primary is down, inhibit application alerts depending on that instance
  - source_matchers:
      - alertname="MySQLMasterDown"
    target_matchers:
      - alertname=~"App.*"
    equal: ['db_instance']

4.3 How It Works

source (NodeDown, instance=node-1, severity=critical)
                    │
                    │  Triggers inhibition
                    ▼
target (CPUHigh, instance=node-1, severity=warning)
                    │
                    │  equal: ['instance']
                    │  instance matches → inhibition applies
                    ▼
              target alert silenced

Key field descriptions:

source_matchers: Conditions for the alert that triggers inhibition
target_matchers: Conditions for the alert to be inhibited
equal: Source and target must have the same values for these labels to trigger inhibition

V. Silence Management

Silences are used to temporarily suppress specific alerts. A typical scenario is avoiding alert noise during planned maintenance (e.g., upgrades, restarts).

5.1 Creating a Silence via Web UI

Navigate to the Alertmanager Web UI (default http://localhost:9093) → Silences → New Silence:

Matchers:
  alertname = NodeHighCpuLoad
  instance  = node-3

Starts At:  2026-07-10 22:00:00
Ends At:    2026-07-10 23:00:00
Created By: xubaojin
Comment:    Planned maintenance - CPU scaling

5.2 Creating a Silence via amtool CLI

amtool is the command-line tool bundled with Alertmanager:

# Create a silence (1 hour)
amtool silence add \
  --comment="Planned maintenance" \
  --author="xubaojin" \
  --duration=1h \
  alertname=NodeHighCpuLoad \
  instance=node-3

# List all silences
amtool silence query \
  --alertmanager.url=http://localhost:9093

# Query by matchers
amtool silence query instance=node-3

# Expire a silence by ID
amtool silence expire <silence-id> \
  --alertmanager.url=http://localhost:9093

# Expire all silences
amtool silence expire $(amtool silence query -o simple | awk '{print $1}')

5.3 Creating a Silence via API

curl -X POST http://localhost:9093/api/v2/silences \
  -H "Content-Type: application/json" \
  -d '{
    "matchers": [
      { "name": "alertname", "value": "NodeHighCpuLoad", "isRegex": false },
      { "name": "instance", "value": "node-3", "isRegex": false }
    ],
    "startsAt": "2026-07-10T22:00:00Z",
    "endsAt": "2026-07-10T23:00:00Z",
    "createdBy": "xubaojin",
    "comment": "Planned maintenance - CPU scaling"
  }'

5.4 amtool Common Commands Cheat Sheet

# View currently active alerts
amtool alert query \
  --alertmanager.url=http://localhost:9093

# View alert routing (check which receiver an alert matches)
amtool config routes test \
  --alertmanager.url=http://localhost:9093 \
  severity=critical team=sre

# View the routing tree (tree display)
amtool config routes show \
  --alertmanager.url=http://localhost:9093

VI. Multi-Channel Notification Configuration

6.1 Email Notifications

receivers:
  - name: email-team
    email_configs:
      - to: 'oncall@example.com'
        from: 'alertmanager@example.com'
        smarthost: 'smtp.example.com:587'
        auth_username: 'alertmanager@example.com'
        auth_password: 'YOUR_SMTP_PASSWORD'
        auth_secret: 'YOUR_SMTP_PASSWORD'
        auth_identity: 'alertmanager@example.com'
        require_tls: true
        headers:
          Subject: '{{ .Status | toUpper }} - {{ .CommonLabels.alertname }}'
        send_resolved: true

6.2 WeCom (Enterprise WeChat) Webhook

receivers:
  - name: wecom-team
    webhook_configs:
      - url: 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=YOUR_WECOM_KEY'
        send_resolved: true
        max_alerts: 0

The WeCom webhook requires a specific message format. You typically need to deploy an intermediary forwarding service (such as prometheus-webhook-dingtalk or a custom adapter) to convert Alertmanager’s default JSON into the WeCom format.

WeCom message template adapter example (Go):

package main

import (
    "encoding/json"
    "fmt"
    "net/http"
    "bytes"

    "github.com/prometheus/alertmanager/template"
)

type WecomMessage struct {
    MsgType  string   `json:"msgtype"`
    Markdown struct {
        Content string `json:"content"`
    } `json:"markdown"`
}

func handler(w http.ResponseWriter, r *http.Request) {
    var data template.Data
    json.NewDecoder(r.Body).Decode(&data)

    content := fmt.Sprintf(
        "## Alert Notification\n"+
            "**Status**: %s\n"+
            "**Alert Count**: %d\n"+
            "**Details**:\n",
        data.Status, len(data.Alerts),
    )
    for _, alert := range data.Alerts {
        content += fmt.Sprintf(
            "- **%s**: %s\n",
            alert.Labels["alertname"],
            alert.Annotations["summary"],
        )
    }

    msg := WecomMessage{MsgType: "markdown"}
    msg.Markdown.Content = content

    payload, _ := json.Marshal(msg)
    http.Post(
        "https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=YOUR_KEY",
        "application/json",
        bytes.NewReader(payload),
    )
    w.WriteHeader(http.StatusOK)
}

func main() {
    http.HandleFunc("/wecom", handler)
    http.ListenAndServe(":8060", nil)
}

6.3 DingTalk Webhook

receivers:
  - name: dingtalk-team
    webhook_configs:
      - url: 'http://localhost:8060/dingtalk/webhook1/send'
        send_resolved: true

DingTalk also requires message format adaptation. The recommended approach is to use the prometheus-webhook-dingtalk project:

# Start the DingTalk webhook adapter
prometheus-webhook-dingtalk \
  --ding.profile="webhook1=https://oapi.dingtalk.com/robot/send?access_token=YOUR_TOKEN" \
  --ding.timeout=5s

VII. Complete alertmanager.yml Configuration Example

The following is a complete configuration covering routing, grouping, inhibition, and multi-channel notifications:

global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: 'alertmanager@example.com'
  smtp_auth_username: 'alertmanager@example.com'
  smtp_auth_password: 'YOUR_SMTP_PASSWORD'
  smtp_require_tls: true

# Template files (custom notification templates)
templates:
  - '/etc/alertmanager/templates/*.tmpl'

# Routing tree
route:
  receiver: default
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Critical → immediate phone call + WeCom
    - matchers:
        - severity="critical"
      receiver: critical-notify
      group_wait: 0s
      repeat_interval: 1h
      continue: true

    # Warning → WeCom
    - matchers:
        - severity="warning"
      receiver: warning-notify
      group_wait: 2m

    # DBA team alerts
    - matchers:
        - team="dba"
      receiver: dba-notify
      group_wait: 1m

    # Info level → archive only, no notification
    - matchers:
        - severity="info"
      receiver: null

# Inhibition rules
inhibit_rules:
  # Node down → inhibit all other alerts on that node
  - source_matchers:
      - alertname="NodeDown"
    target_matchers:
      - alertname!="NodeDown"
    equal: ['instance']

  # Critical inhibits Warning for the same service
  - source_matchers:
      - severity="critical"
    target_matchers:
      - severity="warning"
    equal: ['alertname', 'service']

  # Cluster unreachable → inhibit all alerts in that cluster
  - source_matchers:
      - alertname="ClusterUnavailable"
    target_matchers:
      - alertname!="ClusterUnavailable"
    equal: ['cluster']

# Receiver definitions
receivers:
  - name: default
    email_configs:
      - to: 'oncall@example.com'
        send_resolved: true

  - name: critical-notify
    webhook_configs:
      - url: 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=YOUR_WECOM_KEY'
        send_resolved: true
    email_configs:
      - to: 'critical-oncall@example.com'
        send_resolved: true

  - name: warning-notify
    webhook_configs:
      - url: 'http://localhost:8060/dingtalk/webhook1/send'
        send_resolved: true

  - name: dba-notify
    email_configs:
      - to: 'dba@example.com'
        send_resolved: true
    webhook_configs:
      - url: 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=DBA_WECOM_KEY'
        send_resolved: true

  - name: null
    # Empty receiver, discards alerts

VIII. High Availability Deployment

Alertmanager supports multi-instance HA deployment, synchronizing alert state and silence information via the Gossip protocol:

# Start a 3-node HA cluster
alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager \
  --web.listen-address=:9093 \
  --cluster.peer=alertmanager-1:9094 \
  --cluster.peer=alertmanager-2:9094 \
  --cluster.peer=alertmanager-3:9094 \
  --cluster.listen-address=0.0.0.0:9094

On the Prometheus side, configure multiple Alertmanager addresses for failover:

# prometheus.yml
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager-1:9093
            - alertmanager-2:9093
            - alertmanager-3:9093

Prometheus sends alerts to all Alertmanagers. The instances deduplicate via the Gossip protocol, ensuring that ultimately only one instance sends the notification.

Summary

The core of Alertmanager configuration revolves around three questions:

Who should receive alerts? → Routing tree (route + matchers)
How to reduce alert noise? → Grouping (group_by) + Inhibition (inhibit_rules) + Silences
How are alerts delivered? → Receivers (receivers) + Notification channels

A well-crafted Alertmanager configuration should ensure that on-call engineers rarely receive alerts under normal conditions (because problems are preempted), and only receive precisely targeted, information-rich, non-duplicate alerts when anomalies occur. Remember one principle: it’s not about more alerts, it’s about the right alerts.

For more details, see Prometheus Official Documentation — Alertmanager

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

Prometheus Official Documentation — Alertmanager — Prometheus Authors, referenced for Prometheus Official Documentation — Alertmanager
prometheus-webhook-dingtalk — GitHub, referenced for prometheus-webhook-dingtalk

I. Alertmanager Architecture#

II. Routing Tree Design#

2.1 Routing Tree Structure#

2.2 matchers Syntax#

2.3 continue Keyword#

2.4 Cascading Route Example#

III. Grouping Strategies#

3.1 Core Parameters#

3.2 Grouping Timeline Example#

3.3 Grouping Configuration Recommendations#

IV. Inhibition Rules#

4.1 Typical Scenarios#

4.2 Configuration Example#

4.3 How It Works#

V. Silence Management#

5.1 Creating a Silence via Web UI#

5.2 Creating a Silence via amtool CLI#

5.3 Creating a Silence via API#

5.4 amtool Common Commands Cheat Sheet#

VI. Multi-Channel Notification Configuration#

6.1 Email Notifications#

6.2 WeCom (Enterprise WeChat) Webhook#

6.3 DingTalk Webhook#

VII. Complete alertmanager.yml Configuration Example#

VIII. High Availability Deployment#

Summary#

References & Acknowledgments#