Overview

Prometheus’s conventional monitoring is “inside-out” — Prometheus scrapes Exporter metrics to understand internal system state. But when users access your service, they follow an “outside-in” path: DNS resolution → network routing → load balancing → backend service. A link that looks perfectly healthy from the inside may be completely inaccessible to users due to DNS misconfiguration, CDN cache issues, or expired SSL certificates.

Blackbox Exporter solves the “external probing” problem. It simulates user behavior, probing HTTP/TCP/ICMP/DNS endpoints from the outside, giving you “user perspective” availability data. This article systematically covers Blackbox Exporter configuration, probing methods, SSL certificate monitoring, multi-region probing, and availability SLO measurement.

Reference: Blackbox Exporter Official Documentation

I. Why External Probing

1.1 Blind Spots of Internal Monitoring

User request path:
User → DNS → CDN → Load Balancer → Ingress → Pod → Database

Internal monitoring coverage:
                                      ✓ Pod metrics
                                      ✓ Database metrics
                           ✓ Ingress metrics
                 ✓ LB metrics

External blind spots:
      ✗ Is DNS resolution working?
      ✗ Is CDN caching correct?
      ✗ Has the SSL certificate expired?
      ✗ Is network routing clear?
      ✗ Is access reachable from user's region?

Typical scenarios where internal monitoring falls short:

ScenarioInternal MonitoringExternal Probing
DNS resolution failureNot visible✓ Detectable
SSL certificate expiredNot visible (needs proactive check)✓ Detectable
CDN cache errorNot visible✓ Detectable
Network routing outageNot visible✓ Detectable
Slow access from specific regionNot visible✓ Multi-region probing
Page content tamperedNot visible✓ HTTP content check

1.2 Blackbox Exporter’s Role

┌────────────────────────────────────────────────────┐
│              Monitoring Perspective Comparison      │
│                                                    │
│  Internal Monitoring (Prometheus + Exporter)       │
│  → "Is the system healthy?"                        │
│  → CPU, memory, disk, QPS, latency                 │
│                                                    │
│  External Probing (Blackbox Exporter)              │
│  → "Can users access it?"                          │
│  → HTTP status code, response time, cert validity  │
│  → TCP port reachability, DNS resolution correctness│
│                                                    │
│  Both complement: internal health + external       │
│  reachability = complete availability              │
└────────────────────────────────────────────────────┘

II. Blackbox Exporter Architecture

2.1 How It Works

┌───────────────┐         ┌───────────────────────┐
│  Prometheus   │  scrape │  Blackbox Exporter    │
│  (scraper)    │ ──────→ │  (prober)             │
└───────────────┘         └───────────┬───────────┘
                          ┌────────────┼────────────┐
                          ▼            ▼            ▼
                     ┌─────────┐  ┌─────────┐  ┌─────────┐
                     │ HTTP    │  │  TCP    │  │  ICMP   │
                     │ probe   │  │ probe   │  │ probe   │
                     └────┬────┘  └────┬────┘  └────┬────┘
                          │            │            │
                          ▼            ▼            ▼
                     ┌─────────────────────────────────────┐
                     │       Probed Target                  │
                     │  web.example.com / 10.0.1.5:3306    │
                     └─────────────────────────────────────┘

Core characteristics of Blackbox Exporter:

  • Pull-mode integration: Prometheus scrapes Blackbox Exporter to get probe results
  • Parameterized probing: Target and module specified via URL parameters
  • Modular configuration: Different probe types (HTTP/TCP/ICMP/DNS) use different modules
  • Rich metrics returned: Probe results include status code, response time, certificate info, etc.

2.2 Probe URL Format

# HTTP probe
http://blackbox:9115/probe?target=https://example.com&module=http_2xx

# TCP probe
http://blackbox:9115/probe?target=10.0.1.5:3306&module=tcp_connect

# ICMP probe
http://blackbox:9115/probe?target=10.0.1.5&module=icmp

# DNS probe
http://blackbox:9115/probe?target=dns.example.com&module=dns_example

III. Configuration Details

3.1 Complete Configuration File

# blackbox.yml
modules:
  # HTTP probe module
  http_2xx:
    prober: http
    timeout: 10s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200, 201, 204]  # Acceptable status codes
      method: GET
      headers:
        User-Agent: "Blackbox-Exporter/0.25"
        Accept-Language: "zh-CN"
      follow_redirects: true
      preferred_ip_protocol: ip4  # Prefer IPv4
      ip_protocol_fallback: true

  # HTTP probe - with authentication
  http_with_auth:
    prober: http
    timeout: 10s
    http:
      method: POST
      headers:
        Content-Type: application/json
        Authorization: "Bearer token-xxx"
      body: '{"action":"health_check"}'
      valid_status_codes: [200]

  # HTTP probe - content check
  http_content_check:
    prober: http
    timeout: 10s
    http:
      fail_if_body_matches_regexp:
        - "error"
        - "maintenance"
      fail_if_body_not_matches_regexp:
        - "expected_content"

  # HTTPS probe - strict SSL verification
  http_strict_ssl:
    prober: http
    timeout: 10s
    http:
      fail_if_ssl: false
      tls_config:
        insecure_skip_verify: false
        ca_file: /etc/ssl/certs/ca-certificates.crt

  # TCP port probe
  tcp_connect:
    prober: tcp
    timeout: 5s

  # TCP probe - with protocol handshake
  tcp_mysql:
    prober: tcp
    timeout: 5s
    tcp:
      query_response:
        - send: "\x00\x00\x00\x0a"  # MySQL handshake packet
        - expect: "^[\\x00-\\xff]"  # Match response

  # TCP probe - SSH port
  tcp_ssh:
    prober: tcp
    timeout: 5s
    tcp:
      query_response:
        - expect: "^SSH-2.0-"

  # ICMP probe
  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: ip4

  # DNS probe
  dns_check:
    prober: dns
    timeout: 5s
    dns:
      query_name: "example.com"
      query_type: A
      valid_rcodes: [NOERROR]
      validate_answer_rrs:
        fail_if_matches_regexp:
          - ".*127.0.0.1"
        fail_if_not_matches_regexp:
          - ".*10\\.0\\..*"

  # DNS probe - check SOA
  dns_soa:
    prober: dns
    timeout: 5s
    dns:
      query_name: "example.com"
      query_type: SOA
      transport_protocol: tcp
      valid_rcodes: [NOERROR]

3.2 Docker Deployment

# docker-compose.yml
version: '3.8'

services:
  blackbox-exporter:
    image: prom/blackbox-exporter:v0.25.0
    container_name: blackbox
    ports:
      - "9115:9115"
    volumes:
      - ./blackbox.yml:/etc/blackbox_exporter/config.yml
    command:
      - '--config.file=/etc/blackbox_exporter/config.yml'
      - '--web.listen-address=:9115'
      - '--log.level=info'
    restart: unless-stopped

Note: ICMP probing requires extra permissions. In Docker, add cap_add: [NET_ADMIN] or use host network mode.

IV. HTTP Probing

4.1 Basic HTTP Probe

# Prometheus scrape configuration
scrape_configs:
  - job_name: 'blackbox-http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://www.example.com
          - https://api.example.com/health
          - https://app.example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115

4.2 Metrics Returned by Probing

# Manually test probe results
curl -s "http://blackbox:9115/probe?target=https://example.com&module=http_2xx" | grep probe_

Key metrics returned:

MetricDescription
probe_successWhether probe succeeded (0/1)
probe_duration_secondsTotal probe duration
probe_http_status_codeHTTP status code
probe_http_versionHTTP protocol version
probe_http_content_lengthResponse content length
probe_ssl_earliest_cert_expiryEarliest SSL cert expiry (Unix timestamp)
probe_ssl_last_informationSSL certificate info
probe_dns_lookup_time_secondsDNS resolution duration
probe_tcp_connection_secondsTCP connection duration
probe_tls_handshake_secondsTLS handshake duration
probe_http_duration_seconds{phase=...}Phase durations (dns/connect/tls/transfer)

4.3 Phase-by-Phase Duration

# View HTTP request phase durations
probe_http_duration_seconds{phase="dns"}      # DNS resolution
probe_http_duration_seconds{phase="connect"}  # TCP connection
probe_http_duration_seconds{phase="tls"}      # TLS handshake
probe_http_duration_seconds{phase="transfer"} # Data transfer
probe_http_duration_seconds{phase="processing"}# Server processing

These phase metrics help pinpoint latency bottlenecks — whether it’s slow DNS, slow network, or slow server processing.

V. TCP Port Probing

5.1 Basic TCP Probe

scrape_configs:
  - job_name: 'blackbox-tcp'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets:
          - 'mysql:3306'
          - 'redis:6379'
          - 'kafka:9092'
          - 'ssh:22'
          - 'smtp:25'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115

5.2 Protocol Handshake Probing

Simple port reachability isn’t enough to verify a service is truly available. Use query_response for protocol-layer handshake validation:

modules:
  # MySQL health check
  tcp_mysql:
    prober: tcp
    tcp:
      query_response:
        - expect: "^.*\\x00\\x00\\x00\\x0a.*"  # MySQL handshake response

  # Redis PING
  tcp_redis:
    prober: tcp
    tcp:
      query_response:
        - send: "PING\r\n"
        - expect: "^\\+PONG"

  # SMTP check
  tcp_smtp:
    prober: tcp
    tcp:
      query_response:
        - expect: "^220 .* SMTP"
        - send: "EHLO blackbox\r\n"
        - expect: "^250"

  # PostgreSQL check
  tcp_postgres:
    prober: tcp
    tcp:
      query_response:
        - send: "\x00\x00\x00\x08\x04\xd2\x16\x2f"  # PgSSLRequest
        - expect: "^[SN]"  # S=SSL supported, N=not supported

VI. ICMP Probing

6.1 ICMP Configuration

scrape_configs:
  - job_name: 'blackbox-icmp'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
          - '192.168.1.1'     # Gateway
          - '8.8.8.8'         # Public DNS
          - '10.0.0.1'        # Internal core switch
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115

6.2 ICMP Metrics

MetricDescription
probe_successWhether ICMP response received
probe_duration_secondsRound-trip time (RTT)
probe_ip_protocolIP protocol used (4/6)

ICMP probing is ideal for network device availability monitoring — switches, routers, firewalls, and other devices that don’t support HTTP.

VII. DNS Probing

7.1 DNS Configuration

modules:
  # Check A record resolution
  dns_a_record:
    prober: dns
    dns:
      query_name: "www.example.com"
      query_type: A
      valid_rcodes: [NOERROR]

  # Check specific DNS server
  dns_custom_server:
    prober: dns
    dns:
      query_name: "internal.example.com"
      query_type: A
      transport_protocol: udp
      preferred_ip_protocol: ip4
      ip_protocol_fallback: false

7.2 Prometheus Configuration

scrape_configs:
  - job_name: 'blackbox-dns'
    metrics_path: /probe
    params:
      module: [dns_a_record]
    static_configs:
      - targets:
          - '8.8.8.8'          # Google DNS
          - '1.1.1.1'          # Cloudflare DNS
          - 'dns-internal:53'  # Internal DNS
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115

VIII. SSL Certificate Expiry Monitoring

8.1 Probe Configuration

SSL certificate expiry is the most common “non-technical failure” — a forgotten certificate renewal can take down an entire website.

scrape_configs:
  - job_name: 'ssl-cert-monitor'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://www.example.com
          - https://api.example.com
          - https://admin.example.com
          - https://grafana.example.com
          - https://prometheus.example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115

8.2 Certificate Expiry Alerts

groups:
  - name: ssl-cert
    rules:
      # Certificate expiring within 30 days
      - alert: SSLCertExpiringSoon
        expr: |
          probe_ssl_earliest_cert_expiry - time() < 30 * 24 * 3600          
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "SSL certificate expiring soon: {{ $labels.instance }}"
          description: "Certificate will expire in {{ $value | humanizeDuration }}"

      # Certificate expiring within 7 days
      - alert: SSLCertExpiringCritical
        expr: |
          probe_ssl_earliest_cert_expiry - time() < 7 * 24 * 3600          
        for: 1h
        labels:
          severity: critical
        annotations:
          summary: "SSL certificate critical expiry: {{ $labels.instance }}"
          description: "Certificate will expire within 7 days, renew immediately!"
          runbook: "https://wiki.internal/runbooks/ssl-renewal"

      # Certificate already expired
      - alert: SSLCertExpired
        expr: |
          probe_ssl_earliest_cert_expiry - time() < 0          
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "SSL certificate expired: {{ $labels.instance }}"
          description: "Certificate has expired, website may be inaccessible!"

8.3 Certificate Info Dashboard

# Days until certificate expiry
probe_ssl_earliest_cert_expiry - time() / 86400

# Certificate issuer
probe_ssl_last_information{info="issuer"}

# Certificate subject
probe_ssl_last_information{info="subject"}

IX. Multi-Region Probing

9.1 Multi-Region Deployment Architecture

┌─── Beijing Datacenter ────────────┐
│  Blackbox-Beijing                  │
│  Probes national user entry point  │
└──────────────┬─────────────────────┘
         ┌──────────────┐
         │  Prometheus  │
         │  (aggregated) │
         └──────┬───────┘
┌─── Shanghai Datacenter ───────────┐
│  Blackbox-Shanghai                 │
│  Probes East China user entry      │
└────────────────────────────────────┘

┌─── Guangzhou Datacenter ──────────┐
│  Blackbox-Guangzhou                │
│  Probes South China user entry     │
└────────────────────────────────────┘

┌─── Overseas Node ─────────────────┐
│  Blackbox-Overseas                 │
│  Probes overseas user entry        │
└────────────────────────────────────┘

9.2 Multi-Region Prometheus Configuration

scrape_configs:
  # Beijing node probing
  - job_name: 'blackbox-beijing'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://www.example.com
        labels:
          region: beijing
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-beijing:9115

  # Shanghai node probing
  - job_name: 'blackbox-shanghai'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://www.example.com
        labels:
          region: shanghai
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-shanghai:9115

  # Guangzhou node probing
  - job_name: 'blackbox-guangzhou'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://www.example.com
        labels:
          region: guangzhou
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-guangzhou:9115

9.3 Multi-Region Latency Comparison

# Latency comparison across regions
probe_duration_seconds{job=~"blackbox-.*"} by (region, instance)

# Latency variance between regions
max(probe_duration_seconds) by (instance) - min(probe_duration_seconds) by (instance)

9.4 Multi-Region Alerting

groups:
  - name: multi-region
    rules:
      # All regions unreachable → critical
      - alert: ServiceDownAllRegions
        expr: |
          count by(instance) (probe_success == 0) >= 3          
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Service unreachable from all regions: {{ $labels.instance }}"
          description: "3 regions simultaneously failed probing, service may be completely down"

      # Single region unreachable → warning
      - alert: ServiceDownSingleRegion
        expr: |
          probe_success == 0 and on(instance)
          count by(instance) (probe_success == 0) < 3          
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Service unreachable from single region: {{ $labels.instance }} ({{ $labels.region }})"
          description: "Only {{ $labels.region }} failed probing, likely a network routing issue"

      # High latency variance between regions
      - alert: HighLatencyVariance
        expr: |
          (max by(instance) (probe_duration_seconds) -
           min by(instance) (probe_duration_seconds)) > 2          
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High latency variance between regions: {{ $labels.instance }}"
          description: "Difference between fastest and slowest region exceeds 2 seconds"

X. Availability SLO Measurement

10.1 Calculating Availability

# Availability = successful probes / total probes
avg_over_time(probe_success[30d]) * 100

# By service
avg by(instance) (avg_over_time(probe_success[30d])) * 100

# Compare against SLO target (e.g., 99.9%)
avg_over_time(probe_success[30d]) * 100 < 99.9

10.2 SLO Alerting

groups:
  - name: availability-slo
    rules:
      # Availability below 99.9% (30-day window)
      - alert: AvailabilitySLOBreach
        expr: |
          avg by(instance) (avg_over_time(probe_success[30d])) < 0.999          
        for: 5m
        labels:
          severity: warning
          slo: availability-999
        annotations:
          summary: "Availability SLO breach: {{ $labels.instance }}"
          description: "Past 30-day availability {{ $value | humanizePercentage }}, below 99.9% target"

      # Availability below 99% (critical breach)
      - alert: AvailabilitySLOCritical
        expr: |
          avg by(instance) (avg_over_time(probe_success[30d])) < 0.99          
        for: 5m
        labels:
          severity: critical
          slo: availability-999
        annotations:
          summary: "Critical availability SLO breach: {{ $labels.instance }}"
          description: "Past 30-day availability {{ $value | humanizePercentage }}, below 99%"

10.3 Response Time SLO

groups:
  - name: latency-slo
    rules:
      # P95 response time > 500ms
      - alert: LatencySLOBreach
        expr: |
          histogram_quantile(0.95,
            sum by(le, instance) (rate(probe_duration_seconds_bucket[5m]))
          ) > 0.5          
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Response time SLO breach: {{ $labels.instance }}"
          description: "P95 response time {{ $value }}s, exceeding 500ms target"

XI. Dynamic Target Discovery

11.1 Discovering Probe Targets from Consul

scrape_configs:
  - job_name: 'blackbox-consul'
    metrics_path: /probe
    params:
      module: [http_2xx]
    consul_sd_configs:
      - server: 'consul:8500'
        services: ['web', 'api']
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115
      # Get probe module from Consul metadata
      - source_labels: [__meta_consul_service_metadata_probe_module]
        target_label: __param_module
        regex: (.+)
        replacement: '${1}'

11.2 Discovering from Kubernetes Ingress

scrape_configs:
  - job_name: 'blackbox-k8s-ingress'
    metrics_path: /probe
    params:
      module: [http_2xx]
    kubernetes_sd_configs:
      - role: ingress
    relabel_configs:
      # Only probe Ingresses with prometheus.io/probe annotation
      - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]
        action: keep
        regex: true
      # Use Ingress host as probe target
      - source_labels: [__meta_kubernetes_ingress_scheme, __address__, __meta_kubernetes_ingress_path]
        regex: (.+);(.+);(.+)
        target_label: __param_target
        replacement: ${1}://${2}${3}
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115

XII. Grafana Dashboard

┌─────────────────────────────────────────────────────┐
│              Blackbox Exporter Dashboard              │
│                                                     │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐         │
│  │ Total    │  │ Success  │  │ Avg      │         │
│  │ Probes   │  │ Rate     │  │ Latency  │         │
│  │   142    │  │  99.3%   │  │  127ms   │         │
│  └──────────┘  └──────────┘  └──────────┘         │
│                                                     │
│  ┌─────────────────────────────────────────┐       │
│  │  Probe Results Timeline (success trend) │       │
│  └─────────────────────────────────────────┘       │
│                                                     │
│  ┌─────────────────────────────────────────┐       │
│  │  Probe Latency Timeline                  │       │
│  └─────────────────────────────────────────┘       │
│                                                     │
│  ┌──────────────────────┐ ┌────────────────────┐  │
│  │  Failed Targets      │ │  SSL Cert Days     │  │
│  │  Target | Status     │ │  Domain | Days Left│  │
│  └──────────────────────┘ └────────────────────┘  │
└─────────────────────────────────────────────────────┘

12.2 Key PromQL Queries

# Overall success rate
avg(probe_success) * 100

# Success rate by target
avg by(instance) (probe_success) * 100

# Average probe latency
avg(probe_duration_seconds)

# P95/P99 probe latency
histogram_quantile(0.95, sum by(le) (rate(probe_duration_seconds_bucket[5m])))
histogram_quantile(0.99, sum by(le) (rate(probe_duration_seconds_bucket[5m])))

# SSL certificate days remaining
(probe_ssl_earliest_cert_expiry - time()) / 86400

# Phase duration distribution
probe_http_duration_seconds{phase="dns"}
probe_http_duration_seconds{phase="connect"}
probe_http_duration_seconds{phase="tls"}
probe_http_duration_seconds{phase="transfer"}

Summary

Blackbox Exporter is the only external probing tool in the Prometheus ecosystem, filling the gap of “user-perspective availability monitoring”:

  • Multi-protocol probing: HTTP/TCP/ICMP/DNS — four probe types covering network layer to application layer
  • SSL certificate monitoring: Automatically detects certificate expiry, alerting in advance to prevent cert-related outages
  • Multi-region probing: Probes the same target from different regions, discovering regional network issues and CDN misconfigurations
  • Availability SLO: Calculates 30-day availability based on probe_success, directly measuring user-perspective SLO
  • Dynamic discovery: Supports discovering probe targets from Consul/Kubernetes, automatically following infrastructure changes
  • Rich metrics: Phase-by-phase durations (DNS/TCP/TLS/Transfer) help pinpoint latency bottlenecks

Blackbox Exporter should be a standard component of every monitoring system — internal health (Prometheus + Exporter) and external reachability (Blackbox Exporter) are both indispensable, together forming complete availability monitoring.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

  1. Blackbox Exporter Official Documentation — GitHub, referenced for Blackbox Exporter Official Documentation