Overview
Prometheus’s conventional monitoring is “inside-out” — Prometheus scrapes Exporter metrics to understand internal system state. But when users access your service, they follow an “outside-in” path: DNS resolution → network routing → load balancing → backend service. A link that looks perfectly healthy from the inside may be completely inaccessible to users due to DNS misconfiguration, CDN cache issues, or expired SSL certificates.
Blackbox Exporter solves the “external probing” problem. It simulates user behavior, probing HTTP/TCP/ICMP/DNS endpoints from the outside, giving you “user perspective” availability data. This article systematically covers Blackbox Exporter configuration, probing methods, SSL certificate monitoring, multi-region probing, and availability SLO measurement.
Reference: Blackbox Exporter Official Documentation
I. Why External Probing
1.1 Blind Spots of Internal Monitoring
User request path:
User → DNS → CDN → Load Balancer → Ingress → Pod → Database
Internal monitoring coverage:
✓ Pod metrics
✓ Database metrics
✓ Ingress metrics
✓ LB metrics
External blind spots:
✗ Is DNS resolution working?
✗ Is CDN caching correct?
✗ Has the SSL certificate expired?
✗ Is network routing clear?
✗ Is access reachable from user's region?
Typical scenarios where internal monitoring falls short:
| Scenario | Internal Monitoring | External Probing |
|---|---|---|
| DNS resolution failure | Not visible | ✓ Detectable |
| SSL certificate expired | Not visible (needs proactive check) | ✓ Detectable |
| CDN cache error | Not visible | ✓ Detectable |
| Network routing outage | Not visible | ✓ Detectable |
| Slow access from specific region | Not visible | ✓ Multi-region probing |
| Page content tampered | Not visible | ✓ HTTP content check |
1.2 Blackbox Exporter’s Role
┌────────────────────────────────────────────────────┐
│ Monitoring Perspective Comparison │
│ │
│ Internal Monitoring (Prometheus + Exporter) │
│ → "Is the system healthy?" │
│ → CPU, memory, disk, QPS, latency │
│ │
│ External Probing (Blackbox Exporter) │
│ → "Can users access it?" │
│ → HTTP status code, response time, cert validity │
│ → TCP port reachability, DNS resolution correctness│
│ │
│ Both complement: internal health + external │
│ reachability = complete availability │
└────────────────────────────────────────────────────┘
II. Blackbox Exporter Architecture
2.1 How It Works
┌───────────────┐ ┌───────────────────────┐
│ Prometheus │ scrape │ Blackbox Exporter │
│ (scraper) │ ──────→ │ (prober) │
└───────────────┘ └───────────┬───────────┘
│
┌────────────┼────────────┐
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ HTTP │ │ TCP │ │ ICMP │
│ probe │ │ probe │ │ probe │
└────┬────┘ └────┬────┘ └────┬────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────┐
│ Probed Target │
│ web.example.com / 10.0.1.5:3306 │
└─────────────────────────────────────┘
Core characteristics of Blackbox Exporter:
- Pull-mode integration: Prometheus scrapes Blackbox Exporter to get probe results
- Parameterized probing: Target and module specified via URL parameters
- Modular configuration: Different probe types (HTTP/TCP/ICMP/DNS) use different modules
- Rich metrics returned: Probe results include status code, response time, certificate info, etc.
2.2 Probe URL Format
# HTTP probe
http://blackbox:9115/probe?target=https://example.com&module=http_2xx
# TCP probe
http://blackbox:9115/probe?target=10.0.1.5:3306&module=tcp_connect
# ICMP probe
http://blackbox:9115/probe?target=10.0.1.5&module=icmp
# DNS probe
http://blackbox:9115/probe?target=dns.example.com&module=dns_example
III. Configuration Details
3.1 Complete Configuration File
# blackbox.yml
modules:
# HTTP probe module
http_2xx:
prober: http
timeout: 10s
http:
valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
valid_status_codes: [200, 201, 204] # Acceptable status codes
method: GET
headers:
User-Agent: "Blackbox-Exporter/0.25"
Accept-Language: "zh-CN"
follow_redirects: true
preferred_ip_protocol: ip4 # Prefer IPv4
ip_protocol_fallback: true
# HTTP probe - with authentication
http_with_auth:
prober: http
timeout: 10s
http:
method: POST
headers:
Content-Type: application/json
Authorization: "Bearer token-xxx"
body: '{"action":"health_check"}'
valid_status_codes: [200]
# HTTP probe - content check
http_content_check:
prober: http
timeout: 10s
http:
fail_if_body_matches_regexp:
- "error"
- "maintenance"
fail_if_body_not_matches_regexp:
- "expected_content"
# HTTPS probe - strict SSL verification
http_strict_ssl:
prober: http
timeout: 10s
http:
fail_if_ssl: false
tls_config:
insecure_skip_verify: false
ca_file: /etc/ssl/certs/ca-certificates.crt
# TCP port probe
tcp_connect:
prober: tcp
timeout: 5s
# TCP probe - with protocol handshake
tcp_mysql:
prober: tcp
timeout: 5s
tcp:
query_response:
- send: "\x00\x00\x00\x0a" # MySQL handshake packet
- expect: "^[\\x00-\\xff]" # Match response
# TCP probe - SSH port
tcp_ssh:
prober: tcp
timeout: 5s
tcp:
query_response:
- expect: "^SSH-2.0-"
# ICMP probe
icmp:
prober: icmp
timeout: 5s
icmp:
preferred_ip_protocol: ip4
# DNS probe
dns_check:
prober: dns
timeout: 5s
dns:
query_name: "example.com"
query_type: A
valid_rcodes: [NOERROR]
validate_answer_rrs:
fail_if_matches_regexp:
- ".*127.0.0.1"
fail_if_not_matches_regexp:
- ".*10\\.0\\..*"
# DNS probe - check SOA
dns_soa:
prober: dns
timeout: 5s
dns:
query_name: "example.com"
query_type: SOA
transport_protocol: tcp
valid_rcodes: [NOERROR]
3.2 Docker Deployment
# docker-compose.yml
version: '3.8'
services:
blackbox-exporter:
image: prom/blackbox-exporter:v0.25.0
container_name: blackbox
ports:
- "9115:9115"
volumes:
- ./blackbox.yml:/etc/blackbox_exporter/config.yml
command:
- '--config.file=/etc/blackbox_exporter/config.yml'
- '--web.listen-address=:9115'
- '--log.level=info'
restart: unless-stopped
Note: ICMP probing requires extra permissions. In Docker, add
cap_add: [NET_ADMIN]or use host network mode.
IV. HTTP Probing
4.1 Basic HTTP Probe
# Prometheus scrape configuration
scrape_configs:
- job_name: 'blackbox-http'
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- https://www.example.com
- https://api.example.com/health
- https://app.example.com
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: blackbox:9115
4.2 Metrics Returned by Probing
# Manually test probe results
curl -s "http://blackbox:9115/probe?target=https://example.com&module=http_2xx" | grep probe_
Key metrics returned:
| Metric | Description |
|---|---|
probe_success | Whether probe succeeded (0/1) |
probe_duration_seconds | Total probe duration |
probe_http_status_code | HTTP status code |
probe_http_version | HTTP protocol version |
probe_http_content_length | Response content length |
probe_ssl_earliest_cert_expiry | Earliest SSL cert expiry (Unix timestamp) |
probe_ssl_last_information | SSL certificate info |
probe_dns_lookup_time_seconds | DNS resolution duration |
probe_tcp_connection_seconds | TCP connection duration |
probe_tls_handshake_seconds | TLS handshake duration |
probe_http_duration_seconds{phase=...} | Phase durations (dns/connect/tls/transfer) |
4.3 Phase-by-Phase Duration
# View HTTP request phase durations
probe_http_duration_seconds{phase="dns"} # DNS resolution
probe_http_duration_seconds{phase="connect"} # TCP connection
probe_http_duration_seconds{phase="tls"} # TLS handshake
probe_http_duration_seconds{phase="transfer"} # Data transfer
probe_http_duration_seconds{phase="processing"}# Server processing
These phase metrics help pinpoint latency bottlenecks — whether it’s slow DNS, slow network, or slow server processing.
V. TCP Port Probing
5.1 Basic TCP Probe
scrape_configs:
- job_name: 'blackbox-tcp'
metrics_path: /probe
params:
module: [tcp_connect]
static_configs:
- targets:
- 'mysql:3306'
- 'redis:6379'
- 'kafka:9092'
- 'ssh:22'
- 'smtp:25'
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: blackbox:9115
5.2 Protocol Handshake Probing
Simple port reachability isn’t enough to verify a service is truly available. Use query_response for protocol-layer handshake validation:
modules:
# MySQL health check
tcp_mysql:
prober: tcp
tcp:
query_response:
- expect: "^.*\\x00\\x00\\x00\\x0a.*" # MySQL handshake response
# Redis PING
tcp_redis:
prober: tcp
tcp:
query_response:
- send: "PING\r\n"
- expect: "^\\+PONG"
# SMTP check
tcp_smtp:
prober: tcp
tcp:
query_response:
- expect: "^220 .* SMTP"
- send: "EHLO blackbox\r\n"
- expect: "^250"
# PostgreSQL check
tcp_postgres:
prober: tcp
tcp:
query_response:
- send: "\x00\x00\x00\x08\x04\xd2\x16\x2f" # PgSSLRequest
- expect: "^[SN]" # S=SSL supported, N=not supported
VI. ICMP Probing
6.1 ICMP Configuration
scrape_configs:
- job_name: 'blackbox-icmp'
metrics_path: /probe
params:
module: [icmp]
static_configs:
- targets:
- '192.168.1.1' # Gateway
- '8.8.8.8' # Public DNS
- '10.0.0.1' # Internal core switch
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: blackbox:9115
6.2 ICMP Metrics
| Metric | Description |
|---|---|
probe_success | Whether ICMP response received |
probe_duration_seconds | Round-trip time (RTT) |
probe_ip_protocol | IP protocol used (4/6) |
ICMP probing is ideal for network device availability monitoring — switches, routers, firewalls, and other devices that don’t support HTTP.
VII. DNS Probing
7.1 DNS Configuration
modules:
# Check A record resolution
dns_a_record:
prober: dns
dns:
query_name: "www.example.com"
query_type: A
valid_rcodes: [NOERROR]
# Check specific DNS server
dns_custom_server:
prober: dns
dns:
query_name: "internal.example.com"
query_type: A
transport_protocol: udp
preferred_ip_protocol: ip4
ip_protocol_fallback: false
7.2 Prometheus Configuration
scrape_configs:
- job_name: 'blackbox-dns'
metrics_path: /probe
params:
module: [dns_a_record]
static_configs:
- targets:
- '8.8.8.8' # Google DNS
- '1.1.1.1' # Cloudflare DNS
- 'dns-internal:53' # Internal DNS
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: blackbox:9115
VIII. SSL Certificate Expiry Monitoring
8.1 Probe Configuration
SSL certificate expiry is the most common “non-technical failure” — a forgotten certificate renewal can take down an entire website.
scrape_configs:
- job_name: 'ssl-cert-monitor'
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- https://www.example.com
- https://api.example.com
- https://admin.example.com
- https://grafana.example.com
- https://prometheus.example.com
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: blackbox:9115
8.2 Certificate Expiry Alerts
groups:
- name: ssl-cert
rules:
# Certificate expiring within 30 days
- alert: SSLCertExpiringSoon
expr: |
probe_ssl_earliest_cert_expiry - time() < 30 * 24 * 3600
for: 1h
labels:
severity: warning
annotations:
summary: "SSL certificate expiring soon: {{ $labels.instance }}"
description: "Certificate will expire in {{ $value | humanizeDuration }}"
# Certificate expiring within 7 days
- alert: SSLCertExpiringCritical
expr: |
probe_ssl_earliest_cert_expiry - time() < 7 * 24 * 3600
for: 1h
labels:
severity: critical
annotations:
summary: "SSL certificate critical expiry: {{ $labels.instance }}"
description: "Certificate will expire within 7 days, renew immediately!"
runbook: "https://wiki.internal/runbooks/ssl-renewal"
# Certificate already expired
- alert: SSLCertExpired
expr: |
probe_ssl_earliest_cert_expiry - time() < 0
for: 5m
labels:
severity: critical
annotations:
summary: "SSL certificate expired: {{ $labels.instance }}"
description: "Certificate has expired, website may be inaccessible!"
8.3 Certificate Info Dashboard
# Days until certificate expiry
probe_ssl_earliest_cert_expiry - time() / 86400
# Certificate issuer
probe_ssl_last_information{info="issuer"}
# Certificate subject
probe_ssl_last_information{info="subject"}
IX. Multi-Region Probing
9.1 Multi-Region Deployment Architecture
┌─── Beijing Datacenter ────────────┐
│ Blackbox-Beijing │
│ Probes national user entry point │
└──────────────┬─────────────────────┘
│
▼
┌──────────────┐
│ Prometheus │
│ (aggregated) │
└──────┬───────┘
▲
┌─── Shanghai Datacenter ───────────┐
│ Blackbox-Shanghai │
│ Probes East China user entry │
└────────────────────────────────────┘
┌─── Guangzhou Datacenter ──────────┐
│ Blackbox-Guangzhou │
│ Probes South China user entry │
└────────────────────────────────────┘
┌─── Overseas Node ─────────────────┐
│ Blackbox-Overseas │
│ Probes overseas user entry │
└────────────────────────────────────┘
9.2 Multi-Region Prometheus Configuration
scrape_configs:
# Beijing node probing
- job_name: 'blackbox-beijing'
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- https://www.example.com
labels:
region: beijing
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: blackbox-beijing:9115
# Shanghai node probing
- job_name: 'blackbox-shanghai'
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- https://www.example.com
labels:
region: shanghai
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: blackbox-shanghai:9115
# Guangzhou node probing
- job_name: 'blackbox-guangzhou'
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- https://www.example.com
labels:
region: guangzhou
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: blackbox-guangzhou:9115
9.3 Multi-Region Latency Comparison
# Latency comparison across regions
probe_duration_seconds{job=~"blackbox-.*"} by (region, instance)
# Latency variance between regions
max(probe_duration_seconds) by (instance) - min(probe_duration_seconds) by (instance)
9.4 Multi-Region Alerting
groups:
- name: multi-region
rules:
# All regions unreachable → critical
- alert: ServiceDownAllRegions
expr: |
count by(instance) (probe_success == 0) >= 3
for: 2m
labels:
severity: critical
annotations:
summary: "Service unreachable from all regions: {{ $labels.instance }}"
description: "3 regions simultaneously failed probing, service may be completely down"
# Single region unreachable → warning
- alert: ServiceDownSingleRegion
expr: |
probe_success == 0 and on(instance)
count by(instance) (probe_success == 0) < 3
for: 5m
labels:
severity: warning
annotations:
summary: "Service unreachable from single region: {{ $labels.instance }} ({{ $labels.region }})"
description: "Only {{ $labels.region }} failed probing, likely a network routing issue"
# High latency variance between regions
- alert: HighLatencyVariance
expr: |
(max by(instance) (probe_duration_seconds) -
min by(instance) (probe_duration_seconds)) > 2
for: 10m
labels:
severity: warning
annotations:
summary: "High latency variance between regions: {{ $labels.instance }}"
description: "Difference between fastest and slowest region exceeds 2 seconds"
X. Availability SLO Measurement
10.1 Calculating Availability
# Availability = successful probes / total probes
avg_over_time(probe_success[30d]) * 100
# By service
avg by(instance) (avg_over_time(probe_success[30d])) * 100
# Compare against SLO target (e.g., 99.9%)
avg_over_time(probe_success[30d]) * 100 < 99.9
10.2 SLO Alerting
groups:
- name: availability-slo
rules:
# Availability below 99.9% (30-day window)
- alert: AvailabilitySLOBreach
expr: |
avg by(instance) (avg_over_time(probe_success[30d])) < 0.999
for: 5m
labels:
severity: warning
slo: availability-999
annotations:
summary: "Availability SLO breach: {{ $labels.instance }}"
description: "Past 30-day availability {{ $value | humanizePercentage }}, below 99.9% target"
# Availability below 99% (critical breach)
- alert: AvailabilitySLOCritical
expr: |
avg by(instance) (avg_over_time(probe_success[30d])) < 0.99
for: 5m
labels:
severity: critical
slo: availability-999
annotations:
summary: "Critical availability SLO breach: {{ $labels.instance }}"
description: "Past 30-day availability {{ $value | humanizePercentage }}, below 99%"
10.3 Response Time SLO
groups:
- name: latency-slo
rules:
# P95 response time > 500ms
- alert: LatencySLOBreach
expr: |
histogram_quantile(0.95,
sum by(le, instance) (rate(probe_duration_seconds_bucket[5m]))
) > 0.5
for: 10m
labels:
severity: warning
annotations:
summary: "Response time SLO breach: {{ $labels.instance }}"
description: "P95 response time {{ $value }}s, exceeding 500ms target"
XI. Dynamic Target Discovery
11.1 Discovering Probe Targets from Consul
scrape_configs:
- job_name: 'blackbox-consul'
metrics_path: /probe
params:
module: [http_2xx]
consul_sd_configs:
- server: 'consul:8500'
services: ['web', 'api']
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: blackbox:9115
# Get probe module from Consul metadata
- source_labels: [__meta_consul_service_metadata_probe_module]
target_label: __param_module
regex: (.+)
replacement: '${1}'
11.2 Discovering from Kubernetes Ingress
scrape_configs:
- job_name: 'blackbox-k8s-ingress'
metrics_path: /probe
params:
module: [http_2xx]
kubernetes_sd_configs:
- role: ingress
relabel_configs:
# Only probe Ingresses with prometheus.io/probe annotation
- source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]
action: keep
regex: true
# Use Ingress host as probe target
- source_labels: [__meta_kubernetes_ingress_scheme, __address__, __meta_kubernetes_ingress_path]
regex: (.+);(.+);(.+)
target_label: __param_target
replacement: ${1}://${2}${3}
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: blackbox:9115
XII. Grafana Dashboard
12.1 Recommended Dashboard Layout
┌─────────────────────────────────────────────────────┐
│ Blackbox Exporter Dashboard │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Total │ │ Success │ │ Avg │ │
│ │ Probes │ │ Rate │ │ Latency │ │
│ │ 142 │ │ 99.3% │ │ 127ms │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │
│ ┌─────────────────────────────────────────┐ │
│ │ Probe Results Timeline (success trend) │ │
│ └─────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────┐ │
│ │ Probe Latency Timeline │ │
│ └─────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────┐ ┌────────────────────┐ │
│ │ Failed Targets │ │ SSL Cert Days │ │
│ │ Target | Status │ │ Domain | Days Left│ │
│ └──────────────────────┘ └────────────────────┘ │
└─────────────────────────────────────────────────────┘
12.2 Key PromQL Queries
# Overall success rate
avg(probe_success) * 100
# Success rate by target
avg by(instance) (probe_success) * 100
# Average probe latency
avg(probe_duration_seconds)
# P95/P99 probe latency
histogram_quantile(0.95, sum by(le) (rate(probe_duration_seconds_bucket[5m])))
histogram_quantile(0.99, sum by(le) (rate(probe_duration_seconds_bucket[5m])))
# SSL certificate days remaining
(probe_ssl_earliest_cert_expiry - time()) / 86400
# Phase duration distribution
probe_http_duration_seconds{phase="dns"}
probe_http_duration_seconds{phase="connect"}
probe_http_duration_seconds{phase="tls"}
probe_http_duration_seconds{phase="transfer"}
Summary
Blackbox Exporter is the only external probing tool in the Prometheus ecosystem, filling the gap of “user-perspective availability monitoring”:
- Multi-protocol probing: HTTP/TCP/ICMP/DNS — four probe types covering network layer to application layer
- SSL certificate monitoring: Automatically detects certificate expiry, alerting in advance to prevent cert-related outages
- Multi-region probing: Probes the same target from different regions, discovering regional network issues and CDN misconfigurations
- Availability SLO: Calculates 30-day availability based on
probe_success, directly measuring user-perspective SLO - Dynamic discovery: Supports discovering probe targets from Consul/Kubernetes, automatically following infrastructure changes
- Rich metrics: Phase-by-phase durations (DNS/TCP/TLS/Transfer) help pinpoint latency bottlenecks
Blackbox Exporter should be a standard component of every monitoring system — internal health (Prometheus + Exporter) and external reachability (Blackbox Exporter) are both indispensable, together forming complete availability monitoring.
References & Acknowledgments
This article referenced the following materials during writing. We thank the original authors for their contributions:
- Blackbox Exporter Official Documentation — GitHub, referenced for Blackbox Exporter Official Documentation