Prometheus Service Discovery Mechanisms Explained

Overview

In the Prometheus monitoring system, Service Discovery (SD) is the bridge connecting “monitoring targets” to the “scrape engine.” When your infrastructure scales from a few VMs to hundreds of Kubernetes Pods, cross-AZ cloud instances, and Consul-registered nodes, manually maintaining static_configs becomes a nightmare — every scale-up, scale-down, or migration requires config changes and Prometheus restarts, and alerts may misfire due to unreachable targets.

Prometheus natively supports over a dozen service discovery mechanisms that can automatically detect target changes without restarts. This article starts from static configuration and progressively covers mainstream solutions including file_sd, kubernetes_sd, consul_sd, dns_sd, and ec2_sd. It provides a detailed explanation of relabel_configs — the core label management capability — and concludes with practical multi-cluster monitoring implementations.

Reference: Prometheus Official Documentation — Configuration

I. Why Service Discovery

1.1 Limitations of Static Configuration

Here’s the simplest static configuration:

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets:
          - '192.168.1.10:9100'
          - '192.168.1.11:9100'
          - '192.168.1.12:9100'
        labels:
          env: 'production'
          region: 'beijing'

This works fine when server count is fixed. But consider these scenarios:

Scenario	Static Config Pain Point
Kubernetes Pod scaling	Pod IPs change on every recreation; manual config updates are impractical
Cloud Auto Scaling	New instances after elastic scaling can’t be monitored, creating blind spots
Blue-green / Canary deployments	New version instances need to automatically join monitoring
Multi-datacenter migration	IP ranges change, requiring batch config modifications
Containerized microservices	Instance counts change constantly, with short lifecycles

1.2 Core Value of Service Discovery

┌─────────────┐         ┌───────────────────┐         ┌──────────────┐
│  Service      │ ← senses → │  Prometheus SD    │ ← scrapes → │  Target       │
│  Registry     │           │  (auto-updates)   │           │  (Exporter)   │
│ (Consul/K8s)  │           │                   │           │               │
└─────────────┘         └───────────────────┘         └──────────────┘
                               │
                               ▼
                      ┌───────────────────┐
                      │  relabel_configs   │
                      │  (label filtering  │
                      │   / rewriting)     │
                      └───────────────────┘

Service discovery transforms Prometheus from “passive configuration” to “active sensing”:

Auto-discovery: New instances are automatically added to monitoring without manual intervention
Auto-removal: Offline instances are automatically removed from the target list
Label enrichment: Metadata is fetched from the service registry and automatically applied as labels
Dynamic filtering: Flexible control over scrape scope via relabel

II. file_sd: File-Based Service Discovery

file_sd is the simplest and most flexible service discovery method. Prometheus periodically reads specified files (JSON or YAML), and file content changes take effect automatically.

2.1 Configuration Example

scrape_configs:
  - job_name: 'file-sd-nodes'
    file_sd_configs:
      - files:
          - '/etc/prometheus/targets/nodes/*.yml'
          - '/etc/prometheus/targets/databases/*.json'
        refresh_interval: 30s

Target file format (YAML):

# /etc/prometheus/targets/nodes/web-servers.yml
- targets:
    - 'web-01.example.com:9100'
    - 'web-02.example.com:9100'
  labels:
    env: 'production'
    role: 'web'
    region: 'beijing'
- targets:
    - 'web-03.example.com:9100'
  labels:
    env: 'staging'
    role: 'web'
    region: 'shanghai'

Target file format (JSON):

[
  {
    "targets": ["db-01.example.com:9100", "db-02.example.com:9100"],
    "labels": {
      "env": "production",
      "role": "database",
      "team": "dba"
    }
  }
]

2.2 Use Cases for file_sd

file_sd is essentially a decoupled pattern of “external program writes file, Prometheus reads file.” Its advantages:

Simple integration with existing systems: CMDB and asset management scripts only need to output JSON/YAML files
Version control friendly: Target files can be managed in Git
No extra dependencies: No need to deploy a registry like Consul

A common pattern is to use scripts or CI/CD pipelines to periodically update target files:

#!/bin/bash
# sync-from-cmdb.sh — Sync monitoring targets from CMDB
# Executed by cron every 5 minutes

CMDB_API="http://cmdb.internal/api/v1/hosts"
OUTPUT_DIR="/etc/prometheus/targets/nodes"

# Pull host list from CMDB
curl -s "$CMDB_API?env=production" | \
  jq '[.[] | select(.status == "active") | {
    targets: [.hostname + ":9100"],
    labels: {
      env: .env,
      role: .role,
      region: .region,
      instance_id: .instance_id
    }
  }]' > "$OUTPUT_DIR/production.yml"

echo "[$(date)] Synced $(jq 'map(.targets) | flatten | length' $OUTPUT_DIR/production.yml) targets"

Note: After a file_sd file changes, Prometheus detects it within refresh_interval. If Prometheus reads an incomplete file during writing, it ignores it and retains the last valid configuration — it won’t lose monitoring due to interrupted file writes.

III. kubernetes_sd: Kubernetes Service Discovery

kubernetes_sd is the most commonly used service discovery method in cloud-native environments. Prometheus can directly fetch the list of Kubernetes resources to monitor from the Kubernetes API Server.

3.1 Role Types

kubernetes_sd supports 7 roles, each discovering different Kubernetes resources:

Role	Discovery Target	Typical Use
`node`	Cluster nodes	Monitor node resources (node-exporter)
`pod`	All Pods	Monitor application custom metrics
`service`	Services	Discover targets by service
`endpoints`	Endpoints	Monitor Service backend Pods
`ingress`	Ingress routes	Discover by ingress routing
`eplices`	EndpointSlice	Same as endpoints, recommended for K8s 1.21+
`container`	Containers	Discover by container

3.2 Monitoring Nodes (node role)

scrape_configs:
  - job_name: 'k8s-nodes'
    kubernetes_sd_configs:
      - role: node
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecure_skip_verify: true
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics

Here, relabel_configs changes metrics_path to access node metrics through the API Server proxy. The labelmap action maps K8s node labels (e.g., node-role.kubernetes.io/worker) to Prometheus labels.

3.3 Monitoring Pods (pod role)

scrape_configs:
  - job_name: 'k8s-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Only scrape Pods with the prometheus.io/scrape annotation
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      # Use the port specified in the annotation
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      # Use the path specified in the annotation
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      # Preserve namespace label
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
      # Map all Pod labels
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)

This pattern controls whether a Pod is scraped via Pod annotations:

# Pod is automatically discovered by Prometheus after adding annotations
apiVersion: v1
kind: Pod
metadata:
  name: my-app
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
    prometheus.io/path: "/metrics"

3.4 Monitoring Endpoints (endpoints role)

The endpoints role is the recommended way to discover Service backend instances, especially for monitoring application metrics behind K8s Services:

scrape_configs:
  - job_name: 'k8s-endpoints'
    kubernetes_sd_configs:
      - role: endpoints
    relabel_configs:
      # Only scrape Services with the prometheus.io/scrape annotation
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)
        replacement: ${1}
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_service_name]
        target_label: service

3.5 ServiceMonitor: A More Elegant K8s Monitoring Declaration

In the Kubernetes ecosystem, the Prometheus Operator introduced the ServiceMonitor CRD, which manages scrape configurations declaratively — far more elegant than hand-writing relabel rules:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app-monitor
  namespace: monitoring
  labels:
    release: prometheus  # Match the Prometheus Operator's selector
spec:
  selector:
    matchLabels:
      app: my-app  # Select Services with the app=my-app label
  namespaceSelector:
    matchNames:
      - production
      - staging
  endpoints:
    - port: metrics
      path: /metrics
      interval: 15s
      scrapeTimeout: 10s

ServiceMonitor advantages:

Declarative: Configuration as code, GitOps-ready
Namespace isolation: Different teams manage their own ServiceMonitors
Auto-discovery: Creating a Service resource is all that’s needed for monitoring — no Prometheus config changes required

IV. consul_sd: Consul Service Discovery

Consul is a service registration and discovery tool by HashiCorp, widely used in non-Kubernetes environments (VMs, bare metal).

4.1 Consul SD Configuration

scrape_configs:
  - job_name: 'consul-services'
    consul_sd_configs:
      - server: 'consul:8500'
        services:
          - 'web'
          - 'api'
          - 'worker'
        tags:
          - 'production'
        refresh_interval: 30s
    relabel_configs:
      # Extract labels from Consul metadata
      - source_labels: [__meta_consul_service]
        target_label: service
      - source_labels: [__meta_consul_node]
        target_label: node
      - source_labels: [__meta_consul_service_id]
        target_label: service_id
      - source_labels: [__meta_consul_datacenter]
        target_label: datacenter
      # Extract service tags
      - source_labels: [__meta_consul_tags]
        target_label: env
        regex: '.*,production,.*'
        replacement: 'production'
      # Filter: only scrape services tagged with metrics
      - source_labels: [__meta_consul_service_metadata_metrics]
        action: keep
        regex: .+

4.2 Consul Metadata

Consul SD provides rich __meta_ labels for relabel use:

Metadata Label	Description
`__meta_consul_address`	Service address
`__meta_consul_dc`	Datacenter
`__meta_consul_service`	Service name
`__meta_consul_service_id`	Service instance ID
`__meta_consul_service_address`	Service address
`__meta_consul_service_port`	Service port
`__meta_consul_tags`	Service tags (comma-separated)
`__meta_consul_service_metadata_<key>`	Custom metadata

4.3 Registering Services to Consul

Applications register themselves with Consul on startup:

{
  "ID": "web-01",
  "Name": "web",
  "Tags": ["production", "metrics"],
  "Address": "192.168.1.10",
  "Port": 9100,
  "Meta": {
    "metrics": "true",
    "team": "platform"
  },
  "Check": {
    "HTTP": "http://192.168.1.10:9100/metrics",
    "Interval": "10s"
  }
}

Custom metadata is attached via the Meta field, accessible by Prometheus through __meta_consul_service_metadata_<key>.

V. dns_sd: DNS Service Discovery

dns_sd discovers targets through DNS queries, suitable for scenarios using SRV or A records to manage services.

scrape_configs:
  - job_name: 'dns-sd'
    dns_sd_configs:
      - names:
          - '_metrics._tcp.service.consul'      # SRV record
          - 'api.service.production.consul'
        type: A
        port: 9100
        refresh_interval: 30s
      - names:
          - '_prometheus._tcp.example.com'      # SRV record
        type: SRV

The advantage of dns_sd is that it requires no additional components — as long as the infrastructure supports DNS. The downside is that DNS records carry limited information and can’t provide rich metadata like Consul.

Production tip: dns_sd works best as a backend for Consul — Consul automatically manages DNS records, and Prometheus queries via dns_sd, achieving decoupled service discovery.

VI. ec2_sd: AWS EC2 Service Discovery

If your infrastructure is on AWS, ec2_sd can discover instances directly from the EC2 API:

scrape_configs:
  - job_name: 'ec2-nodes'
    ec2_sd_configs:
      - region: us-east-1
        access_key: AKIAIOSFODNN7EXAMPLE
        secret_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
        port: 9100
        refresh_interval: 60s
        filters:
          - name: tag:Monitoring
            values: [enabled]
    relabel_configs:
      - source_labels: [__meta_ec2_availability_zone]
        target_label: az
      - source_labels: [__meta_ec2_instance_id]
        target_label: instance_id
      - source_labels: [__meta_ec2_private_ip]
        target_label: private_ip
      # Extract instance tags
      - action: labelmap
        regex: __meta_ec2_tag_(.+)
      # Only scrape instances tagged with Monitoring=enabled
      - source_labels: [__meta_ec2_tag_Monitoring]
        action: keep
        regex: enabled

Similarly, Azure uses azure_sd_configs, GCP uses gce_sd_configs, and OpenStack uses openstack_sd_configs.

VII. relabel_configs: The Core of Label Management

relabel_configs is the most powerful mechanism in Prometheus service discovery. It filters, rewrites, and maps labels before a target is scraped. Understanding relabel is the key to mastering Prometheus SD.

7.1 relabel Execution Timing

Service discovery → produces raw targets (with __meta_ labels)
                 │
                 ▼
     ┌─────────────────────┐
     │   relabel_configs    │  ← Before scraping: controls whether to scrape, rewrites address/path
     └──────────┬──────────┘
                │
                ▼
     ┌─────────────────────┐
     │     Scrape metrics   │
     └──────────┬──────────┘
                │
                ▼
     ┌─────────────────────┐
     │  metric_relabel_configs │  ← After scraping, before storage: controls whether to store, rewrites metric labels
     └─────────────────────┘

relabel_configs: Executes before scraping; can control whether to scrape a target, modify the scrape address, path, or scheme
metric_relabel_configs: Executes after scraping and before storage; can drop unwanted metrics or rewrite metric labels

7.2 relabel Actions

Action	Purpose	Typical Scenario
`replace`	Replace or add label values	Rewrite `__address__`, add custom labels
`keep`	Keep matching targets, drop non-matching	Only scrape Pods in specific namespaces
`drop`	Drop matching targets	Exclude targets from specific environments
`labelmap`	Map a set of labels to new labels	Map K8s/Consul `__meta_` labels
`labelkeep`	Keep matching labels	Clean up redundant labels
`labeldrop`	Drop matching labels	Remove high-cardinality labels
`lowercase`	Convert label values to lowercase	Normalize label format
`uppercase`	Convert label values to uppercase	Normalize label format
`hashmod`	Modulo on label values	Multi-Prometheus shard scraping

7.3 Practical Examples

Only scrape production environment Pods:

relabel_configs:
  - source_labels: [__meta_kubernetes_namespace]
    action: keep
    regex: (production|production-.+)

Sharding by label (two Prometheus replicas each scrape half the targets):

relabel_configs:
  - source_labels: [__address__]
    modulus: 2       # Total 2 shards
    target_label: __tmp_hash
    action: hashmod
  - source_labels: [__tmp_hash]
    regex: 0         # Current Prometheus only scrapes targets with hash=0
    action: keep

Drop unwanted high-cardinality metrics:

metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'go_(gc|memstats)_.+'
    action: drop

Rename metric labels:

metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'http_requests_total'
    target_label: __name__
    replacement: 'http_requests_total'
    action: replace

VIII. Label Management Best Practices

Labels are the dimensional identifiers of Prometheus time series. Good label design directly impacts query efficiency and storage overhead.

8.1 Label Design Principles

Principle	Description	Example
Low cardinality	Limited number of label values	`env="prod"` ✓, `user_id="12345"` ✗
Business meaning	Labels used for aggregation and filtering	`service="payment"` ✓, `ip="10.0.1.5"` usually meaningless
Consistent naming	Team-agreed label naming conventions	`env`, `service`, `team`, `severity`
Controlled count	No more than 10 labels per time series	Too many labels impact query performance

8.2 Label Naming Conventions

# Recommended label hierarchy
env       → environment identifier (production/staging/dev)
service   → microservice name
instance  → instance identifier (auto-added by Prometheus)
team      → responsible team
severity  → alert level (critical/warning/info)

8.3 Avoiding High-Cardinality Labels

# Dangerous: user_id as a label, one time series per user
metric_relabel_configs:
  - source_labels: [user_id]
    target_label: user_id  # ✗ Disastrous practice

# Correct: Remove high-cardinality labels, keep only aggregated data
metric_relabel_configs:
  - action: labeldrop
    regex: 'user_id|session_id|request_id'

Storage cost reminder: Each time series in Prometheus costs approximately 1-3 KB in storage. With 100,000 users, the user_id label alone would create 100,000 time series, severely degrading query and write performance.

IX. Multi-Cluster Monitoring Solutions

In multi-Kubernetes-cluster or multi-datacenter environments, service discovery needs to cross cluster boundaries.

9.1 Solution 1: Prometheus + Thanos Sidecar (Recommended)

┌─── Cluster A (K8s) ────────────────────┐
│  Prometheus-A ─── Thanos Sidecar ──────┼──┐
│  (kubernetes_sd: local cluster)        │  │
└────────────────────────────────────────┘  │
                                              │  Thanos Store
┌─── Cluster B (K8s) ────────────────────┐  │  (global query)
│  Prometheus-B ─── Thanos Sidecar ──────┼──┘
│  (kubernetes_sd: local cluster)        │
└────────────────────────────────────────┘

Each cluster deploys an independent Prometheus using kubernetes_sd to monitor its own cluster. Thanos Sidecar uploads data to object storage, and Thanos Query provides a global query entry point.

9.2 Solution 2: Federation

# Global Prometheus configuration
scrape_configs:
  - job_name: 'federate'
    scrape_interval: 30s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="node"}'
        - '{job="kubernetes"}'
        - '{__name__=~"up|prometheus_.*"}'
    static_configs:
      - targets:
          - 'prometheus-cluster-a:9090'
          - 'prometheus-cluster-b:9090'
    relabel_configs:
      - source_labels: [__address__]
        regex: 'prometheus-(.+):9090'
        target_label: cluster
        replacement: '${1}'

Federation is simple but increases query load on sub-Prometheus instances, suitable for small-scale clusters. For large-scale scenarios, Thanos or VictoriaMetrics is recommended.

9.3 Solution 3: Remote Write

# Each cluster's Prometheus configured with remote write
remote_write:
  - url: 'https://mimir-central.example.com/api/v1/push'
    headers:
      X-Scope-OrgID: 'tenant-a'
    write_relabel_configs:
      # Only upload key metrics to reduce bandwidth
      - source_labels: [__name__]
        regex: 'up|node_.+|container_.+|http_requests_total'
        action: keep

Each cluster’s Prometheus pushes data to a central storage (Mimir/Thanos Receive/VictoriaMetrics) via remote_write, enabling centralized monitoring.

X. Multi-Solution Comparison

Dimension	file_sd	kubernetes_sd	consul_sd	dns_sd	ec2_sd
Applicable environment	General	Kubernetes	VMs/Hybrid	DNS infrastructure	AWS EC2
Metadata richness	Low (hand-written)	High (K8s labels/annotations)	High (Consul tags/meta)	Low	Medium (EC2 tags)
Auto-discovery	Semi-auto (needs script)	Fully automatic	Fully automatic	Semi-auto	Fully automatic
Extra dependencies	None	K8s API Server	Consul Server	DNS Server	AWS API
Operational complexity	Low	Low (K8s native)	Medium	Low	Low
Typical scenario	CMDB integration	Cloud-native monitoring	Microservice registry	Simple DNS discovery	AWS infrastructure monitoring

XI. Common Issues and Troubleshooting

11.1 Target Shows as Down

# Check target status
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.health != "up")'

# View target's lastError
curl -s http://localhost:9090/api/v1/targets | \
  jq '.data.activeTargets[] | select(.health != "up") | {scrapeUrl, lastError, labels}'

Common causes:

Network unreachable: Network policy restrictions between Prometheus and target
Certificate issues: Certificate mismatch during HTTPS scraping
Authentication failure: Expired bearer token or insufficient permissions
Misconfigured relabel: relabel changed __address__ to an incorrect address

11.2 Empty Target List

# View raw targets discovered by service discovery
curl -s http://localhost:9090/api/v1/targets?state=any | jq '.data.droppedTargets'

If droppedTargets has data but activeTargets is empty, it means relabel_configs keep/drop rules filtered out all targets. Check whether the keep regex is correct.

11.3 Missing or Incorrect Labels

# View all labels for a target
curl -s http://localhost:9090/api/v1/targets | \
  jq '.data.activeTargets[0].discoveredLabels'

discoveredLabels contains all __meta_ prefixed raw labels — verify whether service discovery returned the expected metadata.

Summary

Service discovery is the “nerve endings” of the Prometheus monitoring system, determining monitoring coverage and automation level. Key takeaways:

Choose the right SD solution: Use kubernetes_sd for Kubernetes environments, consul_sd or file_sd for non-K8s, and native SD (ec2/gce/azure) for cloud providers
Master relabel_configs: It’s the core of label management, determining which targets are scraped, how labels are mapped, and how data is sharded
Design a sound label system: Low cardinality, business-meaningful, consistently named — avoid high-cardinality labels that drag down Prometheus
Use Thanos/VM for multi-cluster: Federation suits small scale; for large-scale scenarios, prefer Thanos or VictoriaMetrics remote write
Leverage ServiceMonitor: In K8s environments, Prometheus Operator + ServiceMonitor is the standard for declarative monitoring management

There’s no “best solution” for service discovery — only “the solution best suited to your environment.” Understanding how each SD mechanism works and its applicable scenarios is key to making the right choice.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

Prometheus Official Documentation — Configuration — Prometheus Authors, referenced for Prometheus Official Documentation — Configuration

Overview#

I. Why Service Discovery#

1.1 Limitations of Static Configuration#

1.2 Core Value of Service Discovery#

II. file_sd: File-Based Service Discovery#

2.1 Configuration Example#

2.2 Use Cases for file_sd#

III. kubernetes_sd: Kubernetes Service Discovery#

3.1 Role Types#

3.2 Monitoring Nodes (node role)#

3.3 Monitoring Pods (pod role)#

3.4 Monitoring Endpoints (endpoints role)#

3.5 ServiceMonitor: A More Elegant K8s Monitoring Declaration#

IV. consul_sd: Consul Service Discovery#

4.1 Consul SD Configuration#

4.2 Consul Metadata#

4.3 Registering Services to Consul#

V. dns_sd: DNS Service Discovery#

VI. ec2_sd: AWS EC2 Service Discovery#

VII. relabel_configs: The Core of Label Management#

7.1 relabel Execution Timing#

7.2 relabel Actions#

7.3 Practical Examples#

VIII. Label Management Best Practices#

8.1 Label Design Principles#

8.2 Label Naming Conventions#

8.3 Avoiding High-Cardinality Labels#

IX. Multi-Cluster Monitoring Solutions#

9.1 Solution 1: Prometheus + Thanos Sidecar (Recommended)#

9.2 Solution 2: Federation#

9.3 Solution 3: Remote Write#

X. Multi-Solution Comparison#

XI. Common Issues and Troubleshooting#

11.1 Target Shows as Down#

11.2 Empty Target List#

11.3 Missing or Incorrect Labels#

Summary#

References & Acknowledgments#