Monitoring Data Governance: From Metrics Explosion to Precision Observability

Overview

Picture this: it’s 3 AM, you get woken up by an alert, you drag yourself to open Grafana, and you flip through dozens of dashboards—none of them tell you anything useful. You do have millions of metrics, sure. But they’re all garbage.

This is not an isolated case. I’ve seen too many teams deploy Prometheus and then forget about it. Metrics pile up, alerting rules multiply, and eventually the monitoring system itself crashes first: Prometheus OOMs on memory, queries time out after 30 seconds, alert evaluation lags by 5+ minutes. The monitoring system became the biggest source of outages. Ironic, isn’t it?

Monitoring data governance solves exactly this problem. It’s not some lofty theory—stripped down to one sentence: figure out what metrics you have, which ones are useful, which ones to delete, and how to manage their entire lifecycle.

This article walks through five stages of monitoring data governance from a metric lifecycle perspective: metric definition, collection strategy, storage optimization, quality measurement, and deprecation retirement. Each stage comes with hands-on code and lessons learned the hard way.

Root Cause of Metrics Explosion: It’s Not Too Much Data, It’s No Control

How Metrics Inflate

Metrics inflation doesn’t happen overnight. The typical path goes like this:

1. Early stage: Node Exporter + cAdvisor, a few hundred metrics, Prometheus runs fine
2. Business onboarding: Each service adds instrumentation, each middleware gets an Exporter, metrics grow to tens of thousands
3. High-cardinality bomb: Someone stuffs user_id, request_id, session_id into labels, time series count jumps from tens of thousands to millions
4. Disaster: Prometheus memory spikes, disk fills up, queries hang

The core culprit here is high-cardinality labels.

Prometheus’s time series model is: metric_name{label1="value1", label2="value2"} → value. Each unique combination of label values creates a new time series. For example:

# Low cardinality: 3 time series
http_requests_total{method="GET",status="200"} 
http_requests_total{method="POST",status="200"}
http_requests_total{method="GET",status="404"}

# High-cardinality bomb: 1M users = 1M time series
http_requests_total{method="GET",status="200",user_id="12345"}
http_requests_total{method="GET",status="200",user_id="12346"}
...

That second block looks harmless, but if user_id has 1 million distinct values, that’s 1 million time series. A single Prometheus instance tops out at roughly 500K active TimeSeries (limited by memory and disk I/O), so it blows past the limit immediately.

Common Sources of High-Cardinality Labels

Here’s a simple rule of thumb: if a label can have more than 100 distinct values, it shouldn’t be a label—it belongs in your logs.

Metric Definition Standards: Control at the Source

Build an Enterprise Metric Catalog

The first step in metric governance isn’t deleting data—it’s figuring out what you have. I recommend maintaining a Metric Catalog that records each metric’s owner, purpose, cardinality, and retention period.

# metric-catalog.yaml — Metric catalog template
metrics:
  - name: http_requests_total
    type: counter
    owner: platform-team
    purpose: "Count total HTTP requests for QPS and error rate calculation"
    labels:
      - name: method
        cardinality: low  # GET, POST, PUT, DELETE etc. ~10 values
      - name: status
        cardinality: low  # 200, 404, 500 etc. ~20 values
      - name: handler
        cardinality: medium  # API routes, ~50-200 values
    retention: 90d
    status: active

  - name: http_request_duration_seconds
    type: histogram
    owner: platform-team
    purpose: "HTTP request latency distribution for P50/P95/P99 calculation"
    labels:
      - name: method
        cardinality: low
      - name: handler
        cardinality: medium
    retention: 90d
    status: active

  - name: go_goroutines
    type: gauge
    owner: platform-team
    purpose: "Go runtime goroutine count"
    labels: []
    retention: 30d
    status: active

This catalog isn’t just for show. It serves three practical purposes:

1. Audit: Periodically scan actual Prometheus metrics and compare against the catalog to find “rogue metrics”
2. Admission: New metrics must be registered in the catalog first, with documented purpose and cardinality assessment
3. Retirement: Metrics marked deprecated get regularly removed from scrape configs

Metric Naming Conventions

A good metric name should be self-explanatory—you should know what it is and how it’s calculated just from the name. Prometheus recommends the format:

<domain>_<subsystem>_<name>_<unit>

The _total suffix is a convention for Counter types (Prometheus automatically appends _total to Counters), while _seconds, _bytes, _ratio are common unit suffixes. These aren’t Prometheus-enforced, but the community follows them, so you should too.

Choosing the Right Metric Type

Prometheus has four metric types. Choosing the wrong one leads to query difficulties and storage waste:

A common mistake: using Gauge to store total request count. Gauges can increase and decrease, but total requests only go up. With Gauge, rate() won’t work correctly, and aggregation gets error-prone.

Collection Strategy: Tiered Management, Collect on Demand

Tiered Scrape Intervals

Not all metrics need 15-second scraping. Blindly setting a uniform 15s interval wastes storage and increases Prometheus load.

# prometheus.yml — Tiered collection config
global:
  scrape_interval: 15s      # Default 15s
  evaluation_interval: 15s   # Alert rule evaluation interval

scrape_configs:
  # Critical business metrics: 15s scrape (payment success rate, core API latency)
  - job_name: 'payment-service'
    scrape_interval: 15s
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['payment-svc:8080']

  # Infrastructure metrics: 60s scrape (CPU, memory, disk)
  - job_name: 'node-exporter'
    scrape_interval: 60s
    static_configs:
      - targets: ['node-1:9100', 'node-2:9100']

  # Debug metrics: 300s or on-demand (GC details, thread stacks)
  - job_name: 'debug-metrics'
    scrape_interval: 300s
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'go_gc_duration_seconds|go_memstats_*'
        action: keep
    static_configs:
      - targets: ['debug-svc:8080']

Tiered collection can reduce data collection volume by 30%-50%. That number might not seem huge, but at millions of metrics, it translates to tens of GB of storage savings and significant memory relief.

Filtering Useless Metrics with metric_relabel_configs

Many Exporters expose hundreds of metrics, but you only use dozens. Use metric_relabel_configs to drop unnecessary metrics at collection time:

scrape_configs:
  - job_name: 'node-exporter'
    scrape_interval: 60s
    metric_relabel_configs:
      # Drop detailed disk IO metrics, keep only summary
      - source_labels: [__name__]
        regex: 'node_disk_.*_bytes_total'
        action: drop

      # Drop network interface details (keep only eth0)
      - source_labels: [__name__, 'device']
        regex: 'node_network_.*;(?!eth0).*'
        action: drop

      # Rename overly long metric names
      - source_labels: [__name__]
        regex: 'node_(.+)'
        target_label: __name__
        replacement: 'node_$1'

action: drop discards data at the collection stage—before it ever touches the TSDB. This is the most effective way to reduce metric count. In contrast, filtering with {__name__!=""} at query time is useless—the data is already stored, still consuming memory and disk.

label_keep and label_drop

Sometimes you don’t want to drop metrics, you want to drop labels. For instance, an Exporter adds an instance_ip label to every metric, with hundreds of values, needlessly inflating cardinality:

metric_relabel_configs:
  # Drop the high-cardinality instance_ip label
  - action: label_drop
    regex: 'instance_ip'

Storage Optimization: Keeping Prometheus Alive Longer

Memory vs. Disk Relationship

Prometheus’s memory usage correlates directly with the number of active time series. Each active time series consumes approximately 1-3 KB of memory (depending on label count and scrape frequency). 500K time series means 500MB - 1.5GB of memory just for series data, plus query cache and indexing, easily eating several GB.

These are rough estimates—actual usage depends on metric types (Histograms consume more than Counters) and scrape frequency. But here’s a rule of thumb: when Prometheus memory usage exceeds 60% of available memory, it’s time to consider sharding.

Data Retention Strategy

# prometheus.yml — Startup parameters
# --storage.tsdb.retention.time=15d    Short-term retention
# --storage.tsdb.retention.size=100GB Disk limit (whichever triggers first)

# Recommended production config: time + disk dual limit
# Startup command:
# prometheus --storage.tsdb.retention.time=15d \
#            --storage.tsdb.retention.size=100GB \
#            --query.max-samples=50000000 \
#            --query.timeout=2m

Note the --storage.tsdb.retention.size parameter. It sets a disk usage limit—when disk usage exceeds this value, Prometheus automatically deletes the oldest data. This is the last line of defense against disk-full-induced Prometheus crashes.

Long-Term Storage: Thanos / VictoriaMetrics

Prometheus local storage is only suitable for short-term data (15-30 days). Long-term storage and cross-instance queries require Thanos or VictoriaMetrics:

My practical recommendation: small-to-medium teams (<500 nodes) should use VictoriaMetrics—it’s hassle-free. Large teams or scenarios requiring multi-tenancy should go with Thanos. Only very large organizations (thousands of nodes+) should consider Cortex/Mimir.

Data Quality Measurement: Quantifying Monitoring Health

Five Core Quality Metrics

Monitoring data governance can’t rely on gut feeling. You need quantifiable metrics to measure “how healthy is your monitoring data.” Here are five core quality metrics:

Detecting High-Cardinality Metrics with PromQL

# Find the top 10 metrics by cardinality
topk(10, count by (__name__)({__name__=~".+"}))

# Find metrics with cardinality over 10,000 (needs external script)
# Prometheus itself isn't great at this kind of aggregate analysis
# Use the Python script below

A Python script to periodically scan for high-cardinality metrics:

#!/usr/bin/env python3
"""Scan Prometheus for high-cardinality metrics"""

import requests
from collections import defaultdict

PROMETHEUS_URL = "http://localhost:9090"
CARDINALITY_THRESHOLD = 10000  # Per-metric series threshold

def get_metric_cardinality():
    """Get the time series count for each metric"""
    # Use the /api/v1/series API to get all time series
    resp = requests.get(
        f"{PROMETHEUS_URL}/api/v1/series",
        params={"match[]": "{__name__=~'.+'}"},
        timeout=30
    )
    data = resp.json()["data"]

    cardinality = defaultdict(int)
    for series in data:
        metric_name = series["__name__"]
        cardinality[metric_name] += 1

    return cardinality

def main():
    cardinality = get_metric_cardinality()

    print("=" * 70)
    print(f"{'Metric Name':<50} {'Series Count':>10}")
    print("=" * 70)

    # Sort by cardinality, output Top 20
    sorted_metrics = sorted(cardinality.items(), key=lambda x: x[1], reverse=True)
    for name, count in sorted_metrics[:20]:
        flag = " ⚠️" if count > CARDINALITY_THRESHOLD else ""
        print(f"{name:<50} {count:>10,}{flag}")

    print("=" * 70)
    total = sum(cardinality.values())
    high_card = sum(1 for c in cardinality.values() if c > CARDINALITY_THRESHOLD)
    print(f"Total series: {total:,}")
    print(f"High-cardinality metrics (> {CARDINALITY_THRESHOLD:,}): {high_card}")

if __name__ == "__main__":
    main()

Run this script once and you’ll know exactly how many time series your Prometheus has and which metrics are exploding. I recommend running it weekly, or wiring it into alerting—automatically notify when total series count exceeds a threshold.

Scrape Failure Monitoring

# Percentage of all scrape targets that are failing
count(up == 0) / count(up) * 100

# Group by scrape job, find the job with most failures
count by (job) (up == 0)

# Targets with longest scrape duration
topk(10, scrape_duration_seconds)

Configure these PromQL expressions as alerts—it’s far more reliable than waiting to discover problems on Grafana:

# alerts.yml — Scrape quality alerts
groups:
  - name: scraping-quality
    rules:
      - alert: ScrapeTargetDown
        expr: up == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Scrape target {{ $labels.instance }} unreachable"
          description: "{{ $labels.instance }} in job {{ $labels.job }} has been unreachable for 5 minutes"

      - alert: ScrapeSlow
        expr: scrape_duration_seconds > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Scrape taking too long: {{ $labels.instance }}"
          description: "Scrape duration {{ $value }}s, exceeding 10s threshold"

      - alert: PrometheusHighCardinality
        # This needs a recording rule for precomputation
        expr: prometheus_tsdb_head_series > 500000
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Prometheus series count too high"
          description: "Current active series count {{ $value }}, exceeding 500K threshold, check for high-cardinality metrics"

Deprecated Metric Retirement: Regular Inventory Cleanup

Metric Deprecation Process

Metrics that only grow and never shrink are a universal monitoring system affliction. Developers instrument a bunch of metrics during feature development, the feature gets deprecated, but the metrics linger—wasting space. You need a regular cleanup process:

1. Discover: Scan for metrics that haven’t been queried in the last 30 days
2. Evaluate: Confirm the metric is truly unused (not just “rarely looked at”)
3. Tag: Mark it as deprecated in the metric catalog
4. Notify: Inform the metric owner, give a 7-day grace period
5. Clean: Remove from scrape config, or drop with metric_relabel_configs

The approach to discovering unqueried metrics: enable Prometheus query logging, record which metric names get queried over a period, then diff against the full metric list.

# prometheus.yml — Enable query logging
# Add --query.log=/var/log/prometheus-query.log to startup params
# Then use a script to analyze the query log and extract queried metric names

#!/usr/bin/env python3
"""Analyze Prometheus query log to find never-queried metrics"""

import json
import re
from collections import defaultdict

def parse_query_log(log_file):
    """Parse query log, extract queried metric names"""
    queried_metrics = set()

    with open(log_file) as f:
        for line in f:
            try:
                entry = json.loads(line)
                query = entry.get("params", {}).get("query", "")
                # Extract metric names from the query
                # Simple regex matching, may need PromQL AST parsing in practice
                metrics = re.findall(r'([a-zA-Z_:][a-zA-Z0-9_:]*)\s*(?:\{|$)', query)
                queried_metrics.update(metrics)
            except json.JSONDecodeError:
                continue

    return queried_metrics

def main():
    queried = parse_query_log("/var/log/prometheus-query.log")

    # Get full metric list via API
    import requests
    resp = requests.get("http://localhost:9090/api/v1/label/__name__/values")
    all_metrics = set(resp.json()["data"])

    # Diff: metrics that exist but were never queried
    unused = all_metrics - queried

    print(f"Total metrics: {len(all_metrics)}")
    print(f"Queried: {len(queried)}")
    print(f"Never queried: {len(unused)}")
    print(f"Deprecation rate: {len(unused)/len(all_metrics)*100:.1f}%")

    if unused:
        print("\nDeprecated metric list (top 50):")
        for m in sorted(unused)[:50]:
            print(f"  {m}")

if __name__ == "__main__":
    main()

Grafana Dashboard Governance

Dashboards are another disaster zone. A team accumulates dozens of dashboards, many created for temporary use and forgotten. Recommendations:

1. Naming convention: Dashboard titles prefixed with team name, e.g., [Platform] API Latency Monitor
2. Tag management: Use Grafana’s tagging feature to mark env=prod, team=platform, status=active
3. Regular review: Clean up quarterly, archive dashboards with no views in 6+ months
4. Dashboard as Code: Manage dashboard definitions with JSON or Terraform, version-controlled

// Dashboard tag example
{
  "title": "[Platform] API Latency Monitor",
  "tags": ["team:platform", "env:prod", "status:active"],
  "folderUid": "platform-dashboards"
}

Production Practice Recommendations

Sharding Strategy

When a single Prometheus instance can’t handle the load, don’t rush to Thanos. Try sharding first:

# Hashmod sharding: distribute scrape targets across multiple Prometheus instances
# prometheus-shard-1.yml
scrape_configs:
  - job_name: 'k8s-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_uid]
        modulus: 3       # Total 3 shards
        target_label: __tmp_hashmod
      - source_labels: [__tmp_hashmod]
        regex: "0"       # This instance only scrapes hashmod=0 targets
        action: keep

This way, 3 Prometheus instances each scrape roughly 1/3 of targets, reducing per-instance load to 1/3 of the original. Federation clusters handle global query aggregation.

A Complete Governance Checklist

• Maintain a metric catalog, recording owner, purpose, cardinality for all metrics
• Check for high-cardinality labels (user_id, request_id, etc.), replace with logs or tracing
• Configure tiered scrape intervals (15s/60s/300s)
• Use metric_relabel_configs to filter useless metrics
• Monitor scrape success rate and scrape duration
• Set time series count alert thresholds
• Regularly scan for unqueried metrics and clean them up
• Dashboard naming standardization and periodic review
• Long-term storage solution deployed (Thanos/VM)
• Establish a metric onboarding process (register → assess → scrape)

Summary

Monitoring data governance is not a one-time project—it’s continuous engineering. The core approach boils down to three things:

Control at the source. Metric definition standards, naming conventions, label strategies—these should all be figured out before metrics enter Prometheus. A single user_id label can blow up an entire monitoring system. I’ve seen this happen more than once.

Tiered management. Not all metrics deserve 15-second scraping. Critical business metrics get high-frequency collection, debug metrics get low-frequency or on-demand, deprecated metrics get deleted outright. Storage and operations costs are real money.

Quantitative measurement. Govern monitoring data with data itself—scrape success rate, series count, unqueried metric ratio. These metrics need alerts. You can’t wait for Prometheus to OOM before discovering metric inflation.

One last piece of experience: the hardest part of monitoring data governance isn’t technology, it’s organizational collaboration. Developers don’t think about cardinality when instrumenting, and ops engineers don’t know which metrics are business-critical. Building a metric catalog and admission process—getting developers and ops aligned—that’s the real key to making governance work.

References & Acknowledgments

The following resources were referenced during the writing of this article. Thanks to the original authors for their contributions:

1. Prometheus Overview — Prometheus official documentation, introduces the data model and core design philosophy
2. Comparison to alternatives | Prometheus — Prometheus official documentation, compares data model differences with Graphite and other monitoring systems
3. Metrics Surge and Query Bottlenecks: Deep Deployment and Tuning of Prometheus/Grafana Monitoring — Detailed analysis of Prometheus performance bottlenecks and sharding strategies at scale
4. How to Efficiently Streamline Prometheus: Full-Chain Strategy from Metric Design to Storage Optimization — Practical approaches for metric naming conventions and metric_relabel_configs filtering
5. Prometheus+Grafana Deep Monitoring: From Metric Collection to Multi-Level Alerting in Production — Analysis of typical monitoring blind spots and collection strategy design
6. Grafana Data Governance Ultimate Guide — Introduces Grafana metadata management and dashboard governance strategies