5000 Alerts Nobody Reads: 6 Architecture Decisions from Fragmented to Unified Observability

Overview

A friend of mine complained last year: his company deployed Prometheus, ELK, and Jaeger — three separate tool stacks maintained by a 7-person ops team. They spent 4 hours a day just triaging alerts. One night at 3 AM, the payment service hit a P0 incident, and it took 42 minutes from alert trigger to root cause identification. The root cause was a database connection pool configuration change causing slow query buildup. Jaeger had the complete trace — nobody looked. ELK had the slow query logs — nobody searched. Prometheus had the connection pool metric anomaly — but the alert was buried among 5,000 daily alerts.

Where’s the problem? Tools are deployed. Data is collected. But three systems operate in silos, without forming “observability.” It’s like installing a speedometer, fuel gauge, and temperature sensor in a car, but putting each gauge in the trunk, glove box, and under the hood respectively — the instruments are all there, but you can’t drive the car.

Observability comes from control theory. The core definition is straightforward: the ability to infer the internal state of a system from its external outputs. In plain terms — without diving into the code, you can answer “Is the system healthy right now? Where’s the problem? Why did it happen?” just by looking at the data.

This article doesn’t cover what the three pillars are (Metrics/Logs/Traces — there are plenty of articles for that). Instead, it covers the architecture decisions you need to make to go from fragmented monitoring to a unified observability system, along with real-world pitfalls for each decision. Every point is validated in production environments, not slide-ware theory.

Monitoring ≠ Observability: Understand What Problem You’re Solving

Many people conflate monitoring with observability. They’re not the same thing.

Monitoring answers “what went wrong” — CPU is maxed, memory is full, API is timing out. It’s preset, based on known failure modes. You define “alert if CPU > 80%” ahead of time, and the system executes accordingly.

Observability answers “why did it go wrong” — why did this API suddenly slow down? Why does this Pod keep restarting? Why did this user’s request fail? It’s exploratory, designed for unknown failure modes. You can’t pre-define every possible failure scenario, but through the correlation of Metrics + Logs + Traces, you can quickly pinpoint the issue after it occurs.

Example: a user reports order placement failure.

• Monitoring only: You see the order service error rate jump from 0.1% to 5%. You stare at Grafana for 10 minutes, guess it might be a database issue, go check DB logs. You find a slow query. But why is there a slow query? No idea. Keep guessing.
• With observability: You see the error rate jump to 5%, click on the Trace link, see the request spending 8 seconds in the DB query Span of the inventory service. Click on this Span’s associated logs, see connection pool wait timeout. Check Metrics, find active connections spiked from 5 to 50 fifteen minutes ago (config change increased the limit, but the database max_connections wasn’t updated). 3 minutes to root cause.

Don’t get lost in the terminology. Remember one thing: monitoring tells you “you’re sick,” observability tells you “where it hurts and why.” If your system is a monolith running on 3 servers, monitoring is sufficient. But if your system is 50 microservices on Kubernetes, without observability, you’re flying blind.

Decision 1: Metrics First or Three Pillars Simultaneously? — Phased Data Collection Strategy

Pitfall Record

I’ve seen teams go for “full observability” from day one: Prometheus + Loki + Jaeger + OpenTelemetry + Grafana, all at once. The result? The ops team’s energy is spread thin across deploying, tuning, and maintaining three systems — each half-baked. Prometheus Recording Rules aren’t written, Loki log formats aren’t unified, Jaeger sampling is set to 100% and storage explodes within a week.

Phased Strategy

The correct approach is layered, phased rollout:

Layer 1: Metrics — the foundation for alerting and trend analysis

Metrics are the lowest-cost, highest-density observability data. A Counter or Gauge takes a few dozen bytes but supports alerting and long-term trend analysis.

Reasons to start with Metrics:

1. Low cost: 10,000 metric series ≈ 500MB/month storage (Prometheus default 15-day retention)
2. Alerting-friendly: PromQL threshold evaluation, sub-second latency
3. Broad coverage: Node Exporter + cAdvisor + custom app metrics cover 80% of daily monitoring needs
4. Quick to start: Prometheus + Grafana can be running in half a day

Layer 2: Logs — the key to root cause analysis

Metrics tell you “which service is slow,” but “why it’s slow” requires logs. The problem with logs is volume — a mid-size microservice cluster generates 50-200GB per day.

Key decisions before deploying log collection:

• Are log formats unified? Structured JSON logs are 10x better than unstructured text
• Are log levels appropriate? Debug logs should be off or sampled in production
• Loki or ELK? Loki is lightweight but limited in query capability; ELK is powerful but resource-hungry

Layer 3: Traces — the silver bullet for microservice debugging

Trace value only materializes in microservice environments. If a request traverses 5+ services, Trace is the only way to see the complete call chain.

But Trace is also the most expensive:

• One request generates 5-15 Spans, each ~500 bytes
• At 1000 QPS, that’s 3 million Spans per minute
• Full collection costs 100x more storage than Metrics

Recommended Rollout Rhythm

Phase 1 (Week 1-2): Prometheus + Grafana + Node Exporter + cAdvisor
                   → Covers basic resource monitoring and app metrics

Phase 2 (Week 3-4): Loki + Promtail + structured logging migration
                   → Covers log search and root cause analysis

Phase 3 (Week 5-6): Tempo/Jaeger + OpenTelemetry SDK
                   → Covers distributed tracing

Phase 4 (Week 7-8): Grafana unified panels + correlation config
                   → Metrics↔Logs↔Traces cross-linking

The benefit of this rhythm: each phase independently produces value. No “3 months to build, still not using it” embarrassment. After Phase 1, alerting works. After Phase 2, logs are searchable. After Phase 3, traces are visible.

Key Lesson

Don’t chase data correlation from day one. Metrics↔Logs↔Traces cross-linking requires a unified Label/Tag system (e.g., service_name + trace_id everywhere), but this unification can’t be designed perfectly upfront. Get things running first, discover correlation needs in practice, then gradually connect them.

I saw a team spend 2 weeks designing a “perfect unified label system,” only to discover after launch that the actual business scenarios needed completely different labels. They spent another week refactoring. Run first, optimize later. Don’t over-design.

Decision 2: Self-Hosted or Managed? — TCO Breakdown of Observability Platforms

Pitfall Record

A company with a 15-person dev team self-hosted the full PLTG stack (Prometheus + Loki + Tempo + Grafana). First 3 months: smooth sailing. 6 months in: problems started. Prometheus couldn’t handle the load (150K samples/sec). Loki queries crawled (30 seconds for searches after hitting 5TB of logs). Tempo storage costs ballooned (2TB/month of Trace data). The 2-person ops team spent half their time maintaining these three systems.

The final tally: self-hosted costs (servers + storage + 2 ops FTEs) were 30% higher than Datadog, with far fewer features.

Cost Breakdown

For a mid-size cluster (50 microservices, 500 Pods, 1000 QPS):

Self-hosted (PLTG Stack: Prometheus + Loki + Tempo + Grafana)

Managed (Datadog as example)

Looks cheaper to go managed? Hold on — there are hidden costs.

Decision Framework

Three dimensions to consider:

1. Team Size

• < 10 devs: Go managed. Self-hosted ops overhead is too high
• 10-30: Self-host Metrics (Prometheus isn’t hard), consider managed for Logs and Traces
• 30-100: Self-host everything — economies of scale, controllable costs
• 100+: Self-host, and consider contributing back upstream

2. Data Sensitivity

• Data can’t leave the datacenter (finance, government, healthcare): Must self-host
• Data can go to cloud but not to third parties: Self-host or open-source managed
• Data isn’t sensitive: Managed is more hassle-free

3. Customization Needs

• Need deep custom alerting logic, custom PromQL functions: Self-host
• Need multi-cluster federation, cross-region aggregation: Self-host (commercial solutions support it but expensively)
• Standard monitoring needs: Managed is sufficient

Hybrid Approach (Recommended)

The most cost-effective approach in practice is hybrid:

# Metrics: Self-hosted Prometheus (low cost, strong control)
# - Infrastructure metrics: Node Exporter + cAdvisor
# - App metrics: OpenTelemetry SDK → Prometheus
# - Long-term storage: Thanos or VictoriaMetrics (10x cheaper than Datadog)

# Logs: Self-hosted Loki (lightweight, sufficient)
# - Collection: Promtail DaemonSet
# - Storage: Local SSD 7 days + S3 30 days
# - Query: Grafana LogQL

# Traces: Self-hosted Tempo or Grafana Cloud (small volume can be managed)
# - Collection: OpenTelemetry Collector
# - Sampling: Dynamic, 1-5% in production
# - Storage: S3 + local cache

This setup costs about ¥8,000-12,000/month — 40-50% of pure managed, covering 90% of use cases. The trade-off is needing 1 ops person spending 20% of their time on maintenance.

Decision 3: High-Cardinality Labels Are Poison — Metric Cardinality Governance

Pitfall Record

This is the most common Prometheus failure mode I’ve seen, bar none.

A team added a user_id label to their HTTP request metrics:

# WRONG: one time series per user
http_requests_total{
  method="GET",
  path="/api/order",
  user_id="12345"  # ← This is a high-cardinality label
} 1

100K users × 5 endpoints × 2 methods = 1 million time series. Prometheus memory went from 2GB to 32GB, queries timed out, alerting lagged 5 minutes behind. They had to restart Prometheus to clear data.

This trap is incredibly common. High-cardinality labels are Prometheus’s “memory bomb.”

What Is Cardinality and Why It Matters

Cardinality refers to the number of time series for a metric. Prometheus’s storage model: each time series = the Cartesian product of metric name + label combinations.

# Cardinality = product of label value combinations
http_requests_total{method="GET", path="/api/order", status="200"}
# If method has 5 values, path has 20, status has 10
# Cardinality = 5 × 20 × 10 = 1000 time series

Rule of thumb: a label with cardinality over 1,000 is suspicious; over 10,000 is almost certainly a bug.

Governance Measures

1. relabel_config to Drop Dangerous Labels

Kill high-cardinality labels at the Prometheus scrape level, before they enter storage:

scrape_configs:
  - job_name: 'app'
    static_configs:
      - targets: ['app:8080']
    metric_relabel_configs:
      # Drop user_id label
      - action: labeldrop
        regex: 'user_id|request_id|session_id|trace_id'
      # Normalize high-cardinality paths to low-cardinality groups
      - source_labels: [path]
        regex: '/api/users/(\d+)'
        target_label: path
        replacement: '/api/users/:id'
      - source_labels: [path]
        regex: '/api/orders/([a-f0-9-]+)'
        target_label: path
        replacement: '/api/orders/:id'

This config does two things: drops user_id and similar high-cardinality labels entirely, and normalizes /api/users/12345 to /api/users/:id, reducing 100K path variations to 50.

2. Recording Rules for Pre-aggregation

For high-frequency, compute-intensive queries, use Recording Rules to pre-aggregate:

groups:
  - name: preaggregation
    rules:
      # Aggregate request rate by job, reducing query computation
      - record: job:http_requests:rate5m
        expr: sum by (job, status) (rate(http_requests_total[5m]))
      
      # Aggregate P99 latency by job
      - record: job:http_request_duration_seconds:p99
        expr: histogram_quantile(0.99, sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))
      
      # Error rate
      - record: job:http_error_rate:ratio
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))          

After pre-aggregation, a query that previously scanned 1 million series now scans 50. Query speed improves 1000x.

3. Monitor Cardinality via API

Prometheus provides /api/v1/status/tsdb to inspect high-cardinality metrics:

# Top 10 metrics by series count
curl -s http://prometheus:9090/api/v1/status/tsdb | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
series = data['data']['seriesCountByMetricName']
for name, count in sorted(series.items(), key=lambda x: -x[1])[:10]:
    print(f'{count:>10} {name}')
"

Add this to your monitoring dashboard. Catch cardinality explosions before they take down Prometheus.

Cost Comparison

Resource consumption before vs. after cardinality governance (real production data):

Memory down 89%, queries 15x faster, storage growth down 91%. Cardinality governance has the highest ROI of all observability optimizations. Bar none.

Decision 4: How Long Should Logs Be Stored? — Tiered Storage and Sampling Strategy

Pitfall Record

An e-commerce team: 500K daily orders, 12 microservices, 180GB of logs per day. They used an ELK cluster with 3 nodes and 30TB SSD, thinking it was enough. 3 months later: disk full. Scale up? No budget. Delete old logs? Compliance requires 90-day retention. They ended up implementing hot/warm/cold tiering — storage costs dropped 60%, and query performance actually improved.

Log Tiered Storage Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Log Data Flow                              │
│                                                               │
│  App logs → Promtail/Fluent Bit → Loki/ELK                    │
│                                      │                        │
│                    ┌─────────────────┼───────────────┐        │
│                    │                 │               │        │
│                ┌───▼───┐     ┌──────▼──────┐  ┌────▼────┐    │
│                │ Hot   │     │   Warm      │  │ Cold    │    │
│                │       │     │             │  │         │    │
│                │ 3 days│     │  7-30 days  │  │ 90 days+│    │
│                │ SSD   │     │  HDD/compr. │  │ S3/OSS  │    │
│                │ Fast  │     │  Slow       │  │ On-demand│    │
│                └───────┘     └─────────────┘  └─────────┘    │
│                                                               │
│  Query priority: Hot → Warm → Cold (degrades layer by layer)  │
└─────────────────────────────────────────────────────────────┘

Loki implementation:

# Loki config: tiered storage
storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/cache
  
  # Hot data: local SSD
  filesystem:
    directory: /loki/data
  
  # Cold data: S3
  aws:
    s3: s3://observability-logs/cold/
    endpoint: s3.cn-north-1.amazonaws.com

# Retention policy
limits_config:
  retention_period: 90d  # Total retention 90 days
  max_query_series: 1000  # Limit query complexity

compactor:
  working_directory: /loki/compactor
  shared_store: s3
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 50

Log Cost Reduction Techniques

1. Structured Logging: The One-Time Investment That Pays Forever

Unstructured logs (plain text) require regex matching in Loki — slow and expensive. Structured JSON logs can be filtered using LogQL labels directly, 10x more efficient:

# Bad: unstructured log
logging.info(f"User 12345 placed order 67890, amount=99.50, method=alipay")
# Query: {app="order"} |= "order" |= "12345"  → full-text scan

# Good: structured log
logging.info({
    "event": "order_placed",
    "user_id": "12345",  # user_id in log content is fine, not a Prometheus label
    "order_id": "67890",
    "amount": 99.50,
    "payment_method": "alipay"
})
# Query: {app="order"} | json | event="order_placed"  → label filter

2. Log Level Sampling

Production Info-level logs account for 80% of volume, but 80% of Info logs are useless when troubleshooting. Sampling strategy:

# Promtail pipeline config: log level sampling
pipeline_stages:
  - json:
      expressions:
        level: level
        message: msg
  
  # Drop DEBUG logs
  - drop:
      source: level
      expression: "DEBUG"
  
  # Sample INFO logs (keep 10%)
  - sampling:
      rate: 0.1
      source: level
      expression: "INFO"
  
  # WARN and ERROR: keep all
  # (not in sampling rules = default keep all)

Measured effect: 180GB/day dropped to 45GB/day, WARN and ERROR logs with 0 loss.

3. Log Field Pruning

Many logs contain redundant fields. Use Promtail pipeline or Fluent Bit Filter to prune:

# Keep only essential fields
pipeline_stages:
  - json:
      expressions:
        timestamp: timestamp
        level: level
        message: msg
        service: service_name
        trace_id: trace_id
        error: error
  # All other fields are dropped

A single log entry goes from 2KB to 200 bytes — 10x compression.

Storage Cost Comparison

The sampling + tiering approach costs only 10% of all-hot storage, and because there’s less hot data, queries are faster. Cost reduction and performance improvement are aligned here.

Decision 5: What Should the Trace Sampling Rate Be? — The Full-Collection Cost Trap

Pitfall Record

A team used OpenTelemetry SDK for trace collection with sampling set to 100% (full collection) — “collect everything for now, adjust later.” One week later, Tempo storage went from 0 to 800GB, adding ¥4,000/month to the S3 bill. More awkwardly, with so much data, querying the Trace list took 20 seconds — worse than not collecting at all.

Full trace collection in production is almost never necessary. You don’t need to see every request’s trace — you need to see the traces of requests that had problems.

Two Sampling Strategies

Head-based Sampling

The sampling decision is made at the request entry point. OpenTelemetry SDK has built-in support:

// OpenTelemetry SDK config: 1% sampling rate
SdkTracerProvider.builder()
    .setSampler(Sampler.traceIdRatioBased(0.01))  // 1% sampling
    .build();

Pros: Simple, low overhead, no additional components needed.

Cons: Random sampling — the errored request might be sampled out. At 1% sampling rate, 99% of error traces are lost.

Tail-based Sampling

The sampling decision is made after the request completes, based on its outcome. Implemented through OpenTelemetry Collector:

# OpenTelemetry Collector config: tail-based sampling
processors:
  tail_sampling:
    decision_wait: 30s  # Wait 30s to collect the complete Trace
    num_traces: 50000   # Traces held in memory
    
    policies:
      # Sample all error requests
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      
      # Sample slow requests (>2s)
      - name: slow
        type: latency
        latency:
          threshold_ms: 2000
      
      # Sample 1% of normal requests
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 1

This config ensures: 100% of error traces retained, 100% of slow traces retained, only 1% of normal traces. Storage drops 95%+, but critical debugging information is fully preserved.

Dynamic Sampling

A more advanced approach: dynamically adjust sampling rate based on system state. 1% during normal operation, auto-increase to 100% when error rate spikes:

processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 50000
    
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      
      - name: dynamic_rate
        type: probabilistic
        probabilistic:
          # Inject sampling rate dynamically via environment variable
          # Normal: 1%, alerting: 100% via API adjustment
          sampling_percentage: ${env:TRACE_SAMPLING_RATE}

Paired with a simple script that auto-increases sampling rate when alerts fire:

import os
import requests

def adjust_sampling_rate():
    """Dynamically adjust trace sampling rate based on error rate"""
    # Query current 5-minute error rate
    query = 'sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))'
    resp = requests.get(
        f'http://prometheus:9090/api/v1/query',
        params={'query': query}
    ).json()
    
    error_rate = float(resp['data']['result'][0]['value'][1])
    
    if error_rate > 0.05:  # Error rate > 5%
        rate = 100
    elif error_rate > 0.01:  # Error rate > 1%
        rate = 50
    else:
        rate = 1  # Normal: 1%
    
    # Update Collector config (via API or environment variable)
    os.environ['TRACE_SAMPLING_RATE'] = str(rate)
    print(f"Error rate: {error_rate:.2%}, sampling rate: {rate}%")

if __name__ == '__main__':
    adjust_sampling_rate()

Cost Comparison

Tail-based sampling is the best solution for most scenarios. Dynamic sampling is smarter but more complex to implement — suitable for teams with higher observability requirements.

Decision 6: Who Are Dashboards For? — Consumer Layering of Observability Data

Pitfall Record

I saw a team’s Grafana setup: 56 dashboards, each with 20-40 panels. The largest “global overview” dashboard had 87 panels and took 40 seconds to load. On-call SREs couldn’t process it, developers found it too complex, management couldn’t understand it. Eventually, nobody looked. Dashboards became shelf-ware.

More dashboards ≠ better. The right person seeing the right information is what matters.

Three-Tier Dashboard Architecture

┌──────────────────────────────────────────────────────────┐
│                   Dashboard Layering Architecture          │
│                                                            │
│  ┌─────────────────┐                                       │
│  │  L1: Executive   │  Audience: On-call SRE               │
│  │  (Real-time)     │  Focus: Any issues right now?        │
│  │  Refresh: 10-30s │  Panels: 5-8                        │
│  │  Data: Live mets │  Layout: Single screen, no scroll    │
│  └────────┬────────┘                                       │
│           │                                                │
│  ┌────────▼────────┐                                       │
│  │  L2: Diagnostic  │  Audience: Devs + SRE               │
│  │  (Service debug) │  Focus: Which service? Why?         │
│  │  Refresh: 1-5m   │  Panels: 15-20                      │
│  │  Data: Mets+Logs │  Layout: Drill-down, global→service  │
│  └────────┬────────┘                                       │
│           │                                                │
│  ┌────────▼────────┐                                       │
│  │  L3: Strategic  │  Audience: Tech mgmt + Product       │
│  │  (SLO dashboard)│  Focus: Are we meeting SLOs?         │
│  │  Refresh: 1h-1d │  Panels: 8-12                        │
│  │  Data: SLO mets │  Layout: Business perspective        │
│  └─────────────────┘                                       │
└──────────────────────────────────────────────────────────┘

L1: Executive Dashboard (On-call SRE)

Design principle: See system health at a glance. No more than 8 panels, single screen, no scrolling.

// L1 Dashboard key panels
// 1. Global health status (green/red)
// 2. Core service error rate (P99 + SLO line)
// 3. Core service latency (P50/P99)
// 4. Alert count (grouped by severity)
// 5. K8s Pod status (Running/Pending/Failed)
// 6. Database connection pool utilization
// 7. Last 1-hour change events
// 8. On-call rotation info

Core PromQL:

// Global health: core service SLO achievement rate
sum by (service) (
  rate(http_requests_total{status!~"5..", service=~"order|payment|user"}[5m])
)
/
sum by (service) (
  rate(http_requests_total{service=~"order|payment|user"}[5m])
)

// Alert count
count by (severity) (
  ALERTS{alertstate="firing"}
)

L2: Diagnostic Dashboard (Devs + SRE)

Design principle: Drill-down troubleshooting from global to single service. Grouped by service, each group’s panel structure is consistent.

Key panels:

• Service-level Request Rate / Error Rate / Latency (RED metrics)
• Service-level resource usage (CPU / Memory / connection pool)
• Log panel: latest 100 ERROR logs
• Trace panel: links to recent slow request traces

The value of L2 panels is in correlation. Grafana supports cross-panel data linking:

# Grafana Dashboard Links config
links:
  # Jump from metrics panel to logs
  - title: "View logs"
    type: link
    url: /explore?orgId=1&left={"datasource":"Loki","queries":[{"expr":"{service=\"$service\"} |= \"ERROR\""}]}
    
  # Jump from metrics panel to traces
  - title: "View traces"
    type: link
    url: /explore?orgId=1&left={"datasource":"Tempo","queries":[{"service_name":"$service","operation":"$operation"}]}

Once configured, the on-call SRE sees a service’s error rate spike on the metrics panel, clicks through to the corresponding logs, then clicks again to see the Trace. This is what “observability” feels like in practice — data is interconnected, no manual switching between tools.

L3: Strategic Dashboard (Tech Management + Product)

Design principle: Tell the technical story in business language. No CPU or memory metrics — show SLO achievement rate, error budget consumption, incident count.

// SLO achievement rate (30-day sliding window)
sum(rate(http_requests_total{status!~"5..", service="order"}[30d]))
/
sum(rate(http_requests_total{service="order"}[30d]))

// Error budget consumption progress
1 - (
  sum(rate(http_requests_total{status!~"5..", service="order"}[30d]))
  /
  sum(rate(http_requests_total{service="order"}[30d]))
)
/
(1 - 0.999)  // SLO target 99.9%

The error budget panel is the most valuable L3 panel. It translates “system reliability” into a language that product and management can understand: “You have 30% of your error budget left this month. If the current error rate continues, you’ll exceed budget next week and need to freeze releases.”

Dashboard Governance Lessons

1. Quarterly dashboard cleanup: Delete panels nobody has viewed in 3 months. I saw a team go from 56 dashboards down to 12 — debugging efficiency actually improved because finding the right dashboard was faster.
2. Unified variable naming: All dashboards use consistent $service, $env, $time_range variables with consistent defaults — reduces cognitive load.
3. Dashboard as Code: Manage dashboards via Grafana Terraform Provider or Jsonnet, under Git version control. Don’t use the UI to click — it’s not reproducible, not auditable, not rollbackable.

Implementation Roadmap: 8-Week Plan from Fragmented to Unified

If your current situation is “tools are deployed but don’t form a system,” here’s a validated 8-week transformation plan:

After 8 weeks, you should see: alert volume down 70%+, P0 root cause localization time from 30+ minutes to under 10 minutes, storage costs down 50%+.

Summary

Observability isn’t just buying three tools and putting them together. It’s a complete engineering system from data collection to data consumption, with architecture decisions at every step.

Recap of the 6 core decisions:

1. Phased collection: Metrics → Logs → Traces, phase by phase — alert first, debug second
2. Self-hosted vs managed: Depends on team size and data sensitivity — hybrid approach is most cost-effective
3. Cardinality governance: High-cardinality labels are Prometheus memory bombs — relabel + Recording Rules are the antidote
4. Log tiering: Hot/Warm/Cold 3-tier + sampling, storage cost down 90%
5. Trace sampling: Tail-based beats head-based — errors and slow requests 100% preserved
6. Dashboard layering: L1 alerting, L2 debugging, L3 decision-making — don’t show SLO panels to SREs or CPU curves to management

One core lesson: the ultimate goal of observability isn’t comprehensive data, it’s fast root cause localization when things break. 5,000 alerts are worse than 50 precise alerts. 120GB of logs is worse than 45GB of structured logs. 100% sampling is worse than 5% tail-based sampling. Less is more. Precision beats coverage.

One honest note: the parameters and thresholds for each decision above need to be adjusted for your actual environment. The numbers in this article are from my production environment — your traffic patterns, service scale, and budget constraints are different. But the decision framework is universal: first figure out “who consumes the data and what problem it solves,” then decide “what to collect, how long to store, how to query.”

References & Acknowledgments

This article references the following materials. Thanks to the original authors for their contributions:

1. Architecture strategies for designing a monitoring system — Microsoft Azure Well-Architected Framework, official reference for observability architecture design strategies
2. Prometheus Agent Mode: Efficient and Cloud-Native Way for Metric Forwarding — Bartek Płotka (Prometheus Maintainer, CNCF TAG Observability tech lead), Prometheus Agent mode and observability paradigm evolution
3. Remote Write Tuning — Prometheus official documentation, Remote Write tuning and memory management practices
4. OpenMetrics 2.0 Specification — Prometheus community/CNCF, metric format standard and OpenTelemetry compatibility design
5. Observability Three Pillars: Metrics, Logs, Traces Comprehensive Guide — CSDN, three-pillar relationships and best practices for collaborative work
6. Building Observability Platform with Prometheus + Loki + Tempo — Tencent Cloud Developer Community, enterprise-grade observability platform deployment in practice
7. Observability: Logs, Metrics, and Traces — Springer, academic survey of observability three pillars and OpenTelemetry practices