Prometheus High Availability and Federation

Overview

Prometheus defaults to a single-node architecture, which is a dangerous liability in production environments. A single Prometheus instance going down means your entire monitoring system goes blind — you can’t see any metrics during the failure, and alerts stop working because rules are no longer evaluated. As data volume grows to the limits of a single machine’s storage and processing capacity, you’ll face write timeouts, slow queries, and disk exhaustion.

This article systematically analyzes Prometheus high-availability and horizontal scaling solutions, from the simplest dual-replica approach to remote storage solutions like Thanos, Mimir, VictoriaMetrics, and Cortex, helping you make the right technical choice based on your actual scenario.

Reference: Prometheus Official Documentation — HA, Thanos Official Documentation

I. Single-Point Problem Analysis

1.1 Risks of Single-Point Prometheus

┌──────────────┐     ┌───────────────┐     ┌──────────┐
│  Exporters   │ ──→ │  Prometheus   │ ──→ │  Grafana │
│  (targets)   │     │  (single node) │     │  Alertmgr│
└──────────────┘     └───────────────┘     └──────────┘
                            │
                     ┌──────┴──────┐
                     │  Local TSDB  │
                     │  (15-30 days)│
                     └─────────────┘

Vulnerabilities of this architecture:

Risk Point	Impact	Probability
Prometheus process crash	Complete monitoring outage, alerts fail	Medium
Host machine down	Same as above, may lose recent data	Medium
Disk failure	Historical data loss	Low-Medium
Disk space exhaustion	Write failures, data loss	Medium-High
Insufficient single-machine memory/CPU	Write delays, query timeouts	High (as data grows)
Network partition	Some targets can’t be scraped	Medium

1.2 Single-Machine Performance Bottlenecks

Prometheus performance is constrained by:

Time series count: ~1-2 GB memory per million active time series
Ingestion rate: Recommended max of 1 million samples/sec per instance
Query complexity: Large-range PromQL queries can cause OOM
Local storage: TSDB defaults to 15-day retention; long-term retention requires significant disk space

# Check Prometheus active time series count
curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName | length'

# Check ingestion rate
curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.headStats'

When a single instance’s active time series exceed 2 million or the ingestion rate exceeds 800K/sec, it’s time to consider horizontal scaling.

II. Dual-Replica Solution

2.1 Dual-Replica Architecture

The most straightforward HA approach is deploying two identical Prometheus instances scraping the same targets:

┌──────────────┐
│  Exporters   │
└──────┬───────┘
       │
   ┌───┴───┐
   ▼       ▼
┌─────────┐ ┌─────────┐
│  Prom-A │ │  Prom-B │
│ (replica1) │ │ (replica2) │
└────┬────┘ └────┬────┘
     │           │
     ▼           ▼
┌──────────────────────┐
│   Alertmanager HA    │
│   (Gossip dedup)     │
└──────────────────────┘

Both Prometheus instances independently scrape and evaluate alerting rules, both sending alerts to Alertmanager. Alertmanager deduplicates across instances via the Gossip protocol, ensuring each alert notification is sent only once.

2.2 Alertmanager HA Configuration

# Alertmanager cluster configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - 'alertmanager-1:9093'
            - 'alertmanager-2:9093'
            - 'alertmanager-3:9093'

Alertmanager instances form a cluster using the --cluster.peer flag on startup:

# alertmanager-1
alertmanager \
  --config.file=/etc/alertmanager/config.yml \
  --storage.path=/data/alertmanager \
  --cluster.listen-address=0.0.0.0:9094 \
  --cluster.peer=alertmanager-2:9094 \
  --cluster.peer=alertmanager-3:9094

# alertmanager-2
alertmanager \
  --config.file=/etc/alertmanager/config.yml \
  --storage.path=/data/alertmanager \
  --cluster.listen-address=0.0.0.0:9094 \
  --cluster.peer=alertmanager-1:9094 \
  --cluster.peer=alertmanager-3:9094

2.3 Limitations of Dual Replicas

Dual replicas solve the single-point-of-failure problem but have these limitations:

Doubled storage: Both instances store complete data, doubling storage costs
Query deduplication needed externally: Grafana queries need to specify a datasource; the two instances have minor data differences (scrape timestamps aren’t perfectly aligned)
No horizontal scaling: The single instance’s scrape and processing capacity ceiling is unchanged
No long-term storage: Still relies on local TSDB, can’t retain more than 30 days of historical data

The dual-replica approach is suitable for small-to-medium scale monitoring needs (active time series < 1 million) and is the lowest-cost HA solution.

III. Remote Storage Solutions Overview

When single-machine performance or storage capacity becomes a bottleneck, remote storage is needed. Prometheus natively supports remote_write and remote_read interfaces, enabling streaming data to external storage systems.

┌─────────────────────────────────────────────────────┐
│              Each cluster's Prometheus               │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐              │
│  │  Prom-1 │  │  Prom-2 │  │  Prom-3 │              │
│  └────┬────┘  └────┬────┘  └────┬────┘              │
│       │            │            │                     │
│       └────────────┼────────────┘                     │
│                    │ remote_write                     │
└────────────────────┼─────────────────────────────────┘
                     │
                     ▼
          ┌──────────────────────┐
          │  Remote Storage      │
          │  (Thanos/Mimir/VM)   │
          └──────────┬───────────┘
                     │
          ┌──────────┴───────────┐
          │  Global Query /       │
          │  Long-term Storage    │
          │  (PromQL compatible)  │
          └──────────────────────┘

Mainstream remote storage solutions compared:

Solution	Developer	Core Features	Object Storage	Multi-Tenant	Maturity
Thanos	Improbable	Sidecar + object storage + global query	Yes	No	High
Mimir	Grafana Labs	Cortex rewrite, strong horizontal scaling	Yes	Yes	High
VictoriaMetrics	VictoriaMetrics	Proprietary storage engine, extreme performance	Yes	Yes (Enterprise)	High
Cortex	Grafana Labs	Mimir predecessor, multi-tenant	Yes	Yes	Maintenance mode
InfluxDB	InfluxData	Time-series DB, not Prom ecosystem	No	Yes	Medium

IV. Thanos: The Most Popular Remote Storage Solution

4.1 Thanos Architecture

┌──────────────────────────────────────────────────────────┐
│                     Thanos Architecture                    │
│                                                           │
│  ┌─────────┐    ┌──────────────┐    ┌──────────────┐    │
│  │ Prom +  │    │  Thanos      │    │  Thanos      │    │
│  │ Sidecar │    │  Store       │    │  Compactor   │    │
│  │ (upload)│    │  (read hist.)│    │  (downsample)│    │
│  └────┬────┘    └──────┬───────┘    └──────┬───────┘    │
│       │                │                   │            │
│       ▼                │                   ▼            │
│  ┌─────────────────────┴───────────────────┐            │
│  │       Object Storage (S3/GCS/MinIO)      │            │
│  └─────────────────────┬───────────────────┘            │
│                        │                                │
│  ┌─────────────────────┴───────────────┐                │
│  │           Thanos Query              │                │
│  │     (global PromQL query entry)     │                │
│  └─────────────────────┬───────────────┘                │
│                        │                                │
│  ┌─────────────────────┴───────────────┐                │
│  │      Thanos Query Frontend          │                │
│  │     (query cache / sharding /       │                │
│  │      rate limiting)                 │                │
│  └─────────────────────────────────────┘                │
└──────────────────────────────────────────────────────────┘

Core Thanos components:

Component	Role
Sidecar	Deployed alongside Prometheus, uploads TSDB blocks to object storage
Store	Reads historical data from object storage, responds to Query requests
Compactor	Downsamples and compacts data in object storage
Query	Receives PromQL queries, fetches results from multiple sources (Sidecar/Store/Ruler)
Query Frontend	Query caching, sharding, rate limiting
Ruler	Evaluates alerting rules independently (without a Prometheus instance)
Receiver	Receives remote_write data (optional, supports HA write path)

4.2 Sidecar Mode Deployment

# Prometheus + Thanos Sidecar (K8s Deployment)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:v2.52.0
          args:
            - '--config.file=/etc/prometheus/prometheus.yml'
            - '--storage.tsdb.path=/prometheus'
            - '--storage.tsdb.retention.time=24h'  # Only retain 24h locally; historical data goes to object storage
            - '--storage.tsdb.min-block-duration=2h'
            - '--storage.tsdb.max-block-duration=2h'
            - '--web.enable-lifecycle'
          ports:
            - containerPort: 9090

        - name: thanos-sidecar
          image: thanosio/thanos:v0.35.0
          args:
            - sidecar
            - --tsdb.path=/prometheus
            - --prometheus.url=http://localhost:9090
            - --objstore.config-file=/etc/thanos/objstore.yml
            - --shipper.upload-compacted
          volumeMounts:
            - name: prometheus-data
              mountPath: /prometheus

Object storage configuration (S3 example):

# objstore.yml
type: S3
config:
  bucket: thanos-storage
  endpoint: s3.us-east-1.amazonaws.com
  region: us-east-1
  access_key: AKIAIOSFODNN7EXAMPLE
  secret_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
  insecure: false

4.3 Global Query

Thanos Query integrates with Grafana to provide cross-cluster global query capabilities:

# Thanos Query deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-query
spec:
  template:
    spec:
      containers:
        - name: query
          image: thanosio/thanos:v0.35.0
          args:
            - query
            - --http-address=0.0.0.0:9090
            - --grpc-address=0.0.0.0:10901
            - --store=thanos-sidecar-cluster-a:10901  # Cluster A real-time data
            - --store=thanos-sidecar-cluster-b:10901  # Cluster B real-time data
            - --store=thanos-store:10901              # Historical data from object storage
            - --query.replica-label=replica           # Deduplication label
          ports:
            - containerPort: 9090

With --query.replica-label=replica, Thanos Query automatically deduplicates data from dual-replica Prometheus instances, returning consistent results.

4.4 Thanos Advantages and Limitations

Advantages:

Non-intrusive integration, minimal Prometheus config changes
Global query, cross-cluster unified view
Low object storage cost (~$0.023/GB/month on S3)
Supports downsampling (5m → 1h → decreasing precision), good long-range query performance
Active community, many production use cases

Limitations:

Sidecar mode depends on Prometheus local TSDB’s 2-hour block mechanism; data upload is delayed
No native horizontal write scaling (requires Receiver + ingestion blocks)
Multiple components, moderate operational complexity
No multi-tenant support

V. Mimir: Grafana Labs’ Next-Generation Solution

5.1 Mimir Architecture

Mimir is Grafana Labs’ rewrite of Cortex, using a microservices architecture with native horizontal scaling and multi-tenancy.

┌──────────────────────────────────────────────────┐
│                    Mimir Architecture              │
│                                                   │
│  Prom ──remote_write──→ ┌──────────────┐         │
│                          │  Distributor │         │
│                          └──────┬───────┘         │
│                                 │                  │
│                          ┌──────┴───────┐         │
│                          │   Ingester   │         │
│                          │ (memory write)│        │
│                          └──────┬───────┘         │
│                                 │                  │
│                          ┌──────┴───────┐         │
│                          │  Store-gw    │         │
│                          │ (historical) │         │
│                          └──────┬───────┘         │
│                                 │                  │
│                          ┌──────┴───────┐         │
│                          │   Querier    │         │
│                          │ (PromQL query)│        │
│                          └──────────────┘         │
│                                                   │
│  Object Storage (S3) ←── blocks / index / meta ──→│
│                                                   │
│  Consul/Memberlist ←── service discovery ──→      │
└──────────────────────────────────────────────────┘

5.2 Prometheus remote_write Configuration

# prometheus.yml
remote_write:
  - url: http://mimir-distributor:8080/api/v1/push
    headers:
      X-Scope-OrgID: tenant-1  # Multi-tenant ID
    write_relabel_configs:
      - source_labels: [__name__]
        regex: 'go_(gc|memstats)_.+'
        action: drop  # Drop unneeded metrics to reduce write volume
    queue_config:
      capacity: 10000
      max_samples_per_send: 2000
      batch_send_deadline: 5s
      min_backoff: 30ms
      max_backoff: 100ms

5.3 Mimir vs Thanos

Dimension	Thanos	Mimir
Write path	Sidecar passive upload (2h block)	remote_write active push (real-time)
Write latency	2 hours (block completion)	Seconds
Horizontal scaling	Limited (Receiver can scale but has bottlenecks)	Native horizontal scaling
Multi-tenancy	Not supported	Native support
Query performance	Medium	High (query caching and sharding)
Component complexity	Medium	High
Object storage	S3/GCS/Azure/MinIO	S3/GCS/Azure/MinIO
Alert evaluation	Ruler component	Ruler component
Compaction/Downsampling	Compactor	Compactor
Community activity	High	High

Selection guidance: For single or few clusters, prefer Thanos — simple and sufficient. Choose Mimir for multi-tenancy, large scale (active time series > 10 million), or when millisecond-level write latency is needed.

VI. VictoriaMetrics: High-Performance Proprietary Storage

6.1 Architecture Characteristics

VictoriaMetrics (VM) uses a proprietary columnar storage engine, significantly outperforming TSDB-based solutions in both write throughput and query performance.

┌──────────────────────────────────────────────────────┐
│              VictoriaMetrics Architecture             │
│                                                      │
│  Prom ──remote_write──→ ┌──────────────────┐         │
│                          │  vminsert         │         │
│                          │  (write entry, LB)│         │
│                          └────────┬──────────┘         │
│                                   │                    │
│                          ┌────────┴──────────┐        │
│                          │  vmstorage (×N)    │        │
│                          │  (data sharding)   │        │
│                          └────────┬──────────┘        │
│                                   │                    │
│                          ┌────────┴──────────┐        │
│                          │  vmselect         │        │
│                          │  (query, merge)   │        │
│                          └───────────────────┘        │
└──────────────────────────────────────────────────────┘

6.2 Cluster Mode Deployment

# Single-node mode (quick start)
./victoria-metrics \
  -storageDataPath=/data \
  -retentionPeriod=12 \
  -httpListenAddr=:8428

# Cluster mode (production)
# vmstorage nodes (data storage, 3 nodes)
./vmstorage \
  -storageDataPath=/data \
  -retentionPeriod=12 \
  -httpListenAddr=:8482

# vminsert nodes (write entry)
./vminsert \
  -httpListenAddr=:8480 \
  -storageNode=vmstorage-1:8400,vmstorage-2:8400,vmstorage-3:8400

# vmselect nodes (query entry)
./vmselect \
  -httpListenAddr=:8481 \
  -storageNode=vmstorage-1:8400,vmstorage-2:8400,vmstorage-3:8400

6.3 VictoriaMetrics Advantages

Advantage	Description
Extreme performance	Columnar compression + memory-mapped files, 3-5x higher write throughput than Prometheus
Low storage cost	5-10x higher compression ratio than Prometheus
PromQL + MetricsQL	PromQL-compatible, with additional MetricsQL extension functions
Native HA	Cluster mode natively supports replicas and sharding
Single binary deployment	Single-node mode is a single binary, extremely simple to deploy
Built-in downsampling	Automatically downsamples data across different time ranges
Low cost	Open-source edition is free; enterprise edition supports multi-tenancy and alerting

6.4 vmagent: Lightweight Scraping Agent

VM also provides vmagent, which can replace Prometheus for scraping, supporting more service discovery methods and remote writes:

./vmagent \
  -promscrape.config=/etc/vmagent/prometheus.yml \
  -remoteWrite.url=http://vminsert:8480/insert/0/prometheus/api/v1/write \
  -remoteWrite.url=http://backup-vminsert:8480/insert/0/prometheus/api/v1/write \
  -remoteWrite.multitenantURL=http://vminsert:8480/insert/multitenant/prometheus/api/v1/write

vmagent supports writing to multiple remote storages simultaneously (multi-active writes) with automatic retry on failure, making it a lightweight replacement for the Prometheus scraping layer.

VII. Federation

7.1 Federation Architecture

Federation is Prometheus’s native HA and aggregation query solution, pulling metrics from one Prometheus to another via the /federate endpoint.

┌─── Region A ──────────┐    ┌─── Region B ──────────┐
│  Prometheus-A         │    │  Prometheus-B         │
│  (scrape Region A     │    │  (scrape Region B     │
│   targets)            │    │   targets)            │
└────────┬──────────────┘    └────────┬──────────────┘
         │                          │
         │   scrape /federate       │
         │ ←────────────────────────│
         ▼                          ▼
┌──────────────────────────────────────────────┐
│        Global Prometheus (Federation)        │
│        (aggregated query, cross-region view) │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
              ┌──────────┐
              │  Grafana  │
              └──────────┘

7.2 Federation Configuration

# Global Prometheus
scrape_configs:
  - job_name: 'federate'
    scrape_interval: 30s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="node"}'
        - '{job="mysql"}'
        - '{__name__=~"up|prometheus_.*|alertmanager_.*"}'
        - '{alertname!=""}'  # Alert metrics
    static_configs:
      - targets:
          - 'prometheus-region-a:9090'
          - 'prometheus-region-b:9090'
    relabel_configs:
      - source_labels: [__address__]
        regex: 'prometheus-(.+):9090'
        target_label: source_prometheus
        replacement: '${1}'

The match[] parameter controls which metrics are federated. honor_labels: true ensures original labels aren’t overwritten.

7.3 Federation Limitations

Query pressure passthrough: Federation queries execute on sub-Prometheus instances; large volumes of federation requests increase sub-node load
Cascading delay: Data flows from sub-Prometheus → federation Prometheus, adding 1-2 scrape cycle delays
No cross-Prometheus query aggregation: sum(rate(metric[5m])) is computed on each sub-Prometheus separately then aggregated, which may produce inaccurate results
No long-term storage: Federation doesn’t solve storage scaling

Conclusion: Federation is suitable for simple aggregation queries across small-scale clusters. For cross-cluster precise aggregate queries or long-term storage, use Thanos/Mimir/VictoriaMetrics.

VIII. Cortex Architecture (For Reference)

Cortex is Grafana Labs’ earlier solution, now gradually being replaced by Mimir. Understanding its architecture helps in understanding Mimir’s design.

Cortex vs Mimir relationship:

Dimension	Cortex	Mimir
Status	Maintenance mode	Active development
Architecture	Microservices	Microservices (optimized)
Storage engine	Chunk-based	Block-based (Prometheus TSDB compatible)
Compatibility	Partial PromQL	Full PromQL compatibility
Performance	Medium	High (2-5x Cortex)
Migration	Can migrate to Mimir	—

New projects should not choose Cortex — use Mimir directly.

IX. Solution Selection Comparison

9.1 Comprehensive Comparison Matrix

Dimension	Dual-Replica	Federation	Thanos	Mimir	VictoriaMetrics
High availability	✓	✓	✓✓	✓✓✓	✓✓
Long-term storage	✗	✗	✓✓	✓✓	✓✓✓
Horizontal scaling	✗	✗	✓	✓✓✓	✓✓✓
Multi-tenancy	✗	✗	✗	✓✓	✓ (Enterprise)
Global query	✗	✓	✓✓	✓✓✓	✓✓
Deployment complexity	Low	Low	Medium	High	Low-Medium
Storage cost	High	Medium	Low (object storage)	Low (object storage)	Very low
Query performance	High	Medium	Medium-High	High	Very high
Operational complexity	Low	Low	Medium	High	Low-Medium

9.2 Selection Decision Framework

Active time series < 500K?
  ├── Yes → Dual-replica + Alertmanager HA (simplest)
  └── No → Active time series < 2M?
              ├── Yes → Single-cluster Thanos Sidecar (object storage + global query)
              └── No → Need multi-tenancy?
                          ├── Yes → Mimir (native multi-tenant + horizontal scaling)
                          └── No → Pursuing extreme performance?
                                      ├── Yes → VictoriaMetrics cluster
                                      └── No → Thanos Receiver or Mimir

9.3 Recommendations by Scenario

Scenario	Recommended Solution	Reason
Startup / Small scale	Dual-replica	Lowest cost, simple ops
Single K8s cluster	Thanos Sidecar	Non-intrusive, mature community
Multiple K8s clusters	Thanos / VM	Global query + object storage
SaaS platform	Mimir	Native multi-tenancy + horizontal scaling
Large scale / Performance-focused	VictoriaMetrics	Highest compression, best query performance
Integrated visualization + alerting	Thanos / Mimir + Grafana	Deep Grafana ecosystem integration

X. Production Practice Recommendations

10.1 Tiered Data Retention

# Thanos Compactor downsampling configuration
# Raw data retained 30 days, 5m downsampled 90 days, 1h downsampled 1 year
compaction:
  retention_resolution_raw: 30d
  retention_resolution_5m: 90d
  retention_resolution_1h: 365d

10.2 remote_write Tuning

remote_write:
  - url: 'http://thanos-receive:19291/api/v1/receive'
    queue_config:
      capacity: 10000              # Queue capacity
      max_samples_per_send: 2000   # Samples per batch
      batch_send_deadline: 5s      # Batch send timeout
      min_backoff: 30ms            # Min retry interval
      max_backoff: 100ms           # Max retry interval
    remote_timeout: 30s
    write_relabel_configs:
      # Drop unneeded metrics
      - source_labels: [__name__]
        regex: 'go_(gc|memstats|threads)_.+'
        action: drop
      - source_labels: [__name__]
        regex: 'prometheus_.+'
        action: drop

10.3 Monitor Your Monitoring

Don’t forget to monitor Prometheus itself and remote storage:

# Monitor Thanos components
scrape_configs:
  - job_name: 'thanos-sidecar'
    static_configs:
      - targets: ['thanos-sidecar:10902']
  - job_name: 'thanos-query'
    static_configs:
      - targets: ['thanos-query:10902']

# Key alerting rules
groups:
  - name: prometheus-ha
    rules:
      - alert: PrometheusRemoteWriteFailures
        expr: rate(prometheus_remote_storage_failed_samples_total[5m]) > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Prometheus remote write failures"
          description: "Instance {{ $labels.instance }} remote write failure rate > 0"

      - alert: ThanosSidecarUploadFailing
        expr: rate(thanos_sidecar_bucket_uploads_failures_total[5m]) > 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Thanos Sidecar upload failures"

Summary

Prometheus HA and horizontal scaling involve progressive choices — there’s no one-size-fits-all solution. Core decision path:

Assess scale first: Evaluate active time series count, ingestion rate, and storage retention needs before deciding on a solution
Dual-replica is the baseline: Regardless of scale, dual-replica Prometheus + Alertmanager HA is the basic HA guarantee
Thanos for multi-cluster: Non-intrusive Sidecar mode, low object storage cost, mature community — the first choice for multi-cluster scenarios
Mimir for large-scale multi-tenancy: Native horizontal scaling and multi-tenancy, suitable for SaaS platforms and ultra-large-scale monitoring
VictoriaMetrics for performance: Optimal storage compression and query performance, suitable for cost- and performance-sensitive scenarios
Federation only for small scale: Simple but limited; large-scale scenarios should migrate to Thanos or VM

Remember: the availability of your monitoring system determines your operational ceiling. An unreliable monitoring system is more dangerous than no monitoring at all — because it gives you a false sense of security.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

Prometheus Official Documentation — HA — Prometheus Authors, referenced for Prometheus Official Documentation — HA
Thanos Official Documentation — Thanos Project, referenced for Thanos Official Documentation

Overview#

I. Single-Point Problem Analysis#

1.1 Risks of Single-Point Prometheus#

1.2 Single-Machine Performance Bottlenecks#

II. Dual-Replica Solution#

2.1 Dual-Replica Architecture#

2.2 Alertmanager HA Configuration#

2.3 Limitations of Dual Replicas#

III. Remote Storage Solutions Overview#

IV. Thanos: The Most Popular Remote Storage Solution#

4.1 Thanos Architecture#

4.2 Sidecar Mode Deployment#

4.3 Global Query#

4.4 Thanos Advantages and Limitations#

V. Mimir: Grafana Labs’ Next-Generation Solution#

5.1 Mimir Architecture#

5.2 Prometheus remote_write Configuration#

5.3 Mimir vs Thanos#

VI. VictoriaMetrics: High-Performance Proprietary Storage#

6.1 Architecture Characteristics#

6.2 Cluster Mode Deployment#

6.3 VictoriaMetrics Advantages#

6.4 vmagent: Lightweight Scraping Agent#

VII. Federation#

7.1 Federation Architecture#

7.2 Federation Configuration#

7.3 Federation Limitations#

VIII. Cortex Architecture (For Reference)#

IX. Solution Selection Comparison#

9.1 Comprehensive Comparison Matrix#

9.2 Selection Decision Framework#

9.3 Recommendations by Scenario#

X. Production Practice Recommendations#

10.1 Tiered Data Retention#

10.2 remote_write Tuning#

10.3 Monitor Your Monitoring#

Summary#

References & Acknowledgments#