Thanos Deployment and Practices: Prometheus Long-Term Storage and Global Query

Overview

Prometheus is the de facto standard in the cloud-native monitoring space, but it has notable shortcomings in long-term data storage and global querying: local storage retains only 15 days of data by default, single instances cannot aggregate queries across clusters, and high-availability solutions are relatively complex. Thanos, a CNCF incubating project, achieves unlimited-capacity long-term storage by uploading Prometheus data to object storage (such as S3, GCS, MinIO) and provides a cross-cluster global view through distributed query components.

This article will deeply analyze Thanos’s architecture design and, combined with production environment experience, detail the configuration, deployment, and operations of each component.

What Problems Does Thanos Solve

Before introducing Thanos, we need to understand the limitations of native Prometheus storage:

Dimension	Prometheus Native	Thanos Enhanced
Data retention	Default 15 days, limited by local disk	Theoretically unlimited, depends on object storage capacity
High availability	Requires Thanos Sidecar or remote_write dual-write	Sidecar + Query natively supported
Global query	Federation approach, limited and prone to data loss	Query component aggregates all Store APIs
Downsampling	Not supported	Compactor auto-downsampling, optimizes long-range queries
Historical data query	Lost beyond retention period	Can query data from months or even years ago
Cross-cluster view	Requires additional federation config	Native support for multi-cluster unified query

The core idea is: do not modify Prometheus itself; upload data to object storage via a Sidecar bypass, then unify queries through the Query component. This design preserves Prometheus’s simplicity while gaining enterprise-grade storage and query capabilities.

Core Architecture and Components

Overall Architecture

Thanos’s architecture revolves around three core processes: “Sidecar uploads, Store reads, Query aggregates”:

┌─────────────────────────────────────────────────────────┐
│              Object Storage (S3/MinIO/GCS)                │
│                                                          │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐              │
│  │ Block 1  │  │ Block 2  │  │ Block N  │              │
│  │ (2h raw) │  │ (5m down)│  │ (1h down)│              │
│  └──────────┘  └──────────┘  └──────────┘              │
│        ▲                                  ▲              │
│        │                                  │              │
│  ┌─────┴───────┐                  ┌──────┴──────┐       │
│  │  Sidecar    │                  │    Store    │       │
│  │ (Upload Blk)│                  │ (Read Blk)  │       │
│  └─────┬───────┘                  └──────┬──────┘       │
│        │                                  │              │
│  ┌─────┴───────┐                  ┌──────┴──────┐       │
│  │ Prometheus  │                  │  Compactor  │       │
│  │ (Local TSDB)│                  │ (Compress+  │       │
│  │             │                  │  Downsample)│       │
│  └─────────────┘                  └─────────────┘       │
│        │                                  │              │
│        └──────────┐    ┌──────────────────┘              │
│                   ▼    ▼                                  │
│              ┌──────────────┐                            │
│              │    Query     │  ◄── Grafana / PromQL     │
│              │ (Global Qry) │                            │
│              └──────┬───────┘                            │
│                     │                                    │
│              ┌──────┴───────┐                            │
│              │    Ruler     │  ──► Alertmanager         │
│              │ (Global Alert)│                            │
│              └──────────────┘                            │
└─────────────────────────────────────────────────────────┘

Component Responsibilities

Component	Responsibility	Deployment
Sidecar	Deployed in the same Pod as Prometheus, uploads TSDB Blocks to object storage, exposes Store API	DaemonSet/Sidecar
Query	Aggregates data from multiple Store API backends, executes PromQL queries, handles deduplication	Deployment (multi-replica HA)
Store Gateway	Reads Block data from object storage, exposes Store API to Query	Deployment (multi-replica HA)
Compactor	Compresses and downsamples Block data, enforces retention policies	Single instance (StatefulSet)
Ruler	Evaluates alert rules, sends alerts to Alertmanager	Deployment (multi-replica HA)
Receiver	Receives Prometheus data via remote_write (optional, for scenarios where Sidecar is impractical)	StatefulSet

Refer to the Thanos official architecture documentation for the complete design philosophy.

Production Deployment Practices

Prerequisites: Object Storage Configuration

Using MinIO (S3-compatible) as an example, first prepare the bucket and access credentials:

# Create bucket using mc client
mc alias set thanos-minio http://minio:9000 thanos-admin thanos-password
mc mb thanos-minio/thanos-data
mc admin policy set thanos-minio readwrite user=thanos-admin

Thanos object storage configuration file objstore.yml:

type: S3
config:
  bucket: thanos-data
  endpoint: minio:9000
  region: ""
  access_key: thanos-admin
  secret_key: thanos-password
  insecure: true
  http_config:
    idle_conn_timeout: 90s
    response_header_timeout: 15s
    insecure_skip_verify: false
  trace:
    enable: false
  list_objects_version: "v2"

Production recommendation: access_key and secret_key should be injected via Kubernetes Secrets, not stored in plaintext in a ConfigMap. The example above is for demonstration only.

Sidecar Deployment

The Sidecar runs in the same Pod as Prometheus and must meet two conditions:

Prometheus must be started with --storage.tsdb.min-block-duration=2h and --storage.tsdb.max-block-duration=2h parameters
The Sidecar needs objstore.yml configuration to upload data

# prometheus-with-thanos-sidecar.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: thanos-objstore-config
  labels:
    app.kubernetes.io/name: thanos
data:
  objstore.yml: |
    type: S3
    config:
      bucket: thanos-data
      endpoint: minio:9000
      access_key: thanos-admin
      secret_key: thanos-password
      insecure: true    
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: prometheus
  labels:
    app.kubernetes.io/name: prometheus
spec:
  serviceName: prometheus
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: prometheus
  template:
    metadata:
      labels:
        app.kubernetes.io/name: prometheus
        thanos-store: "true"
    spec:
      serviceAccountName: prometheus
      containers:
        - name: prometheus
          image: prom/prometheus:v2.55.0
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/data"
            - "--storage.tsdb.retention.time=6h"
            - "--storage.tsdb.min-block-duration=2h"
            - "--storage.tsdb.max-block-duration=2h"
            - "--web.enable-lifecycle"
            - "--web.enable-admin-api"
          ports:
            - containerPort: 9090
              name: web
          volumeMounts:
            - name: config
              mountPath: /etc/prometheus
            - name: data
              mountPath: /data
        - name: thanos-sidecar
          image: thanosio/thanos:v0.37.0
          args:
            - "sidecar"
            - "--tsdb.path=/data"
            - "--prometheus.url=http://localhost:9090"
            - "--objstore.config-file=/etc/thanos/objstore.yml"
            - "--shipper.upload-compacted"
            - "--reloader.config-file=/etc/prometheus/prometheus.yml"
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          ports:
            - containerPort: 10902
              name: grpc
            - containerPort: 10901
              name: http
          volumeMounts:
            - name: data
              mountPath: /data
            - name: thanos-config
              mountPath: /etc/thanos
      volumes:
        - name: config
          configMap:
            name: prometheus-config
        - name: thanos-config
          configMap:
            name: thanos-objstore-config
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 50Gi

Key parameter explanations:

--storage.tsdb.retention.time=6h: Only keep 6 hours of data locally; historical data can be cleaned from local storage after upload to object storage
--storage.tsdb.min-block-duration=2h and --max-block-duration=2h: Ensure Block size is fixed at 2 hours, which is a prerequisite for Thanos upload
--shipper.upload-compacted: Allow uploading compacted Blocks
--web.enable-admin-api: Sidecar needs to call Prometheus Admin API to get Block information

Query Component Deployment

Query is Thanos’s query entry point, responsible for collecting data from multiple Store API backends and executing PromQL:

# thanos-query.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-query
  labels:
    app.kubernetes.io/name: thanos-query
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: thanos-query
  template:
    metadata:
      labels:
        app.kubernetes.io/name: thanos-query
    spec:
      containers:
        - name: thanos-query
          image: thanosio/thanos:v0.37.0
          args:
            - "query"
            - "--grpc-address=0.0.0.0:10901"
            - "--http-address=0.0.0.0:10902"
            - "--log.level=info"
            - "--query.replica-label=prometheus_replica"
            - "--query.mode=distributed"
            - "--query.auto-downsampling"
            # Point to all Sidecar Store APIs
            - "--store=dnssrv+_grpc._tcp.prometheus.default.svc.cluster.local"
            # Point to Store Gateway
            - "--store=thanos-store-gateway:10901"
            # Point to Ruler
            - "--store=thanos-ruler:10901"
          ports:
            - containerPort: 10902
              name: http
            - containerPort: 10901
              name: grpc
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: 2
              memory: 4Gi
          livenessProbe:
            httpGet:
              path: /-/healthy
              port: http
          readinessProbe:
            httpGet:
              path: /-/ready
              port: http

Deduplication key: --query.replica-label=prometheus_replica tells the Query component which label to use for identifying multiple replicas of the same data. If two Prometheus instances collect the same metrics, Query will automatically deduplicate.

Store Gateway Deployment

The Store Gateway acts as a read proxy for object storage, allowing Query to access uploaded historical data:

# thanos-store-gateway.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-store-gateway
  labels:
    app.kubernetes.io/name: thanos-store-gateway
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: thanos-store-gateway
  template:
    metadata:
      labels:
        app.kubernetes.io/name: thanos-store-gateway
    spec:
      containers:
        - name: thanos-store-gateway
          image: thanosio/thanos:v0.37.0
          args:
            - "store"
            - "--data-dir=/data"
            - "--grpc-address=0.0.0.0:10901"
            - "--http-address=0.0.0.0:10902"
            - "--objstore.config-file=/etc/thanos/objstore.yml"
            - "--cache-index-header"
          ports:
            - containerPort: 10902
              name: http
            - containerPort: 10901
              name: grpc
          volumeMounts:
            - name: data
              mountPath: /data
            - name: thanos-config
              mountPath: /etc/thanos
          resources:
            requests:
              cpu: 200m
              memory: 512Mi
            limits:
              cpu: 1
              memory: 2Gi
      volumes:
        - name: thanos-config
          configMap:
            name: thanos-objstore-config
        - name: data
          emptyDir: {}

Compactor Deployment

The Compactor handles three tasks: compressing Blocks, executing downsampling, and enforcing retention policies. It is deployed as a single instance and cannot be multi-replica:

# thanos-compactor.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-compactor
  labels:
    app.kubernetes.io/name: thanos-compactor
spec:
  serviceName: thanos-compactor
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: thanos-compactor
  template:
    metadata:
      labels:
        app.kubernetes.io/name: thanos-compactor
    spec:
      containers:
        - name: thanos-compactor
          image: thanosio/thanos:v0.37.0
          args:
            - "compact"
            - "--data-dir=/data"
            - "--objstore.config-file=/etc/thanos/objstore.yml"
            - "--compact.concurrency=4"
            - "--retention.resolution-raw=90d"
            - "--retention.resolution-5m=180d"
            - "--retention.resolution-1h=365d"
            - "--wait"
          ports:
            - containerPort: 10902
              name: http
          volumeMounts:
            - name: data
              mountPath: /data
            - name: thanos-config
              mountPath: /etc/thanos
          resources:
            requests:
              cpu: 200m
              memory: 512Mi
            limits:
              cpu: 1
              memory: 2Gi
      volumes:
        - name: thanos-config
          configMap:
            name: thanos-objstore-config
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 100Gi

The retention policy is the most critical Compactor configuration:

Data Resolution	Retention Period	Use Case
Raw (original)	90 days	Short-term precise queries, alert evaluation
5min downsampled	180 days	Medium-term trend analysis
1h downsampled	365 days	Long-term capacity planning, annual reports

The principle of downsampling is to aggregate 2-hour raw Blocks into 5-minute or 1-hour resolution Blocks, dramatically reducing data volume. For example, 2 hours of raw data might be hundreds of MB, but downsampled to 1h resolution it’s only a few KB.

Ruler Deployment

Ruler allows defining and evaluating alert rules within Thanos, with rule evaluation based on global data rather than a single Prometheus instance:

# thanos-ruler.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-ruler
  labels:
    app.kubernetes.io/name: thanos-ruler
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: thanos-ruler
  template:
    metadata:
      labels:
        app.kubernetes.io/name: thanos-ruler
    spec:
      containers:
        - name: thanos-ruler
          image: thanosio/thanos:v0.37.0
          args:
            - "rule"
            - "--grpc-address=0.0.0.0:10901"
            - "--http-address=0.0.0.0:10902"
            - "--data-dir=/data"
            - "--rule-file=/etc/thanos/rules/*.yaml"
            - "--alertmanagers.url=http://alertmanager:9093"
            - "--query=thanos-query:10902"
            - "--label=ruler_cluster=\"prod\""
            - "--label=replica=\"$(POD_NAME)\""
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          ports:
            - containerPort: 10902
              name: http
            - containerPort: 10901
              name: grpc
          volumeMounts:
            - name: rules
              mountPath: /etc/thanos/rules
            - name: data
              mountPath: /data
          resources:
            requests:
              cpu: 200m
              memory: 512Mi
            limits:
              cpu: 1
              memory: 2Gi
      volumes:
        - name: rules
          configMap:
            name: thanos-rules
        - name: data
          emptyDir: {}

Example alert rule file rules/global-alerts.yaml:

groups:
  - name: thanos-component-health
    rules:
      - alert: ThanosSidecarDown
        expr: up{job="thanos-sidecar"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Thanos Sidecar is down"
          description: "Sidecar for Prometheus {{ $labels.instance }} has been offline for more than 5 minutes"

      - alert: ThanosCompactHalted
        expr: thanos_compactor_halted == 1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Thanos Compactor has halted"
          description: "Compactor component is in halted state; data compression and downsampling have stopped"

      - alert: ThanosQueryHighErrorRate
        expr: |
          sum(rate(thanos_query_concurrent_selects_gate_queries_in_flight[5m])) by (job)
          / sum(rate(thanos_query_range_requested_timespan_seconds_sum[5m])) by (job)
          > 0.05          
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Thanos Query error rate is too high"
          description: "Query component {{ $labels.job }} has a query error rate exceeding 5%"

High Availability Design

Query High Availability

The Query component itself is stateless; simply deploy multiple replicas. Load balancing is handled automatically through a Kubernetes Service:

apiVersion: v1
kind: Service
metadata:
  name: thanos-query
spec:
  selector:
    app.kubernetes.io/name: thanos-query
  ports:
    - name: http
      port: 10902
      targetPort: 10902
    - name: grpc
      port: 10901
      targetPort: 10901

When using with Grafana, point Grafana’s Prometheus data source to http://thanos-query:10902.

Prometheus High Availability

The core idea of Prometheus HA is: deploy two identical Prometheus instances (distinguished by the prometheus_replica label) that collect the same targets. Thanos Query automatically deduplicates via --query.replica-label:

# Prometheus 1
- "--storage.tsdb.retention.time=6h"
- "--storage.tsdb.min-block-duration=2h"
- "--storage.tsdb.max-block-duration=2h"
# External labels to distinguish replicas
external_labels:
  prometheus_replica: "prometheus-1"
  cluster: "prod-cluster"

# Prometheus 2
external_labels:
  prometheus_replica: "prometheus-2"
  cluster: "prod-cluster"

Important: The prometheus_replica in external_labels must match the Query’s --query.replica-label parameter. Otherwise, deduplication will not take effect and query results will contain duplicate data.

Object Storage Consistency

Object storage is Thanos’s single point of dependency. Ensure:

MinIO cluster has at least 4 nodes, using erasure coding mode to tolerate single-node failures
Regularly back up buckets to prevent accidental deletion
Monitor object storage health: Thanos Sidecar upload failures are recorded in the thanos_shipper_last_upload_success_timestamp_seconds metric

Performance Tuning

Query Performance Optimization

Thanos query performance is affected by multiple factors. Here are some key tuning points:

# Query tuning parameters
- "--query.timeout=2m"          # Single query timeout
- "--query.max-concurrent=20"   # Maximum concurrent queries
- "--query.max-concurrent-select=4"  # Maximum concurrent Store API calls per query
- "--query.auto-downsampling"   # Auto-downsampling
- "--query.replica-label=prometheus_replica"
- "--query.default-evaluation-interval=30s"

--query.auto-downsampling is an important feature: when the query time span is long, Query automatically selects appropriate downsampled data instead of raw data:

Query Time Span	Auto-Selected Resolution
< 40 hours	Raw (original data)
40h ~ 10 days	5min downsampled
> 10 days	1h downsampled

This significantly reduces query data volume and response time. For example, querying 30 days of CPU usage trends using 1h downsampled data requires only 1/720 of the raw data volume.

Store Gateway Caching

The Store Gateway’s index header caching is critical for query performance:

# Enable index header caching
- "--cache-index-header"
- "--store.caching-bucket.enabled=true"
- "--store.caching-bucket.max-size=1GB"

Compactor Performance

The Compactor can become a bottleneck when processing large numbers of Blocks:

# Increase concurrency
- "--compact.concurrency=4"
# Compactor data directory needs sufficient space
# Recommend at least 100GB SSD

Key metrics for monitoring the Compactor:

# Compaction iteration duration
thanos_compact_iterations_total

# Object storage operation latency
rate(thanos_objstore_bucket_operations_duration_seconds_sum[5m])
/ rate(thanos_objstore_bucket_operations_duration_seconds_count[5m])

# Number of Blocks produced by downsampling
increase(thanos_compact_downsample_total[1h])

Monitoring Thanos Itself

Key Metrics

Metric	Description	Alert Threshold Suggestion
`thanos_shipper_last_upload_success_timestamp_seconds`	Last successful upload time	No upload for over 4 hours
`thanos_compactor_halted`	Whether Compactor is halted	Alert when == 1
`thanos_query_concurrent_selects_gate_queries_in_flight`	Queries in progress	Continuously approaching max-concurrent
`thanos_objstore_bucket_operations_failures_total`	Object storage operation failures	Continuously increasing
`thanos_store_grpc_client_connections_inuse`	Store API connections	Suddenly drops to 0
`thanos_ruler_evaluation_failures_total`	Ruler evaluation failures	Continuously increasing

Thanos Self-Monitoring Dashboard

Recommended Grafana Dashboard templates provided by Thanos: ID 14388 (Thanos Overview) and 14389 (Thanos Receive).

After importing into Grafana, ensure the data source points to the Thanos Query component. Core panels include:

Component health status: up status of each component
Upload latency: Sidecar Block upload latency trends
Query latency distribution: P50/P95/P99 query latency
Object storage operations: Read/write latency and error rates
Compactor running status: Compression and downsampling progress

Common Troubleshooting

Sidecar Upload Failures

# Check Sidecar logs
kubectl logs -l app.kubernetes.io/name=prometheus -c thanos-sidecar | grep -i "upload\|error"

# Check object storage connectivity
kubectl exec -it prometheus-0 -c thanos-sidecar -- \
  thanos tools bucket verify \
  --objstore.config-file=/etc/thanos/objstore.yml

# View Block upload status
kubectl exec -it prometheus-0 -c thanos-sidecar -- \
  thanos tools bucket inspect \
  --objstore.config-file=/etc/thanos/objstore.yml \
  --output=json

Common causes:

Symptom	Possible Cause	Solution
Upload returns 403	Insufficient credentials	Check S3/MinIO access policies
Blocks never upload	Prometheus Block hasn’t completed 2h duration	Check TSDB Block configuration
Upload timeout	Insufficient network bandwidth	Increase timeout or optimize network
Object storage bucket full	Storage quota limit	Scale up or adjust retention policy

Slow Queries

# View Query's Store API connection status
curl -s http://thanos-query:10902/api/v1/status/stores | jq .

# Analyze slow queries
curl -s 'http://thanos-query:10902/api/v1/query?query=up' | jq .stats

Optimization directions:

Enable --query.auto-downsampling for long-range queries using downsampled data
Increase Store Gateway replicas to distribute read load
Optimize PromQL queries to avoid high-cardinality labels and full rate() calculations
Check object storage read/write latency and upgrade storage performance if needed

Compactor Stuck

A stuck Compactor will prevent data from being compressed and downsampled, ultimately affecting query performance:

# Check Compactor status
curl -s http://thanos-compactor:10902/api/v1/status/config | jq .

# View Block status
kubectl exec thanos-compactor-0 -- \
  thanos tools bucket inspect \
  --objstore.config-file=/etc/thanos/objstore.yml

Common causes and solutions:

Insufficient disk space: Compactor needs temporary space to download and compress Blocks. Increase PVC capacity or clean up --data-dir
Block corruption: Use thanos tools bucket verify to check and repair corrupted Blocks
Concurrency conflicts: Multiple Compactor replicas operating simultaneously. Ensure only one Compactor instance is running

Multi-Cluster Monitoring Architecture

An important use case for Thanos is unified monitoring across multiple Kubernetes clusters. Recommended architecture:

Cluster A                        Cluster B
┌─────────────────────┐        ┌─────────────────────┐
│ Prometheus + Sidecar│        │ Prometheus + Sidecar│
│         │           │        │         │           │
│    ┌────▼────┐      │        │    ┌────▼────┐      │
│    │ Sidecar │──────┼────────┼───►│ Sidecar │      │
│    └─────────┘      │        │    └─────────┘      │
└─────────┬───────────┘        └─────────┬───────────┘
          │                              │
          ▼                              ▼
   ┌──────────────────────────────────────────┐
   │       Object Storage (S3/MinIO)          │
   └──────────────────┬───────────────────────┘
                      │
              ┌───────┴───────┐
              │ Store Gateway │
              └───────┬───────┘
                      │
              ┌───────┴───────┐
              │     Query     │ ◄── Grafana
              └───────┬───────┘
                      │
              ┌───────┴───────┐
              │     Ruler     │ ──► Alertmanager
              └───────────────┘

Each cluster needs unique external_labels to ensure Query can distinguish data sources:

# Cluster A's Prometheus
external_labels:
  cluster: "cluster-a"
  prometheus_replica: "prometheus-1"

# Cluster B's Prometheus
external_labels:
  cluster: "cluster-b"
  prometheus_replica: "prometheus-1"

In Grafana, you can switch between clusters using the $cluster variable:

# View CPU usage by cluster
sum(rate(node_cpu_seconds_total{mode!="idle", cluster="$cluster"}[5m])) by (instance)

# Cross-cluster comparison
sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (cluster)

Cost Optimization

Object Storage Costs

The primary cost of Thanos comes from object storage. Optimization strategies:

Strategy	Effect	Implementation
Adjust retention policy	Reduce 30-50% storage	Reasonably set raw/5m/1h retention periods
Increase scrape interval	Reduce data volume	Adjust from 15s to 30s or 60s
Filter unused metrics	Reduce data volume	Use `metric_relabel_configs`
Lifecycle policies	Auto-tier storage	Configure S3 Lifecycle rules

Compute Resource Costs

Component	Minimum Resources	Recommended Resources	Scaling Method
Sidecar	100m CPU / 128Mi	200m / 512Mi	One per Prometheus
Query	500m / 1Gi	2 CPU / 4Gi	Multi-replica load balancing
Store Gateway	200m / 512Mi	1 CPU / 2Gi	Multi-replica read distribution
Compactor	200m / 512Mi	1 CPU / 2Gi	Single instance
Ruler	200m / 512Mi	1 CPU / 2Gi	Multi-replica + deduplication

Comparison with Other Solutions

Feature	Thanos	Cortex	VictoriaMetrics
Architecture model	Sidecar bypass upload	remote_write centralized	remote_write centralized
Changes to Prometheus	None required	Requires remote_write config	Requires remote_write config
Object storage	Required	Optional	Optional
Global query	Native support	Native support	Native support
Downsampling	Compactor automatic	Manual configuration	Automatic
Multi-tenancy	Label-based isolation	Native support	Native support
Deployment complexity	Medium	High	Low
Community activity	CNCF incubating	CNCF incubating	Independent project

Selection recommendations:

Thanos: Suitable for existing Prometheus deployments, no desire to change collection methods, need for cross-cluster global queries
Cortex/Mimir: Suitable for multi-tenant, remote_write centralized SaaS scenarios
VictoriaMetrics: Suitable for pursuing high performance and low resource consumption, acceptable with vendor lock-in

Summary

Thanos elegantly solves Prometheus’s long-term storage and global query problems through its Sidecar + object storage + distributed query architecture, while maintaining zero intrusion into Prometheus. In production deployments, focus on these key points:

Object storage is the foundation: Ensure high availability and data consistency of S3/MinIO, as it is the single point of dependency for the entire Thanos system
Reasonable retention policies: Raw data retained for 90 days meets daily alerting and troubleshooting needs; 5min and 1h downsampled data retained for 180 and 365 days respectively for trend analysis
HA starts with Prometheus: Dual-replica Prometheus + replica labels + Query deduplication is the foundation of high availability
Monitor Thanos itself: Upload failures, Compactor halts, query timeouts all need timely detection
Controllable costs: Through retention policies, scrape interval optimization, and metric filtering, keep object storage costs within reasonable limits

Thanos’s design philosophy is “each component does one thing and does it well.” This UNIX-style modular design allows each component to be deployed, scaled, and operated independently, making it well-suited for large-scale production environments.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

Prometheus TSDB 格式 — Prometheus Authors, referenced for Prometheus TSDB 格式
Google SRE Book — Monitoring Distributed Systems — Google SRE Team, referenced for Google SRE Book — Monitoring Distributed Systems
Thanos 官方文档 — Thanos Project, referenced for Thanos 官方文档
Thanos Bucket Inspector 工具 — Thanos Project, referenced for Thanos Bucket Inspector 工具
Thanos 存储 API 设计 — Thanos Project, referenced for Thanos 存储 API 设计
Thanos 官方架构文档 — Thanos Project, referenced for Thanos 官方架构文档

Overview#

What Problems Does Thanos Solve#

Core Architecture and Components#

Overall Architecture#

Component Responsibilities#

Production Deployment Practices#

Prerequisites: Object Storage Configuration#

Sidecar Deployment#

Query Component Deployment#

Store Gateway Deployment#

Compactor Deployment#

Ruler Deployment#

High Availability Design#

Query High Availability#

Prometheus High Availability#

Object Storage Consistency#

Performance Tuning#

Query Performance Optimization#

Store Gateway Caching#

Compactor Performance#

Monitoring Thanos Itself#

Key Metrics#

Thanos Self-Monitoring Dashboard#

Common Troubleshooting#

Sidecar Upload Failures#

Slow Queries#

Compactor Stuck#

Multi-Cluster Monitoring Architecture#

Cost Optimization#

Object Storage Costs#

Compute Resource Costs#

Comparison with Other Solutions#

Summary#

References & Acknowledgments#