VictoriaMetrics Deployment and Practices: A High-Performance Alternative for Prometheus Long-Term Storage

Overview

As the de facto standard for cloud-native monitoring, Prometheus has become the default choice for microservice and Kubernetes monitoring. However, as business scale grows, Prometheus’s local storage architecture increasingly exposes significant bottlenecks: limited single-machine storage capacity (default 15-day retention), no native horizontal scaling, memory spikes under high-cardinality scenarios, and difficulty querying historical data. Many teams find themselves facing the “triple dilemma” of disk IO pressure, storage cost inflation, and query latency growth once time series exceed the million-level mark.

To address these issues, the community has developed various long-term storage solutions including Thanos, Cortex, and VictoriaMetrics. Among them, VictoriaMetrics (hereafter VM) has become the preferred choice for an increasing number of teams thanks to its outstanding compression ratio, minimal operational complexity, and excellent query performance.

This article systematically covers VictoriaMetrics’s architecture design, deployment modes, data migration, performance tuning, and production best practices to help you make the right technical choices and implement them in real projects.

Why VictoriaMetrics

Prometheus Storage Bottlenecks

To understand VictoriaMetrics’s value, we first need to understand where Prometheus’s storage bottlenecks lie:

Problem	Cause	Impact
Short data retention	Default TSDB retains only 15 days	Cannot do long-term trend analysis
No horizontal scaling	Single-instance architecture, no sharding	Single-machine memory and disk become hard limits
High-cardinality memory bloat	Label combination explosion leads to index inflation	Frequent OOMs, monitoring unavailable
Difficult global queries	Data scattered across instances	Cross-cluster queries require additional solutions
Remote storage latency	remote_write sync model	Network issues cause data loss

Core Advantages of VictoriaMetrics

VictoriaMetrics is designed with the principle of providing higher performance and lower resource consumption while being fully compatible with the Prometheus ecosystem.

1. Superior Data Compression

VM uses a self-developed columnar storage engine, deeply optimized for time series data characteristics. According to official benchmarks and extensive community practice, VM typically requires only 1/5 to 1/7 of the disk space compared to Prometheus TSDB for the same monitoring data.

# Data Compression Comparison (based on 1M active time series, 30 days of data)

Storage Solution       Disk Usage   Compression   Memory Usage
─────────────────────────────────────────────────────────────────
Prometheus (local)     350 GB       1x           8 GB
Thanos (S3)            120 GB       2.9x         6 GB (incl. Sidecar)
VictoriaMetrics         55 GB       6.4x         3 GB
VictoriaMetrics (cluster) 58 GB     6.0x         3.2 GB (total)

Source: VictoriaMetrics official benchmarks and community practice reports. Actual results vary by data characteristics; testing in your own scenario is recommended.

2. Minimal Operational Complexity

Compared to Thanos’s multi-component architecture (Sidecar, Store, Compactor, Query, Receiver, Rule) and Cortex’s microservice architecture (Distributor, Ingester, Querier, Compactor, Store Gateway, Ruler, Alertmanager), VM’s architecture is extremely simple:

Solution	Core Components	External Dependencies	Operational Complexity
Thanos	6+	Object storage (S3/MinIO)	High
Cortex	7+	Object storage + DynamoDB/etcd	Very high
VictoriaMetrics (single-node)	1	None	Very low
VictoriaMetrics (cluster)	3	None	Low

3. High-Performance Queries

VM efficiently utilizes all available CPU cores for parallel processing. A single instance can handle millions of data points per second for ingestion and scan billions of rows per query.

4. Multi-Protocol Compatibility

# Data Ingestion Protocols Supported by VictoriaMetrics

Pull mode (Prometheus compatible):
  └── vmagent → scrape Prometheus exporters → write to VM

Push mode:
  ├── Prometheus remote_write → VM (most common)
  ├── Graphite plaintext protocol
  ├── OpenTSDB telnet/HTTP
  ├── InfluxDB line protocol
  ├── OpenTelemetry
  └── CSV import

Query compatibility:
  ├── PromQL (fully compatible)
  ├── MetricsQL (PromQL superset with extended features)
  └── Grafana native integration

Architecture Design Deep Dive

Storage Engine

The core of VictoriaMetrics is its self-developed columnar storage engine. Understanding its internal mechanisms helps optimize usage.

Data Ingestion Flow:

Data ingestion → Protobuf encoding/serialization
               → Memory buffer (batch flush)
               → Label index construction (inverted index + TSID)
               → Columnar compressed write to disk
               → Background merging and downsampling

Key design decisions:

TSID (Time Series ID): VM maps label combinations to internal efficient TSIDs, avoiding scanning all labels for every query. This is more efficient than Prometheus’s label index.
Shared string pool: Identical label values are stored only once in memory, reused via references, significantly reducing memory consumption in high-cardinality scenarios.
Lazy loading: Queries only load the required data blocks, not entire time series, reducing IO overhead.
Columnar compression: Each data column (timestamp, value, labels) is compressed independently using optimal algorithms for different data types.

Storage Directory Structure:

/var/lib/victoria-metrics-data/
├── data/
│   ├── small/               # Small table partitions (recent data)
│   │   ├── 2024_01/        # Monthly partitioned data blocks
│   │   │   ├── index.bin   # Inverted index
│   │   │   ├── timestamps.bin
│   │   │   └── values.bin
│   │   └── ...
│   ├── big/                 # Large table partitions (historical data)
│   │   └── ...
│   └── indexdb/            # Index database
│       ├── index.bin
│       └── metadata.json
├── metadata/
│   └── ...
└── snapshots/               # Snapshots (for backup)
    └── ...

Cluster Architecture Components

The VictoriaMetrics cluster version consists of three core components:

                    ┌──────────────┐
                    │   vmagent    │  (optional: data collection/sharding/replication)
                    │   (N instances)│
                    └──────┬───────┘
                           │
          ┌────────────────┼────────────────┐
          │                │                │
          ▼                ▼                ▼
  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
  │  vminsert    │ │  vminsert    │ │  vminsert    │
  │  (inst 1)    │ │  (inst 2)    │ │  (inst N)    │
  └──────┬───────┘ └──────┬───────┘ └──────┬───────┘
         │                │                │
         └────────────────┼────────────────┘
                          │
          ┌───────────────┼───────────────┐
          │               │               │
          ▼               ▼               ▼
  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
  │ vmstorage    │ │ vmstorage    │ │ vmstorage    │
  │  (node 1)    │ │  (node 2)    │ │  (node N)    │
  │  data store  │ │  data store  │ │  data store  │
  └──────┬───────┘ └──────┬───────┘ └──────┬───────┘
         │                │                │
         └────────────────┼────────────────┘
                          │
          ┌───────────────┼───────────────┐
          │               │               │
          ▼               ▼               ▼
  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
  │  vmselect    │ │  vmselect    │ │  vmselect    │
  │  (inst 1)    │ │  (inst 2)    │ │  (inst N)    │
  └──────┬───────┘ └──────┬───────┘ └──────┬───────┘
         │                │                │
         └────────────────┼────────────────┘
                          │
                    ┌─────┴──────┐
                    │  Grafana    │
                    │  / Client   │
                    └────────────┘

Component Responsibilities:

Component	Responsibility	Stateless	Horizontally Scalable
vminsert	Receives write requests, routes to vmstorage nodes	Yes	Yes
vmstorage	Data storage and query execution	No (stateful)	Yes (sharding)
vmselect	Receives query requests, fetches and merges results from vmstorage	Yes	Yes
vmagent	Data collection, sharding, replication (optional)	Yes	Yes
vmalert	Alerting rule evaluation (optional)	Yes	Yes
vmbackup	Data backup (optional)	Yes	-

Key Design Decisions:

vminsert and vmselect are stateless and can be freely scaled
vmstorage is stateful, sharded via consistent hashing; rebalancing is needed when scaling
Routing from vminsert to vmstorage uses consistent hashing to ensure the same time series always writes to the same storage node
vmselect queries all vmstorage nodes and merges results

Deployment Practices

Single-Node Deployment

Suitable for small-to-medium scale (under 1 million active time series) or testing environments.

#!/bin/bash
# VictoriaMetrics single-node deployment script

# Download binary
VM_VERSION="v1.115.0"
wget "https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/${VM_VERSION}/victoria-metrics-linux-amd64-v${VM_VERSION#v}.tar.gz"
tar -xzf "victoria-metrics-linux-amd64-v${VM_VERSION#v}.tar.gz"

# Create data directory
mkdir -p /var/lib/victoria-metrics-data

# Start single node
./victoria-metrics-prod \
  -storageDataPath=/var/lib/victoria-metrics-data \
  -retentionPeriod=90d \
  -httpListenAddr=:8428 \
  -memory.allowedBytes=4GB \
  -search.maxConcurrentQueries=8 \
  > /var/log/victoria-metrics.log 2>&1 &

echo "VictoriaMetrics single node started, listening on port 8428"

systemd Service Configuration:

# /etc/systemd/system/victoria-metrics.service
[Unit]
Description=VictoriaMetrics Single Node
After=network.target

[Service]
Type=simple
User=victoria-metrics
Group=victoria-metrics
ExecStart=/usr/local/bin/victoria-metrics-prod \
  -storageDataPath=/var/lib/victoria-metrics-data \
  -retentionPeriod=90d \
  -httpListenAddr=:8428 \
  -memory.allowedBytes=4GB
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Configure Prometheus Remote Write:

# prometheus.yml — add remote_write configuration
global:
  scrape_interval: 15s
  evaluation_interval: 15s

remote_write:
  - url: "http://victoria-metrics:8428/api/v1/write"
    queue_config:
      capacity: 10000
      max_samples_per_send: 2000
      batch_send_deadline: 5s
      min_backoff: 1s
      max_backoff: 30s
    # For HA, configure multiple remote write endpoints
    # remote_timeout: 30s

# Keep existing scrape configs unchanged
scrape_configs:
  - job_name: "node-exporter"
    static_configs:
      - targets: ["node-exporter:9100"]

  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod

Cluster Deployment

Suitable for large scale (over 1 million active time series) or HA scenarios.

Docker Compose Deployment Example:

# docker-compose.yml
version: '3.8'

services:
  # --- vmstorage nodes (stateful, need persistence) ---
  vmstorage-1:
    image: victoriametrics/vmstorage:v1.115.0-cluster
    command:
      - '--storageDataPath=/storage'
      - '--retentionPeriod=180d'
      - '--httpListenAddr=:8482'
    volumes:
      - vmstorage-1-data:/storage
    ports:
      - "8482"
    restart: unless-stopped

  vmstorage-2:
    image: victoriametrics/vmstorage:v1.115.0-cluster
    command:
      - '--storageDataPath=/storage'
      - '--retentionPeriod=180d'
      - '--httpListenAddr=:8482'
    volumes:
      - vmstorage-2-data:/storage
    ports:
      - "8482"
    restart: unless-stopped

  # --- vminsert nodes (stateless) ---
  vminsert-1:
    image: victoriametrics/vminsert:v1.115.0-cluster
    command:
      - '--httpListenAddr=:8480'
      - '--storageNode=vmstorage-1:8400'
      - '--storageNode=vmstorage-2:8400'
    ports:
      - "8480"
    depends_on:
      - vmstorage-1
      - vmstorage-2
    restart: unless-stopped

  vminsert-2:
    image: victoriametrics/vminsert:v1.115.0-cluster
    command:
      - '--httpListenAddr=:8480'
      - '--storageNode=vmstorage-1:8400'
      - '--storageNode=vmstorage-2:8400'
    ports:
      - "8480"
    depends_on:
      - vmstorage-1
      - vmstorage-2
    restart: unless-stopped

  # --- vmselect nodes (stateless) ---
  vmselect-1:
    image: victoriametrics/vmselect:v1.115.0-cluster
    command:
      - '--httpListenAddr=:8481'
      - '--storageNode=vmstorage-1:8401'
      - '--storageNode=vmstorage-2:8401'
    ports:
      - "8481"
    depends_on:
      - vmstorage-1
      - vmstorage-2
    restart: unless-stopped

  vmselect-2:
    image: victoriametrics/vmselect:v1.115.0-cluster
    command:
      - '--httpListenAddr=:8481'
      - '--storageNode=vmstorage-1:8401'
      - '--storageNode=vmstorage-2:8401'
    ports:
      - "8481"
    depends_on:
      - vmstorage-1
      - vmstorage-2
    restart: unless-stopped

  # --- Load Balancer ---
  lb-insert:
    image: nginx:alpine
    volumes:
      - ./nginx-insert.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "8480:8480"
    depends_on:
      - vminsert-1
      - vminsert-2
    restart: unless-stopped

  lb-select:
    image: nginx:alpine
    volumes:
      - ./nginx-select.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "8481:8481"
    depends_on:
      - vmselect-1
      - vmselect-2
    restart: unless-stopped

  # --- Grafana ---
  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    ports:
      - "3000:3000"
    restart: unless-stopped

volumes:
  vmstorage-1-data:
  vmstorage-2-data:

Nginx Load Balancer Configuration (Write):

# nginx-insert.conf
events {}

http {
    upstream vminsert {
        least_conn;
        server vminsert-1:8480;
        server vminsert-2:8480;
    }

    server {
        listen 8480;

        location / {
            proxy_pass http://vminsert;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_connect_timeout 10s;
            proxy_send_timeout 30s;
            proxy_read_timeout 30s;
        }
    }
}

Nginx Load Balancer Configuration (Query):

# nginx-select.conf
events {}

http {
    upstream vmselect {
        least_conn;
        server vmselect-1:8481;
        server vmselect-2:8481;
    }

    server {
        listen 8481;

        location / {
            proxy_pass http://vmselect;
            proxy_set_header Host $host;
            proxy_connect_timeout 10s;
            proxy_send_timeout 60s;
            proxy_read_timeout 60s;
        }
    }
}

Kubernetes Deployment (VictoriaMetrics Operator)

For production environments, the VictoriaMetrics Operator is recommended for managing VM clusters on Kubernetes.

# vmcluster.yaml — Using VictoriaMetrics Operator
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
  name: vm-cluster
  namespace: monitoring
spec:
  retentionPeriod: "180d"
  replicationFactor: 2

  # vmstorage configuration
  vmstorage:
    replicaCount: 3
    storageDataPath: "/vm-data"
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: fast-ssd
          resources:
            requests:
              storage: 500Gi
    resources:
      limits:
        cpu: 4
        memory: 16Gi
      requests:
        cpu: 2
        memory: 8Gi

  # vminsert configuration
  vminsert:
    replicaCount: 2
    resources:
      limits:
        cpu: 2
        memory: 4Gi
      requests:
        cpu: 1
        memory: 2Gi

  # vmselect configuration
  vmselect:
    replicaCount: 2
    resources:
      limits:
        cpu: 2
        memory: 4Gi
      requests:
        cpu: 1
        memory: 2Gi
    cacheMountPath: "/cache"
    storage:
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: 10Gi
---
# vmagent configuration — replacing Prometheus scraping
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
  name: vm-agent
  namespace: monitoring
spec:
  replicaCount: 2
  serviceScrapeNamespaceSelector: {}
  podScrapeNamespaceSelector: {}
  nodeScrapeNamespaceSelector: {}
  staticScrapeNamespaceSelector: {}
  remoteWrite:
    - url: "http://vm-cluster-vminsert.monitoring.svc:8480/insert/0/prometheus/api/v1/write"
  resources:
    limits:
      cpu: 1
      memory: 1Gi
    requests:
      cpu: 500m
      memory: 512Mi
---
# vmalert configuration — alerting rule evaluation
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
  name: vm-alert
  namespace: monitoring
spec:
  replicaCount: 2
  datasource:
    url: "http://vm-cluster-vmselect.monitoring.svc:8481/select/0/prometheus"
  notifier:
    url: "http://alertmanager.monitoring.svc:9093"
  evaluationInterval: "30s"
  ruleNamespaceSelector: {}
  resources:
    limits:
      cpu: 1
      memory: 1Gi
    requests:
      cpu: 500m
      memory: 512Mi

Reference: VictoriaMetrics Operator Documentation

Data Collection and Migration

Using vmagent to Replace Prometheus Scraping

vmagent is VictoriaMetrics’s data collection component. It can directly scrape Prometheus exporters, supports sharding and replication, and is an ideal replacement for Prometheus in production environments.

# vmagent configuration example
global:
  scrape_interval: 15s
  external_labels:
    cluster: "production"
    region: "us-east-1"

# Scrape configs (fully compatible with Prometheus)
scrape_configs:
  - job_name: "kubernetes-nodes"
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - source_labels: [__address__]
        regex: "(.*):.*"
        target_label: __address__
        replacement: "${1}:9100"

  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"

# Remote write to VictoriaMetrics
remote_write:
  - url: "http://vminsert:8480/insert/0/prometheus/api/v1/write"
    # For HA: write to multiple vminsert instances
    # vmagent automatically handles duplicate data

vmagent Advanced Feature — Sharded Scraping:

# Multi-instance vmagent sharded scraping (each instance scrapes only a subset of targets)
# Instance 1
./vmagent-prod \
  -promscrape.config=/etc/vmagent/scrape.yml \
  -remoteWrite.url=http://vminsert:8480/insert/0/prometheus/api/v1/write \
  -promscrape.cluster.membersCount=2 \
  -promscrape.cluster.memberNum=0

# Instance 2
./vmagent-prod \
  -promscrape.config=/etc/vmagent/scrape.yml \
  -remoteWrite.url=http://vminsert:8480/insert/0/prometheus/api/v1/write \
  -promscrape.cluster.membersCount=2 \
  -promscrape.cluster.memberNum=1

vmagent natively supports -remoteWrite.shardByURL since v1.86, and v1.138.0 further upgraded the data distribution algorithm to consistent hashing, significantly reducing data redistribution during node changes.

Migrating Historical Data from Prometheus

If you need to migrate historical data from an existing Prometheus instance to VictoriaMetrics, there are several approaches:

Method 1: Restore from Snapshot using vmrestore

# 1. Create a TSDB snapshot on the Prometheus side
curl -XPOST http://prometheus:9090/api/v1/admin/tsdb/snapshot

# 2. Import the snapshot into VM using vmrestore
./vmrestore-prod \
  -src=s3://my-bucket/prom-snapshots/ \
  -dst=/var/lib/victoria-metrics-data/

Method 2: Migrate using vmctl Tool

# vmctl is the official data migration tool, supporting multiple data sources

# Migrate from Prometheus TSDB
./vmctl-prod prometheus \
  -src.address=http://prometheus:9090 \
  -dst.url=http://victoria-metrics:8428 \
  -dst.addr=http://victoria-metrics:8428

# Migrate from InfluxDB
./vmctl-prod influxdb \
  -src.addr=http://influxdb:8086 \
  -src.database=monitoring \
  -dst.url=http://victoria-metrics:8428

# Migrate from OpenTSDB
./vmctl-prod opentsdb \
  -src.addr=http://opentsdb:4242 \
  -dst.url=http://victoria-metrics:8428

# Migrate from remote Prometheus-compatible storage
./vmctl-prod remote \
  -src.addr=http://remote-storage:9090 \
  -dst.url=http://victoria-metrics:8428 \
  -time-filter='{"start":"2025-01-01T00:00:00Z","end":"2026-07-01T00:00:00Z"}'

Method 3: Dual-Write Transition Period

# Transition: simultaneously write to Prometheus local and VictoriaMetrics
# prometheus.yml
remote_write:
  - url: "http://victoria-metrics:8428/api/v1/write"
    queue_config:
      capacity: 10000

# Steps:
# 1. Configure Prometheus remote_write to VM (start dual-write)
# 2. Observe data consistency, confirm no loss
# 3. Migrate historical data (vmctl)
# 4. Verify historical data completeness
# 5. Switch Grafana datasource to VM
# 6. After confirming stability, disable Prometheus remote_write
# 7. Eventually replace Prometheus entirely with vmagent

Performance Tuning

Write Performance Optimization

1. Tune remote_write Batch Parameters

# Prometheus remote_write tuning
remote_write:
  - url: "http://victoria-metrics:8428/api/v1/write"
    queue_config:
      capacity: 25000          # Queue capacity (default 10000)
      max_samples_per_send: 5000  # Samples per batch (default 100)
      batch_send_deadline: 2s   # Batch send timeout (default 5s)
      min_backoff: 500ms       # Min retry interval (default 1s)
      max_backoff: 10s         # Max retry interval (default 30s)
    remote_timeout: 30s

2. Tune VM Memory Limits

# vmstorage memory allocation
./victoria-metrics-prod \
  -memory.allowedBytes=8GB \
  # VM uses ~60% of allowed memory as cache
  # The rest is for OS and other processes
  
  # Cache size tuning
  -cacheExpireDuration=6h \
  -dedup.minScrapeInterval=30s
  
  # Write optimization
  -insert.maxQueueDuration=1m \
  -insert.maxBlockDuration=5m

3. Control High-Cardinality Metrics

High cardinality is the number one killer of monitoring systems. The following labels need strict control:

# Check high-cardinality metrics
# Sort by time series count, find the most resource-consuming metrics
topk(20, count by (__name__)({__name__=~".+"}))

# Check label cardinality
topk(20, count by (__name__, job)({__name__=~".+"}))

# Find metrics with label combination explosion
topk(10, count by (__name__)({__name__=~"http_request_.*"}))

High-Cardinality Governance Recommendations:

Scenario	Problem	Solution
HTTP requests with `path` label	Each URL path generates a time series	Normalize paths, remove IDs and params
Containers with `container_id`	Each container instance generates a series	Use `container_name` instead
User-level monitoring with `user_id`	Each user generates a series	Aggregate to tenant/team level
Exception tracking with `stack_trace`	Each exception stack generates a series	Keep only exception type and message

Query Performance Optimization

1. Optimize Queries with MetricsQL

MetricsQL is VictoriaMetrics’s extension of PromQL, providing additional optimization functions:

# Standard PromQL query
rate(http_requests_total[5m])

# MetricsQL optimized — using range_first / range_last
# Only takes window start and end values, reducing computation
# Suitable for counter queries over large ranges
rate(http_requests_total[5m] @ end())

# MetricsQL's keep_metric_names modifier
# Preserves original metric names for identification after aggregation
sum by (job) (rate(http_requests_total[5m])) keep_metric_names

# MetricsQL's lag() function
# Handles delayed data, preventing calculation bias
rate(http_requests_total[5m] lag(30s))

2. Downsampling Queries

For long-term data queries, use downsampling to reduce data volume:

# Original query (30 days of data, one point per 15s = 172,800 points)
rate(cpu_usage[30d])

# Using MetricsQL's downgrade
# Downsample 30 days to 1-hour granularity = 720 points
# Significantly reduces query data and response time
rate(cpu_usage[30d:1h])

# Manual downsampling
# Take one average per hour
avg_over_time(rate(cpu_usage[5m])[30d:1h])

3. Query Caching

# vmselect query caching
./vmselect-prod \
  -cacheDataPath=/cache \
  # Query result cache
  -search.cacheSize=2GB \
  -search.cacheTTL=5m \
  # Index cache
  -search.indexCacheSize=1GB \
  # Filter cache
  -search.filterCacheSize=1GB

Storage Optimization

# vmstorage storage optimization parameters
./vmstorage-prod \
  -storageDataPath=/vm-data \
  -retentionPeriod=180d \
  
  # Downsampling configuration
  -downsampling.period=30d:5m,180d:1h \
  # Meaning: data older than 30 days is downsampled to 5-min granularity,
  # data older than 180 days is downsampled to 1-hour granularity
  
  # Data deduplication
  -dedup.minScrapeInterval=30s \
  # When multiple vmagents scrape the same target, auto-deduplicate
  
  # Index optimization
  -index.maxSeriesPerIndexBlock=300000 \
  
  # Memory management
  -memory.allowedBytes=16GB

High Availability Architecture Design

Replication and Disaster Recovery

The VictoriaMetrics cluster version supports data replication, ensuring no data loss when a single node fails:

# vminsert configured with data replication
# replicationFactor=2 means each data point is written to 2 vmstorage nodes
./vminsert-prod \
  -httpListenAddr=:8480 \
  -storageNode=vmstorage-1:8400 \
  -storageNode=vmstorage-2:8400 \
  -storageNode=vmstorage-3:8400 \
  -replicationFactor=2

# Replication Architecture (replicationFactor=2, 3 storage nodes)

Write Request → vminsert
                  │
                  ├──→ vmstorage-1 (primary)  ──→ vmstorage-2 (replica)  ✓ success
                  ├──→ vmstorage-2 (primary)  ──→ vmstorage-3 (replica)  ✓ success
                  └──→ vmstorage-3 (primary)  ──→ vmstorage-1 (replica)  ✓ success

# After vmstorage-1 goes down:
# vminsert automatically writes data to vmstorage-2 and vmstorage-3
# vmselect fetches data from surviving nodes, transparent to users

Backup and Recovery

#!/bin/bash
# VictoriaMetrics backup script

# 1. Create snapshot
SNAPSHOT_PATH=$(curl -s -X POST http://localhost:8428/snapshot/create | jq -r '.snapshot')
echo "Created snapshot: ${SNAPSHOT_PATH}"

# 2. Push to S3 using vmbackup
./vmbackup-prod \
  -storageDataPath=/var/lib/victoria-metrics-data \
  -snapshotName="${SNAPSHOT_PATH}" \
  -dst=s3://monitoring-backup/vm-snapshots/$(date +%Y%m%d)/

# 3. Verify backup integrity
./vmbackupmanager-prod verify \
  -dst=s3://monitoring-backup/vm-snapshots/$(date +%Y%m%d)/

# 4. Clean up old snapshots (keep last 7 days)
curl -X POST "http://localhost:8428/snapshot/delete?keep=7"

echo "Backup completed"

# Recovery process
./vmrestore-prod \
  -src=s3://monitoring-backup/vm-snapshots/20260711/ \
  -dst=/var/lib/victoria-metrics-data/

Grafana Integration

Data Source Configuration

# Grafana datasource configuration (Provisioning)
apiVersion: 1
datasources:
  # VictoriaMetrics as a Prometheus datasource
  - name: VictoriaMetrics
    type: prometheus
    access: proxy
    url: http://vmselect:8481/select/0/prometheus/
    isDefault: true
    jsonData:
      timeInterval: "15s"
      httpMethod: "POST"
      # Enable MetricsQL extensions
      customQueryParameters: "extra_label=cluster=production"

  # For multi-tenant setups
  - name: VictoriaMetrics (tenant-a)
    type: prometheus
    access: proxy
    url: http://vmselect:8481/select/tenant-a/prometheus/
    jsonData:
      timeInterval: "15s"

Recommended Dashboards

VictoriaMetrics provides a rich set of Grafana dashboard templates:

# Import official dashboards (Grafana → Import → ID)

Dashboard ID    Description
────────────────────────────────────
11176          VictoriaMetrics Single Node Overview
14289          VictoriaMetrics Cluster Overview
14592          vmagent Status Overview
14594          vmalert Rule Execution Overview
14595          vmrestore Backup Overview
14596          vmbackup Backup Status

Production Best Practices

Capacity Planning

Based on VictoriaMetrics official documentation and community experience:

Scale	Active Time Series	Ingestion Rate	Single-Node Config	Cluster Config
Small	<1M	<20K samples/s	4C/8GB/100GB SSD	Not needed
Medium	1-5M	20K-100K	8C/16GB/500GB SSD	3 storage × 8C/16GB
Large	5-20M	100K-500K	Not recommended	3-5 storage × 16C/64GB
Extra Large	>20M	>500K	Not recommended	5-10 storage × 32C/128GB

Storage Capacity Estimation Formula:

#!/usr/bin/env python3
"""
VictoriaMetrics Storage Capacity Estimation Tool
Estimates based on active time series count, retention period, and scrape interval
"""

def estimate_storage(
    active_series: int,
    retention_days: int,
    scrape_interval_sec: int = 15,
    avg_label_size: int = 100,  # Average label bytes per series
):
    """
    Estimate disk space required for VictoriaMetrics
    
    Parameters:
        active_series: Number of active time series
        retention_days: Data retention period in days
        scrape_interval_sec: Scrape interval in seconds
        avg_label_size: Average label size in bytes
    """
    # Size per data point (approx 1-2 bytes after VM compression)
    bytes_per_point = 1.5
    
    # Calculate total data points
    points_per_series_per_day = 86400 / scrape_interval_sec
    total_points = active_series * points_per_series_per_day * retention_days
    
    # Data point storage
    data_storage = total_points * bytes_per_point
    
    # Index storage (approx 20-30% of data storage)
    index_storage = data_storage * 0.25
    
    # Label storage
    label_storage = active_series * avg_label_size * 2  # After compression
    
    # Total storage
    total_storage = data_storage + index_storage + label_storage
    
    # Additional overhead (WAL, temp files, etc., approx 10%)
    total_with_overhead = total_storage * 1.1
    
    return {
        "active_series": active_series,
        "retention_days": retention_days,
        "scrape_interval_sec": scrape_interval_sec,
        "total_points": int(total_points),
        "data_storage_gb": data_storage / 1024**3,
        "index_storage_gb": index_storage / 1024**3,
        "label_storage_gb": label_storage / 1024**3,
        "total_storage_gb": total_with_overhead / 1024**3,
    }

# Example calculations
configs = [
    ("Small", 100_000, 90, 15),
    ("Medium", 1_000_000, 180, 15),
    ("Large", 5_000_000, 180, 15),
    ("XLarge", 20_000_000, 365, 30),
]

print(f"{'Scale':<8} {'Series':>12} {'Days':>8} {'Storage(GB)':>12} {'Storage(TB)':>12}")
print("-" * 58)

for name, series, days, interval in configs:
    result = estimate_storage(series, days, interval)
    print(f"{name:<8} {series:>12,} {days:>8} {result['total_storage_gb']:>12.1f} {result['total_storage_gb']/1024:>12.2f}")

Sample output:

Scale       Series      Days   Storage(GB)   Storage(TB)
----------------------------------------------------------
Small        100,000       90          1.2          0.00
Medium      1,000,000      180         25.6          0.02
Large       5,000,000      180        128.0          0.13
XLarge     20,000,000      365        768.0          0.80

Note: The above are theoretical estimates. Actual storage consumption is significantly affected by data characteristics (label cardinality, data distribution, etc.). Testing with your own data is recommended.

Monitoring VictoriaMetrics Itself

# vmagent scrape config for VictoriaMetrics self-monitoring
scrape_configs:
  - job_name: "victoria-metrics"
    static_configs:
      - targets: ["victoria-metrics:8428"]
    # VM exposes its own metrics on the /metrics endpoint

  - job_name: "vmstorage"
    static_configs:
      - targets: ["vmstorage-1:8482", "vmstorage-2:8482"]

  - job_name: "vminsert"
    static_configs:
      - targets: ["vminsert-1:8480", "vminsert-2:8480"]

  - job_name: "vmselect"
    static_configs:
      - targets: ["vmselect-1:8481", "vmselect-2:8481"]

Key Alert Rules:

# vmalert rules — VictoriaMetrics self-health monitoring
groups:
  - name: victoriametrics-alerts
    rules:
      # vmstorage disk usage
      - alert: VMStorageDiskUsageHigh
        expr: |
          100 * (1 - vm_data_disk_free_bytes / vm_data_disk_total_bytes) > 85          
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "VM storage disk usage exceeds 85%"
          description: "Node {{ $labels.instance }} disk usage: {{ $value }}%"

      # Ingestion rate anomaly
      - alert: VMIngestionRateDrop
        expr: |
          rate(vm_rows_ingested_total[5m]) < 100          
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "VM data ingestion rate dropped abnormally"
          description: "Current ingestion rate: {{ $value }} rows/s"

      # High query latency
      - alert: VMSlowQueries
        expr: |
          histogram_quantile(0.95, 
            rate(vm_select_query_duration_seconds_bucket[5m])
          ) > 10          
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "VM query P95 latency exceeds 10 seconds"
          description: "P95 latency: {{ $value }}s"

      # High memory usage
      - alert: VMHighMemoryUsage
        expr: |
          100 * vm_memory_bytes / vm_memory_allowed_bytes > 90          
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "VM memory usage exceeds 90%"
          description: "Memory usage: {{ $value }}%"

      # Node unreachable
      - alert: VMNodeDown
        expr: up{job=~"victoria-metrics|vmstorage|vminsert|vmselect"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "VM node unreachable"
          description: "{{ $labels.instance }} is unreachable"

Thanos vs VictoriaMetrics Selection Guide

Dimension	Thanos	VictoriaMetrics
Architecture Complexity	High (6+ components)	Low (1-3 components)
External Dependencies	Object storage (S3/MinIO)	None
Compression Ratio	Moderate (2-3x)	High (5-7x)
Query Performance	Moderate	High
Operational Cost	High	Low
Global Queries	Native support	Native support
Downsampling	Compactor component	Built-in
High Availability	Sidecar + Receiver	Cluster replication
Ecosystem Compatibility	Fully Prometheus compatible	Fully compatible + MetricsQL extensions
Learning Curve	Steep	Gentle
Suitable Scale	Medium to large	Small to extra large

Selection Recommendations:

Choose VictoriaMetrics: Want minimal operations, pursue high compression and query performance, no object storage infrastructure, limited team size
Choose Thanos: Already have object storage infrastructure, want to use S3 for cold data archival, need deep integration with existing Prometheus, experienced with object storage ecosystem
Hybrid approach: VictoriaMetrics for hot data (recent 3 months), Thanos for cold data archival (3+ months) to S3

Reference comparison: VictoriaMetrics vs Thanos, Community selection discussion

Summary

VictoriaMetrics, as a Prometheus long-term storage solution, offers significant advantages in compression ratio, query performance, and operational complexity. Key takeaways for choosing and implementing VM:

Start with single-node: For most small-to-medium scenarios, VM single-node is powerful enough. A single instance can handle 1 million active time series and 2 million samples per second ingestion. No need to start with the cluster version.
vmagent is a powerful collection tool: Using vmagent instead of Prometheus for data collection provides sharding, replication, and protocol conversion capabilities while being fully compatible with Prometheus configuration format.
Replication ensures availability: Configure replicationFactor=2 in the cluster version for data redundancy. Single-node failure does not affect reads or writes. Combined with vmbackup for off-site backup, a complete disaster recovery solution is achieved.
Govern high cardinality first: No storage solution can withstand unlimited label explosion. Before migrating to VM, first govern high-cardinality metrics — this is the foundation of a healthy monitoring system.
Leverage MetricsQL: MetricsQL’s downsampling, lag handling, and other extension functions can optimize query performance without modifying collection configurations.
Monitor the monitoring system: VictoriaMetrics’s own health needs monitoring too. Disk usage, ingestion rate, query latency, and memory usage are the four core metrics.
Clear migration path: Dual-write transition → historical data migration → datasource switch → decommission old Prometheus. Each step is verifiable, and risk is controllable.

Reference Documentation:
VictoriaMetrics Official Documentation — https://docs.victoriametrics.com/
VictoriaMetrics GitHub — https://github.com/VictoriaMetrics/VictoriaMetrics
VictoriaMetrics FAQ — https://docs.victoriametrics.com/FAQ.html
VictoriaMetrics Operator — https://docs.victoriametrics.com/operator/

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

VictoriaMetrics Operator Documentation — VictoriaMetrics, referenced for VictoriaMetrics Operator Documentation
VictoriaMetrics vs Thanos — VictoriaMetrics, referenced for VictoriaMetrics vs Thanos
Community selection discussion — CSDN, referenced for Community selection discussion
docs.victoriametrics.com — VictoriaMetrics, referenced for technical content
github.com — GitHub, referenced for VictoriaMetrics
docs.victoriametrics.com — VictoriaMetrics, referenced for FAQ.html

Overview#

Why VictoriaMetrics#

Prometheus Storage Bottlenecks#

Core Advantages of VictoriaMetrics#

Architecture Design Deep Dive#

Storage Engine#

Cluster Architecture Components#

Deployment Practices#

Single-Node Deployment#

Cluster Deployment#

Kubernetes Deployment (VictoriaMetrics Operator)#

Data Collection and Migration#

Using vmagent to Replace Prometheus Scraping#

Migrating Historical Data from Prometheus#

Performance Tuning#

Write Performance Optimization#

Query Performance Optimization#

Storage Optimization#

High Availability Architecture Design#

Replication and Disaster Recovery#

Backup and Recovery#

Grafana Integration#

Data Source Configuration#

Recommended Dashboards#

Production Best Practices#

Capacity Planning#

Monitoring VictoriaMetrics Itself#

Thanos vs VictoriaMetrics Selection Guide#

Summary#

References & Acknowledgments#