Log Monitoring System: Loki + Promtail Deployment

Why Choose Loki

The traditional ELK (Elasticsearch + Logstash + Kibana) stack is powerful, but has two core pain points:

High storage costs: Elasticsearch fully indexes log content, with index bloat reaching 3-5x the raw data
Operational complexity: ES cluster scaling, shard rebalancing, and index lifecycle management are complex, making production clusters costly to maintain

Loki, open-sourced by Grafana Labs, is designed around the philosophy of “doing for logs what Prometheus did for metrics.” It only indexes log labels, not the log content itself, and performs full-text search via LogQL. This design reduces storage costs by more than 10x.

This article is based on Loki 3.x. See Loki Official Documentation

Loki vs ELK Comparison

Dimension	ELK (Elasticsearch)	Loki
Indexing method	Full-text inverted index	Only indexes labels, content is not indexed
Storage cost	High (index bloat 3-5x)	Low (label index + compressed content)
Query language	Lucene Query / KQL	LogQL (PromQL-like syntax)
Scalability	Horizontal scaling, complex sharding	Microservices mode, independently scalable components
Use cases	Full-text search, complex analysis	Log monitoring, metric-style queries, Grafana integration
Resource consumption	High (JVM, memory-intensive)	Low (written in Go, memory-friendly)

Loki is not intended to fully replace ES. If your core requirement is full-text search and complex text analysis, ES remains the better choice. However, for SRE log monitoring, metric-based alerting, and troubleshooting scenarios, the Loki + Grafana combination offers significant advantages in both cost and efficiency.

Loki Architecture Principles

Log Storage Model

Loki’s core design: label indexing + compressed log streams.

┌─────────────────────────────────────────────┐
│              Label Index (inverted)           │
│  app=nginx  →  stream_id_1, stream_id_2     │
│  env=prod   →  stream_id_1, stream_id_3     │
└──────────────────┬──────────────────────────┘
                   │ After finding stream_id
                   ▼
┌─────────────────────────────────────────────┐
│           Chunk Store (compressed chunks)     │
│  stream_id_1: [ts] log line 1               │
│              [ts] log line 2               │
│              ... compressed (Snappy/ZSTD)     │
└─────────────────────────────────────────────┘

Stream: Defined by a unique set of labels, e.g., {app="nginx", env="prod"}. Logs within the same stream are appended in chronological order.
Chunk: Logs in a stream are split into fixed-size chunks (default 1MB or 1h), compressed, and stored in object storage (S3/GCS/local filesystem).
Label Index: Only indexes the mapping from labels to streams, resulting in a very small footprint.

This design means the index data volume is proportional to label cardinality, not log volume — even with 1TB of logs per day, the index size barely grows as long as labels remain unchanged.

Microservices Architecture Components

                    ┌──────────┐
   Promtail ──────► │ Distributor │ ── write ──► Ingester ──► Chunk Store (S3)
                    └──────────┘                    │
                                                    ▼
   Grafana ────►  ┌──────────┐                Query Frontend
                   │  Querier  │ ◄─── read ──── Ingester + S3
                   └──────────┘
                        │
                   ┌──────────┐
                   │ Compactor │ ─── merge/expire cleanup
                   └──────────┘

Component	Responsibility
Distributor	Receives log writes, validates labels, distributes to Ingesters by stream hash
Ingester	Caches recently written logs, flushes full chunks to object storage
Querier	Handles query requests, checks Ingester cache first, then object storage
Query Frontend	Query preprocessing: splits large queries, parallelizes, caches results
Compactor	Merges chunks, executes log retention policies (TTL)

Single Binary mode packages all components into one process, suitable for development and small-scale deployments. Microservices mode deploys each component independently, suitable for large-scale production environments.

Promtail Configuration

Promtail is Loki’s log collection agent, similar in function to Logstash/Filebeat: it collects, parses, labels, and pushes logs to Loki.

Basic Configuration

# promtail-config.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml  # Records the read position for each log file

clients:
  - url: http://loki:3100/loki/api/v1/push
    tenant_id: default           # Distinguishes tenants in multi-tenant scenarios

scrape_configs:
  # Collect Nginx access logs
  - job_name: nginx
    static_configs:
      - targets:
          - localhost
        labels:
          job: nginx
          app: nginx
          env: prod
          __path__: /var/log/nginx/*.log

  # Collect Docker container logs (labeled by container name)
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
        filters:
          - name: label
            values: ["logging=loki"]
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: container_name
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: stream
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: service

  # Collect Kubernetes Pod logs
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
      - source_labels: [__meta_kubernetes_pod_container_name]
        target_label: container

Log Parsing and Label Extraction

Use pipeline_stages to parse raw log lines and extract structured fields as labels or structured data within logs:

scrape_configs:
  - job_name: app-json-logs
    static_configs:
      - targets: [localhost]
        labels:
          job: app
          app: my-service
          __path__: /var/log/app/*.log
    pipeline_stages:
      # 1. Parse JSON-format logs
      - json:
          expressions:
            level: level
            trace_id: trace_id
            method: http.method
            status: http.status_code
            duration_ms: duration_ms

      # 2. Set extracted fields as labels
      - labels:
          level:
          status:
          method:

      # 3. Keep only level=error and status>=500 logs
      - match:
          selector: '{job="app"} |~ "ERROR"'
          action: keep

      # 4. Convert duration field to numeric
      - template:
          source: duration_ms
          template: '{{ .ToFloat .duration_ms | printf "%.1f" }}'

      # 5. Structured output
      - output:
          source: message

For detailed Promtail pipeline configuration, see Promtail Documentation

Log Position Management

positions.yaml records the read offset for each log file. Promtail resumes from the last checkpoint after restarts. In production, place this file on a persistent volume:

positions:
  filename: /data/promtail/positions.yaml

If logs are rotated (e.g., old files deleted after logrotate), Promtail tracks files by inode and can still correctly continue reading new files.

LogQL Query Syntax

LogQL is Loki’s query language, designed with PromQL as its inspiration. It has two categories: log queries and metric queries.

Stream Selector

Use curly braces to select log streams, similar to PromQL’s label selectors:

# Exact match
{app="nginx", env="prod"}

# Regex match
{app=~"nginx|gateway|api.*"}

# Exclude label values
{namespace="default", container!="istio-proxy"}

Log Pipeline

Append pipeline operators after the stream selector to filter and parse log lines:

# Contains keyword
{app="nginx"} |= "error"

# Excludes keyword
{app="nginx"} != "timeout"

# Regex match
{app="app"} |~ "ERROR|WARN|PANIC"

# Filter by field after JSON parsing
{app="my-service"}
  | json
  | level="error"
  | status >= 500

# Log formatting (extract fields and reformat display)
{app="my-service"}
  | json
  | line_format "{{.trace_id}} [{{.level}}] {{.message}}"

# Regex extraction (non-JSON logs)
{app="nginx"}
  | regexp `(?P<method>\w+) (?P<path>\S+) (?P<status>\d+) (?P<duration>\d+)`
  | status >= 500

Metric Query

Converts log streams into time-series metrics — Loki’s most powerful feature: generating Grafana metric panels directly from logs:

# 1. Log line count rate (QPS)
sum(rate({app="nginx"}[5m])) by (status)

# 2. P99 latency after field extraction
quantile_over_time(0.99,
  {app="my-service"} | json | unwrap duration_ms [5m]
) by (method)

# 3. Error rate
sum(rate({app="my-service"} | json | level="error" [5m]))
  /
sum(rate({app="my-service"} [5m]))

# 4. Top 10 slowest requests
topk(10,
  sum by (path) (
    rate({app="my-service"} | json | unwrap duration_ms [5m])
  )
)

The unwrap keyword extracts numeric fields from logs for aggregation, supporting functions like rate, avg_over_time, quantile_over_time, sum_over_time, and more.

Grafana Integration

Adding Loki as a Data Source

In Grafana → Configuration → Data Sources, add Loki with the URL http://loki:3100. Grafana 10+ supports auto-detecting Loki version and capabilities.

Log Panels

Create a new Log Panel in a Dashboard and enter a LogQL query:

{app="my-service", env="prod"}
  | json
  | line_format "{{.trace_id}} [{{.level}}] {{.method}} {{.path}} {{.duration_ms}}ms"

Set Visualization to Logs and enable Color lines based on field (colored by the level field).

Metric Panel and Log Panel Linking

Leverage Grafana’s Dashboard variables and panel links to implement an “anomaly in metrics → one-click jump to logs” troubleshooting workflow:

Variable definition: Create a $trace_id variable that gets its value list from log queries

# Variable query
label_values({app="my-service"} | json | level="error", trace_id)

Metric panel: Display P99 latency trend

quantile_over_time(0.99,
  {app="my-service"} | json | unwrap duration_ms [5m]
) by (path)

Log panel linking: Use the variable for filtering in the log panel

{app="my-service"} | json | trace_id="$trace_id"

Panel links: Configure a jump link in the metric panel’s Data Links, passing the current time range and Trace ID to the log panel:

/d/abc123/my-dashboard?from=$__from&to=$__to&var-trace_id=$__field.trace_id

This way, when P99 latency spikes for a particular path on the metric panel, clicking that data point jumps directly to the corresponding time window’s logs.

Production Deployment

Complete docker-compose.yml

The following is a complete deployment suitable for small-to-medium production environments, including Loki (single binary mode), Promtail, Grafana, and MinIO (S3-compatible storage):

# docker-compose.yml
version: '3.8'

networks:
  loki-net:
    driver: bridge

volumes:
  minio-data:
  grafana-data:
  promtail-data:

services:
  # ===== MinIO: S3-compatible object storage =====
  minio:
    image: minio/minio:RELEASE.2024-01-01T00-00-00Z
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      - MINIO_ROOT_USER=loki
      - MINIO_ROOT_PASSWORD=loki123456
    command: server /data --console-address ":9001"
    volumes:
      - minio-data:/data
    networks: [loki-net]
    restart: unless-stopped

  # Initialize MinIO bucket
  minio-init:
    image: minio/mc:RELEASE.2024-01-01T00-00-00Z
    depends_on: [minio]
    networks: [loki-net]
    entrypoint: >
      /bin/sh -c "
      sleep 5;
      mc alias set local http://minio:9000 loki loki123456;
      mc mb local/loki-chunks --ignore-existing;
      mc mb local/loki-ruler --ignore-existing;
      mc anonymous set download local/loki-chunks;
      "      
    restart: "no"

  # ===== Loki: Log storage and query =====
  loki:
    image: grafana/loki:3.0.0
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/loki-config.yaml
    volumes:
      - ./loki-config.yaml:/etc/loki/loki-config.yaml:ro
    networks: [loki-net]
    restart: unless-stopped
    depends_on: [minio]

  # ===== Promtail: Log collection =====
  promtail:
    image: grafana/promtail:3.0.0
    ports:
      - "9080:9080"
    command: -config.file=/etc/promtail/promtail-config.yaml
    volumes:
      - ./promtail-config.yaml:/etc/promtail/promtail-config.yaml:ro
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - promtail-data:/data
    networks: [loki-net]
    restart: unless-stopped

  # ===== Grafana: Visualization =====
  grafana:
    image: grafana/grafana:11.0.0
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin123
      - GF_INSTALL_PLUGINS=grafana-loki-datasource
    volumes:
      - grafana-data:/var/lib/grafana
    networks: [loki-net]
    restart: unless-stopped
    depends_on: [loki]

Loki Configuration File

# loki-config.yaml
auth_enabled: false  # Single-tenant mode

server:
  http_listen_port: 3100

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

# Use S3-compatible storage (MinIO)
storage_config:
  aws:
    endpoint: minio:9000
    bucketnames: loki-chunks
    access_key_id: loki
    secret_access_key: loki123456
    insecure: true
    sse_encryption: false
  ruler:
    storage:
      type: local
      local:
        path: /tmp/loki/rules

# Schema config: defines the index structure for logs
schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: aws
      schema: v13
      index:
        prefix: index_
        period: 24h

# Retention policy
limits_config:
  retention_period: 720h          # Retain logs for 30 days
  max_query_series: 5000
  max_query_parallelism: 16
  ingestion_rate_mb: 10           # Max 10MB/s per tenant
  ingestion_burst_size_mb: 20
  reject_old_samples: true
  reject_old_samples_max_age: 168h # Reject logs older than 7 days

compactor:
  working_directory: /tmp/loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  delete_request_store: filesystem

ruler:
  storage:
    type: local
    local:
      path: /tmp/loki/rules
  rule_path: /tmp/loki/rules-temp
  alertmanager_url: http://alertmanager:9093
  enable_api: true

Loki Microservices Mode Deployment

For large-scale production environments, microservices mode (Distributed) is recommended, where each component scales independently. Deploy with Helm:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install loki grafana/loki-distributed \
  --namespace observability \
  --create-namespace \
  --set storage.type=s3 \
  --set storage.s3.endpoint=s3.us-east-1.amazonaws.com \
  --set storage.s3.bucket=loki-chunks-prod

In microservices mode, each component can be independently horizontally scaled:

# Key values.yaml configuration
loki:
  structuredConfig:
    ingester:
      max_chunk_age: 2h
      chunk_idle_period: 30m
      chunk_target_size: 1572864    # 1.5MB
    query_scheduler:
      max_outstanding_requests_per_tenant: 4096

  # Ingester replicas (scale up when write throughput is bottlenecked)
  ingester:
    replicas: 3

  # Querier replicas (scale up when query concurrency is bottlenecked)
  querier:
    replicas: 4

  # Query Frontend splits large queries
  queryFrontend:
    replicas: 2

S3 Storage Backend Configuration

When using AWS S3 as the storage backend in production, note the following configuration:

storage_config:
  aws:
    endpoint: s3.us-east-1.amazonaws.com
    bucketnames: loki-chunks-prod
    access_key_id: ${S3_ACCESS_KEY}     # Recommended: inject via environment variables
    secret_access_key: ${S3_SECRET_KEY}
    region: us-east-1
    insecure: false
    sse_encryption: true               # Enable SSE encryption
    http_config:
      idle_conn_timeout: 90s
      response_header_timeout: 30s

# Storage lifecycle: Use S3 Lifecycle for automatic archival
# Configure bucket lifecycle in the S3 console:
# - Transition to Glacier archive after 30 days
# - Delete after 90 days

Log Alerting

Loki has a built-in Ruler component that supports LogQL-based alerting rules:

# /tmp/loki/rules/alerts.yaml
groups:
  - name: app-alerts
    interval: 30s
    rules:
      # Error log surge
      - alert: HighErrorRate
        expr: |
          sum(rate({app="my-service"} | json | level="error" [5m]))
            by (app)
          /
          sum(rate({app="my-service"} [5m]))
            by (app)
          > 0.05          
        for: 5m
        labels:
          severity: critical
          team: sre
        annotations:
          summary: "{{ $labels.app }} error rate > 5%"
          description: "Current error rate: {{ $value }}"

      # Specific keyword alert
      - alert: OOMKilled
        expr: sum(count_over_time({container="my-service"} |~ "OOMKilled|OutOfMemoryError" [5m])) > 0
        labels:
          severity: critical
        annotations:
          summary: "Container OOMKilled detected"

Summary

Loki + Promtail + Grafana form a lightweight and efficient log monitoring system that complements Prometheus metric monitoring. Implementation recommendations:

Design labels first: Label cardinality directly impacts index size and query performance. Avoid using high-cardinality values like user_id and request_id as labels — they should be in log content and extracted via pipelines
Control label cardinality: Keep the number of distinct values per label under a few thousand. container_name and pod_name can have high cardinality in large K8s clusters and need evaluation
Set reasonable retention periods: Retain core business logs for 30-90 days, access logs for 7-14 days, controlled via limits_config.retention_period
Structure your logs: Applications should output JSON-format logs (with level, trace_id, duration_ms, etc.), which Promtail can directly parse and extract, significantly improving query efficiency
Optimize storage costs: S3 storage costs an order of magnitude less than ES. Combined with S3 Lifecycle policies (hot data in standard storage → cold data in Glacier archive → auto-delete), long-term log costs can be further compressed
Alert integration: Use Loki Ruler to turn log anomalies into alerts, unified with Prometheus Alertmanager for alert distribution

Grafana Loki official documentation: https://grafana.com/docs/loki/latest/

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

loki:3100 — loki:3100, referenced for loki:3100
[loki:3100。Grafana](http://loki:3100。Grafana) — loki:3100。Grafana, referenced for loki:3100。Grafana
Loki 官方文档 — Grafana, referenced for Loki 官方文档
Promtail 文档 — Grafana, referenced for Promtail 文档

Why Choose Loki#

Loki vs ELK Comparison#

Loki Architecture Principles#

Log Storage Model#

Microservices Architecture Components#

Promtail Configuration#

Basic Configuration#

Log Parsing and Label Extraction#

Log Position Management#

LogQL Query Syntax#

Stream Selector#

Log Pipeline#

Metric Query#

Grafana Integration#

Adding Loki as a Data Source#

Log Panels#

Metric Panel and Log Panel Linking#

Production Deployment#

Complete docker-compose.yml#

Loki Configuration File#

Loki Microservices Mode Deployment#

S3 Storage Backend Configuration#

Log Alerting#

Summary#

References & Acknowledgments#