Elasticsearch + Kibana Log Analysis Platform

Overview

Among the three pillars of observability (Metrics, Logs, Traces), logs are the data closest to the application layer. When an online service misbehaves, the first instinct is usually “check the logs.” The ELK Stack (Elasticsearch + Logstash + Kibana) is the de facto standard in log analysis, offering powerful capabilities in full-text search, log parsing, and visual analytics.

With the rise of “lightweight” log solutions like Loki, ELK faces criticism for “high storage costs and operational complexity.” However, ELK’s capabilities in full-text search, complex text analysis, and structured log aggregation remain irreplaceable. This article systematically covers ELK log analysis platform construction practices — from architecture principles to deployment configuration — and provides a comparative analysis with Loki to help you decide when to choose ELK and when to choose Loki.

Reference: Elastic Official Documentation

I. ELK Stack Architecture

1.1 Overall Architecture

┌──────────────────────────────────────────────────────────────┐
│                    ELK Stack Architecture                      │
│                                                              │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐                     │
│  │ App/Node│  │ App/Node│  │ App/Node│  ← Log sources       │
│  │ log file│  │ log file│  │ log file│                     │
│  └────┬────┘  └────┬────┘  └────┬────┘                     │
│       │            │            │                           │
│  ┌────┴────┐  ┌────┴────┐  ┌────┴────┐                     │
│  │Filebeat │  │Filebeat │  │Filebeat │  ← Lightweight shipper │
│  └────┬────┘  └────┬────┘  └────┬────┘                     │
│       │            │            │                           │
│  ┌────┴────┐  ┌────┴────┐  ┌────┴────┐                     │
│  │Logstash │  │Logstash │  │Logstash │  ← Parsing/transform   │
│  └────┬────┘  └────┬────┘  └────┬────┘                     │
│       │            │            │                           │
│  └──────────────────┬───────────────────┘                   │
│                     │                                        │
│                     ▼                                        │
│             ┌────────────┐                                   │
│             │Elasticsearch│  ← Storage + search               │
│             └─────┬──────┘                                   │
│                   │                                          │
│             ┌─────┴──────┐                                   │
│             │   Kibana   │  ← Visualization + querying       │
│             └────────────┘                                   │
└──────────────────────────────────────────────────────────────┘

1.2 Component Responsibilities

Component	Responsibility	Language	Characteristics
Filebeat	Log collection	Go	Lightweight, low resource consumption, replaces legacy Logstash collection
Logstash	Log parsing/transformation	JRuby	Powerful, but high memory consumption
Elasticsearch	Storage + search	Java	Full-text search engine, distributed
Kibana	Visualization	Node.js	Querying, dashboards, alerting

1.3 Simplified Architecture: Filebeat → Elasticsearch

In modern ELK architectures, Logstash is often streamlined. Filebeat has built-in processing capabilities (Ingest Node) and can write directly to Elasticsearch:

Log file → Filebeat (collection + basic processing) → Elasticsearch (Ingest Pipeline) → Kibana

Logstash is only needed when complex parsing (such as Grok multiline parsing, multi-source data enrichment) is required.

II. Elasticsearch Index Management

2.1 Index Design

An index in Elasticsearch is similar to a table in a database. Log data is typically indexed by date:

Index naming: logs-app-2026.07.10
              │     │    └─ Date (one index per day)
              │     └────── Application name
              └──────────── Prefix

Advantages of date-based indexing:

Easy time-range queries (only relevant indices are queried)
Facilitates ILM (Index Lifecycle Management)
Deleting old data is as simple as deleting entire indices

2.2 Index Templates

An Index Template defines the mapping and settings for an index, automatically applied to new indices matching a name pattern:

PUT _index_template/logs-app
{
  "index_patterns": ["logs-app-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.refresh_interval": "5s",
      "index.lifecycle.name": "logs-ilm-policy",
      "index.lifecycle.rollover_alias": "logs-app"
    },
    "mappings": {
      "properties": {
        "@timestamp":    { "type": "date" },
        "level":         { "type": "keyword" },
        "service":       { "type": "keyword" },
        "host":          { "type": "keyword" },
        "message":       { "type": "text", "analyzer": "standard" },
        "request_id":    { "type": "keyword" },
        "duration_ms":   { "type": "integer" },
        "status_code":   { "type": "integer" },
        "url":           { "type": "keyword" }
      }
    }
  },
  "priority": 100
}

Field type selection principles:

Type	Use Case	Notes
`keyword`	Exact match, aggregation, sorting	Not analyzed, e.g., service name, level
`text`	Full-text search	Analyzed (tokenized), e.g., message
`date`	Time fields	Supports time-range queries
`integer/long`	Numeric values	Supports range queries and aggregation
`ip`	IP addresses	Supports CIDR queries
`object`	Nested JSON	Default type
`flattened`	Dynamic JSON	Reduces field explosion

Key recommendation: Set fields that need exact matching and aggregation as keyword, and only set fields that need full-text search as text. Incorrectly setting high-cardinality fields as text causes severe index bloat.

2.3 Index Lifecycle Management (ILM)

ILM automatically manages the full lifecycle of an index from creation to deletion:

PUT _ilm/policy/logs-ilm-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_primary_shard_size": "50gb"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Phase	Time	Action	Purpose
Hot	0-7 days	Rollover (create new index)	High-performance write and query
Warm	7-30 days	Shrink + Force Merge	Reduce resource usage
Cold	30-90 days	Freeze (freeze index)	Save memory, slower queries
Delete	> 90 days	Delete (delete index)	Free disk space

2.4 Sharding Strategy

Shard count directly impacts query performance and resource consumption:

// View index shard distribution
GET _cat/shards/logs-app-*?v

// View shard sizes
GET _cat/indices/logs-app-*?v&h=index,pri,rep,docs.count,store.size,pri.store.size

Shard design principles:

Keep each shard between 30-50 GB
Shard count = estimated log volume / 50GB
Replica count: at least 1 for production
No more than 20 shards per GB of heap memory (e.g., 32GB heap → max 640 shards)

III. Filebeat Log Collection

3.1 Filebeat Architecture

┌──────────────────────────────────────────────┐
│                  Filebeat                     │
│                                              │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐     │
│  │ Input   │  │ Input   │  │ Input   │     │
│  │ (log)   │  │ (stdin) │  │ (tcp)   │     │
│  └────┬────┘  └────┬────┘  └────┬────┘     │
│       │            │            │           │
│       ▼            ▼            ▼           │
│  ┌──────────────────────────────────┐       │
│  │         Harvester                 │       │
│  │  One per file, reads line by line│       │
│  └──────────────┬──────────────────┘       │
│                 │                            │
│                 ▼                            │
│  ┌──────────────────────────────────┐       │
│  │          Spooler (event pool)     │       │
│  │  Aggregates events, batch send    │       │
│  └──────────────┬──────────────────┘       │
│                 │                            │
│                 ▼                            │
│  ┌──────────────────────────────────┐       │
│  │         Output                    │       │
│  │  Elasticsearch / Logstash / etc │       │
│  └──────────────────────────────────┘       │
└──────────────────────────────────────────────┘

3.2 Filebeat Configuration

# filebeat.yml
filebeat.inputs:
  # Collect Nginx access logs
  - type: log
    enabled: true
    paths:
      - /var/log/nginx/access.log
    fields:
      service: nginx
      env: production
    fields_under_root: true
    multiline:
      pattern: '^\d{4}-\d{2}-\d{2}'   # Match date at line start
      negate: true
      match: after                     # Append non-matching lines to previous

  # Collect application JSON logs
  - type: log
    enabled: true
    paths:
      - /var/log/app/*.json
    fields:
      service: my-app
      env: production
    json.keys_under_root: true        # Promote JSON fields to top level
    json.add_error_key: true          # Add error field on parse failure
    json.message_key: message         # Specify message field for multiline

  # Collect container logs (Docker)
  - type: container
    enabled: true
    paths:
      - /var/lib/docker/containers/*/*.log
    stream: all
    cri: parse

# Output to Elasticsearch
output.elasticsearch:
  hosts: ["es-01:9200", "es-02:9200", "es-03:9200"]
  index: "logs-%{[service]}-%{+yyyy.MM.dd}"
  username: "elastic"
  password: "${ES_PASSWORD}"
  ssl.certificate_authority: ["/etc/filebeat/ca.crt"]

# Index template
setup.template:
  name: "logs-app"
  pattern: "logs-*-%{+yyyy.MM.dd}*"
  enabled: true

# Kibana dashboard (optional)
setup.kibana:
  host: "kibana:5601"

# Processors
processors:
  - add_host_metadata: ~          # Add host metadata
  - add_cloud_metadata: ~         # Add cloud metadata
  - add_docker_metadata: ~        # Add Docker metadata
  - drop_fields:
      fields: ["agent.ephemeral_id", "agent.id", "agent.type", "agent.version"]
      ignore_missing: true

# Tuning
queue.mem:
  events: 4096                     # Memory queue event count
  flush.min_events: 2048          # Minimum batch size
  flush.timeout: 1s               # Batch timeout

logging.level: info
logging.to_files: true

3.3 Multiline Log Handling

Java exception stack traces are the most common multiline log scenario:

multiline:
  # Match lines starting with timestamp as new log entry
  pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}'
  negate: true
  match: after
  timeout: 5s    # Force send current multiline event after timeout

Processing result:

Original logs:
2026-07-10 10:00:00 ERROR NullPointerException
    at com.example.Service.handle(Service.java:45)
    at com.example.Controller.process(Controller.java:23)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

After merging:
2026-07-10 10:00:00 ERROR NullPointerException
    at com.example.Service.handle(Service.java:45)
    at com.example.Controller.process(Controller.java:23)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

IV. Logstash Log Parsing

4.1 Logstash Pipeline

When Filebeat’s Ingest capabilities are insufficient for complex logs, Logstash is introduced for deep parsing:

Filebeat → Logstash (Input → Filter → Output) → Elasticsearch

# logstash.conf
input {
  beats {
    port => 5044
  }
}

filter {
  # Parse Nginx access logs
  if [service] == "nginx" {
    grok {
      match => {
        "message" => '%{IPORHOST:client_ip} - %{DATA:user} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATHPARAM:url} HTTP/%{NUMBER:http_version}" %{NUMBER:status_code} %{NUMBER:bytes} "%{DATA:referrer}" "%{DATA:user_agent}" rt=%{NUMBER:request_time}'
      }
      overwrite => ["message"]
    }
    # Extract browser/OS from User-Agent
    useragent {
      source => "user_agent"
      target => "ua"
    }
  }

  # Parse JSON-format application logs
  if [service] == "my-app" {
    json {
      source => "message"
      target => "app"
    }
    # Convert field types
    mutate {
      convert => {
        "[app][duration_ms]" => "integer"
        "[app][status_code]" => "integer"
      }
    }
  }

  # Common processing
  date {
    match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
    target => "@timestamp"
  }

  # GeoIP parsing (extract geolocation from IP)
  geoip {
    source => "client_ip"
    target => "geo"
  }

  # Remove unnecessary fields
  mutate {
    remove_field => ["user", "agent", "ecs", "input", "log"]
  }
}

output {
  elasticsearch {
    hosts => ["es-01:9200", "es-02:9200", "es-03:9200"]
    index => "logs-%{[service]}-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "${ES_PASSWORD}"
    ssl_certificate_verification => false
  }
}

4.2 Grok Patterns

Grok is Logstash’s most powerful log parsing tool — essentially predefined named regex patterns:

# Common Grok patterns
%{IP:ip}              # Match IP address
%{WORD:method}        # Match a word
%{NUMBER:status}      # Match a number
%{HTTPDATE:timestamp} # Match HTTP date format
%{IPORHOST:host}      # Match IP or hostname
%{DATA:path}          # Match non-greedy data
%{GREEDYDATA:message} # Match greedy data

Custom Grok patterns:

# Define in patterns/ directory
# nginx_patterns
NGINX_ACCESS %{IPORHOST:client_ip} - %{DATA:user} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATHPARAM:url} HTTP/%{NUMBER:http_version}" %{NUMBER:status_code} %{NUMBER:bytes}

# Use in filter
grok {
  patterns_dir => ["/etc/logstash/patterns"]
  match => { "message" => "%{NGINX_ACCESS}" }
}

4.3 Ingest Pipeline: Replacing Logstash

Elasticsearch 5.0+ introduced Ingest Node, enabling log processing within ES itself without Logstash:

PUT _ingest/pipeline/logs-pipeline
{
  "description": "Log parsing pipeline",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{IPORHOST:client_ip} %{WORD:method} %{URIPATHPARAM:url} %{NUMBER:status_code} %{NUMBER:duration_ms}"
        ]
      }
    },
    {
      "convert": {
        "field": "status_code",
        "type": "integer"
      }
    },
    {
      "convert": {
        "field": "duration_ms",
        "type": "integer"
      }
    },
    {
      "geoip": {
        "field": "client_ip",
        "target_field": "geo"
      }
    },
    {
      "date": {
        "field": "@timestamp",
        "formats": ["ISO8601"]
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "tags",
        "value": "parse-failed"
      }
    }
  ]
}

Filebeat directly specifies the pipeline:

output.elasticsearch:
  hosts: ["es:9200"]
  pipeline: "logs-pipeline"

Recommendation: Prefer Ingest Pipeline. Only introduce Logstash when complex enrichment (e.g., database lookups) or multiple output targets are needed.

V. Kibana Visualization

5.1 Discover: Log Search

Kibana Discover is the core interface for log querying:

KQL (Kibana Query Language) query examples:

# Exact match
service: "nginx" and level: "ERROR"

# Full-text search
message: "NullPointerException"

# Range query
status_code >= 500 and status_code < 600

# Time range
@timestamp >= "2026-07-10T00:00:00" and @timestamp < "2026-07-11T00:00:00"

# Combined query
service: "my-app" and (level: "ERROR" or level: "WARN") and duration_ms > 1000

5.2 Dashboard: Dashboards

Create common log analysis dashboards:

Visualization Type	Use Case	Example
Line Chart	Time trends	Request volume/error rate over time
Pie Chart	Distribution	By service/status code
Data Table	Detail list	Top 20 slow requests
Metric	Key numbers	Today’s total requests, error rate
Tile Map	Geographic	Access source IP distribution
Tag Cloud	Keywords	High-frequency keywords in logs
Gauge	Dashboard	Real-time error rate

5.3 Kibana Alerting

Kibana 7.x+ has built-in alerting functionality, allowing alert rules based on ES queries:

POST /api/alerts/rule
{
  "name": "High Error Rate",
  "consumer": "alerts",
  "rule_type_id": ".es-query",
  "params": {
    "query": [
      {
        "filter": {
          "bool": {
            "filter": [
              { "term": { "level": "ERROR" } }
            ]
          }
        },
        "timeWindowSize": 300,
        "timeWindowUnit": "s"
      }
    ],
    "size": 100,
    "threshold": [
      { "comparator": ">", "threshold": [10] }
    ],
    "index": ["logs-*"]
  },
  "actions": [
    {
      "id": "webhook-action",
      "params": { "message": "More than 10 error logs in the past 5 minutes" }
    }
  ]
}

VI. Performance Optimization

6.1 Elasticsearch Performance Tuning

JVM Heap:

# jvm.options
-Xms31g              # Initial heap = max heap, avoid dynamic resizing
-Xmx31g              # No more than 50% of physical memory, leave half for Lucene file cache
-XX:+UseG1GC         # Use G1 garbage collector
-XX:MaxGCPauseMillis=200

Index refresh interval:

// Reduce refresh frequency during write-intensive periods
PUT logs-app-*/_settings
{
  "index.refresh_interval": "30s"  // Default 1s, change to 30s to reduce write pressure
}

Bulk writes:

POST /_bulk
{ "index": { "_index": "logs-app-2026.07.10" } }
{ "@timestamp": "2026-07-10T10:00:00Z", "level": "INFO", "message": "..." }
{ "index": { "_index": "logs-app-2026.07.10" } }
{ "@timestamp": "2026-07-10T10:00:01Z", "level": "ERROR", "message": "..." }

Bulk writes are 10-100x more efficient than single writes. Recommended batch size: 5-15 MB.

Shards and replicas:

Scenario	Shards	Replicas
Daily logs < 5 GB	1	1
Daily logs 5-50 GB	2-3	1
Daily logs 50-200 GB	5-10	1
Daily logs > 200 GB	10+ or hot-warm-cold architecture	1-2

6.2 Filebeat Performance Tuning

# filebeat.yml tuning
queue.mem:
  events: 8192              # Increase queue
  flush.min_events: 4096    # Increase batch size
  flush.timeout: 1s

output.elasticsearch:
  worker: 4                 # Concurrent write workers
  bulk_max_size: 2048      # Max documents per batch

# Adjust harvester count
filebeat.inputs:
  - type: log
    paths: ["/var/log/app/*.log"]
    harvester_buffer_size: 16384    # Read buffer
    max_bytes: 10485760              # Max bytes per line (10MB)

6.3 Storage Optimization

Optimization	Effect	Notes
Force Merge	Reduces segment count	Merge to 1 segment in warm phase
Shrink	Reduces shard count	Shrink to 1 shard in warm phase
Freeze	Saves memory	Freeze index in cold phase
Best Compression	Reduces storage	Uses DEFLATE compression
Drop unnecessary fields	Reduces storage	Exclude unused fields in mapping

// Use best_compression
PUT logs-app-*/_settings
{
  "index": {
    "codec": "best_compression"
  }
}

VII. ELK vs Loki Comparison

7.1 Design Philosophy Comparison

Dimension	ELK (Elasticsearch)	Loki
Indexing method	Full-text inverted index	Only indexes labels, content is not indexed
Storage cost	High (index bloat 3-5x)	Low (label index + compressed content)
Query capability	Full-text search, complex aggregation	Label filtering + regex matching
Query language	KQL / Lucene	LogQL
Resource consumption	High (JVM, memory-intensive)	Low (Go, memory-friendly)
Deployment complexity	High (ES cluster + JVM tuning)	Low-Medium
Use case	Full-text search, complex analysis	Log monitoring, troubleshooting
Ecosystem integration	Ships with Kibana	Grafana ecosystem

7.2 Cost Comparison

Assumption: 100GB logs per day, 30-day retention

ELK:
  Storage: 100GB × 3-5 (index bloat) × 30 days = 9-15 TB
  Servers: 3-5 ES nodes (32GB RAM, 4TB SSD each)
  Monthly cost: ~$2,000-4,000

Loki:
  Storage: 100GB × 0.1 (label+compression only) × 30 days = 300 GB
  Servers: 1-2 Loki nodes + S3 object storage
  Monthly cost: ~$200-500

7.3 When to Choose ELK

Need full-text search capability (searching for any keyword in log content)
Need complex aggregation analysis (cross-dimensional statistics)
Need deep analysis of structured logs (e.g., API request analysis)
Logs contain large amounts of text content requiring tokenized search
Team already has ES operations experience

7.4 When to Choose Loki

Primary need is log monitoring and troubleshooting
Need integration with Grafana / Prometheus
Sensitive to storage costs
High log volume but infrequent querying
Team prefers lightweight solutions

Recommendation: If unsure, start with Loki. Loki meets 80% of log scenario needs at 1/10 the cost of ELK. Introduce ELK when full-text search requirements are clearly identified.

VIII. Production Deployment Practices

8.1 Cluster Topology

┌─── Production ELK Cluster ──────────────────────┐
│                                                │
│  Hot Nodes (3 × 64GB RAM, 4TB NVMe SSD)       │
│  ├── Recent 7-day indices                      │
│  └── High write and query throughput           │
│                                                │
│  Warm Nodes (2 × 32GB RAM, 8TB HDD)           │
│  ├── 7-30 day indices                          │
│  └── Read-only, low query frequency            │
│                                                │
│  Cold Nodes (1 × 16GB RAM, 16TB HDD)           │
│  ├── 30-90 day indices (frozen state)          │
│  └── Occasional queries                        │
│                                                │
│  Coordinator Nodes (2 × 8GB RAM)               │
│  └── Query routing + aggregation               │
│                                                │
│  Kibana (2 × 4GB RAM)                         │
│  └── Load balanced                             │
└────────────────────────────────────────────────┘

8.2 Docker Compose Deployment

version: '3.8'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.14.0
    environment:
      - discovery.type=single-node
      - ES_JAVA_OPTS=-Xms2g -Xmx2g
      - xpack.security.enabled=false
      - cluster.name=elk-cluster
      - bootstrap.memory_lock=true
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - es_data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"

  kibana:
    image: docker.elastic.co/kibana/kibana:8.14.0
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

  filebeat:
    image: docker.elastic.co/beats/filebeat:8.14.0
    user: root
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    depends_on:
      - elasticsearch

volumes:
  es_data:

8.3 Monitoring ELK Itself

# Prometheus scraping ES metrics
scrape_configs:
  - job_name: 'elasticsearch'
    static_configs:
      - targets: ['es-01:9208']
    metrics_path: /metrics

Key alerting rules:

groups:
  - name: elasticsearch
    rules:
      - alert: ElasticsearchClusterHealthRed
        expr: elasticsearch_cluster_health_status{color="red"} == 1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "ES cluster status is RED"

      - alert: ElasticsearchDiskSpaceLow
        expr: |
          1 - (elasticsearch_filesystem_data_available_bytes /
               elasticsearch_filesystem_data_size_bytes) > 0.85          
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "ES disk space low: {{ $labels.instance }}"

      - alert: ElasticsearchJVMHeapHigh
        expr: |
          elasticsearch_jvm_memory_used_bytes{area="heap"} /
          elasticsearch_jvm_memory_max_bytes{area="heap"} > 0.85          
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "ES JVM heap usage too high: {{ $labels.instance }}"

Summary

The ELK Stack, after years of development, remains the most comprehensive solution in the log analysis space:

Elasticsearch’s full-text search capability is irreplaceable — when you need to search for any keyword in log content, ELK is the only choice
Index management is ELK’s core — proper mapping, ILM policies, and shard planning directly determine cluster performance and cost
Filebeat + Ingest Pipeline is the recommended collection architecture for modern ELK — lighter than traditional Logstash; only complex parsing requires Logstash
Performance optimization requires systematic tuning — JVM heap, refresh interval, bulk writes, shard strategy, and compression algorithm are all indispensable
Cost is ELK’s main weakness — full-text indexing causes 3-5x storage bloat, making it significantly more expensive than Loki at scale
Complementary to Loki, not mutually exclusive — use ELK for full-text search, Loki for log monitoring; both can coexist in the same Grafana

Choosing ELK or Loki depends on your core requirement: “full-text search” or “log monitoring.” The former calls for ELK, the latter for Loki. When both are needed, deploy both in a hybrid setup.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

Elastic Official Documentation — Elastic, referenced for Elastic Official Documentation

Overview#

I. ELK Stack Architecture#

1.1 Overall Architecture#

1.2 Component Responsibilities#

1.3 Simplified Architecture: Filebeat → Elasticsearch#

II. Elasticsearch Index Management#

2.1 Index Design#

2.2 Index Templates#

2.3 Index Lifecycle Management (ILM)#

2.4 Sharding Strategy#

III. Filebeat Log Collection#

3.1 Filebeat Architecture#

3.2 Filebeat Configuration#

3.3 Multiline Log Handling#

IV. Logstash Log Parsing#

4.1 Logstash Pipeline#

4.2 Grok Patterns#

4.3 Ingest Pipeline: Replacing Logstash#

V. Kibana Visualization#

5.1 Discover: Log Search#

5.2 Dashboard: Dashboards#

5.3 Kibana Alerting#

VI. Performance Optimization#

6.1 Elasticsearch Performance Tuning#

6.2 Filebeat Performance Tuning#

6.3 Storage Optimization#

VII. ELK vs Loki Comparison#

7.1 Design Philosophy Comparison#

7.2 Cost Comparison#

7.3 When to Choose ELK#

7.4 When to Choose Loki#

VIII. Production Deployment Practices#

8.1 Cluster Topology#

8.2 Docker Compose Deployment#

8.3 Monitoring ELK Itself#

Summary#

References & Acknowledgments#