K8s Log Collection Strategy: Choosing Between DaemonSet, Sidecar, and Agentless Patterns

Overview

Production blows up, you open Kibana to search logs, and discover the critical Pod was OOM-killed 3 minutes ago—the old logs vanished with the container. This is extremely common in K8s environments: Pods are ephemeral, and logs can’t disappear with them.

K8s log collection is fundamentally different from traditional VM environments. In a traditional setup, logs sit quietly in /var/log/ and you just SSH in to read them. In K8s, Pods can be scheduled to any node at any time, destroyed and recreated at will, with logs scattered across the entire cluster. Without centralized collection, you can’t find logs during incident response.

This article breaks down the main K8s log collection approaches: DaemonSet mode, Sidecar mode, and node Agent-based solutions. Each approach is covered with use cases, configurations, and pitfalls. Finally, a production-ready decision framework for choosing the right approach.

Three Types of K8s Logs

Before discussing collection strategies, let’s understand what types of logs exist in K8s. Different log types require different collection strategies.

1. Container Standard Output Logs

This is the most common type. Applications write logs to stdout/stderr, and the container runtime (containerd or Docker) writes these logs to the node’s /var/log/containers/ directory.

# View container log files on a node
ls /var/log/containers/

# Sample output:
# nginx-xxx_default_app-abc123.log
# redis-yyy_default_app-def456.log

# These files are actually symlinks to /var/log/pods/
ls -l /var/log/containers/nginx-xxx_default_app-abc123.log
# lrwxrwxrwx 1 root root 99 ... nginx-xxx_default_app-abc123.log -> /var/log/pods/default_nginx-xxx_abc123/app/0.log

The advantage of this type is easy collection—DaemonSet mode reads /var/log/containers/*.log directly. The disadvantage is that applications must output logs to stdout, not files.

2. In-Container File Logs

Some applications (especially Java apps) write logs to files, like /app/logs/app.log. These logs don’t appear in /var/log/containers/ and require special handling.

# Typical Java application log configuration
apiVersion: apps/v1
kind: Deployment
metadata:
  name: java-app
spec:
  template:
    spec:
      containers:
        - name: app
          image: openjdk:17
          # Application logs written to /app/logs/app.log
          volumeMounts:
            - name: logs
              mountPath: /app/logs
      volumes:
        - name: logs
          emptyDir: {}

Collecting these logs is tricky. DaemonSet mode can’t read files inside containers by default—you need Sidecar mode or have the application also output to stdout.

3. Node System Logs

System logs from the node itself, such as kubelet, container runtime, and kernel logs. These are outside the container’s scope and are typically collected through systemd journal or files on the node.

# View key system log locations on the node
# kubelet logs
journalctl -u kubelet

# Container runtime logs
journalctl -u containerd

# Kernel logs
dmesg | tail -20

Comparison of Three Log Collection Architectures

Architecture 1: DaemonSet Mode

DaemonSet mode is the most mainstream K8s log collection approach. A log collection Agent (Fluent Bit, Filebeat, etc.) runs on each node, mounts the node’s log directory, and collects logs from all containers on that node.

Think of it this way: DaemonSet is like a building’s property manager—each floor gets one, responsible for collecting trash from all rooms on that floor. Tenants don’t need to worry about how trash gets hauled out.

┌─────────────────────────────────────┐
│            Node 1                   │
│  ┌──────────┐  ┌──────────┐        │
│  │  Pod A   │  │  Pod B   │        │
│  │ stdout → │  │ stdout → │        │
│  └──────────┘  └──────────┘        │
│       │              │              │
│       ▼              ▼              │
│  /var/log/containers/*.log          │
│       │                             │
│  ┌────▼──────────┐                  │
│  │  Fluent Bit   │ ← DaemonSet     │
│  │  (Agent)       │                  │
│  └───────────────┘                  │
│       │                             │
└───────┼─────────────────────────────┘
        ▼
    Backend (ES / Loki / Kafka)

Advantages:

Low resource overhead, only one Agent per node
Non-intrusive to applications, no Pod configuration changes needed
Simple deployment and maintenance, one deployment covers the entire cluster

Disadvantages:

Cannot customize collection rules per Pod
Difficult to collect in-container file logs
Log isolation is challenging in multi-tenant scenarios

Use cases: Standard containerized applications with unified log formats outputting to stdout. This covers 90% of clusters.

Architecture 2: Sidecar Mode

Sidecar mode runs an additional log collection container in each Pod, sharing storage volumes with the application container.

┌─────────────────────────────────┐
│            Pod A                │
│  ┌──────────┐  ┌─────────────┐ │
│  │ App      │  │ Fluent Bit  │ │
│  │ writes → │←│ Collector   │ │
│  │ /app/log │  │ (Sidecar)   │ │
│  └──────────┘  └─────────────┘ │
│       shared emptyDir volume    │
└─────────────────────────────────┘
         │
         ▼
    Backend (ES / Loki / Kafka)

Sidecar has two variants:

Variant 1: Log Agent Sidecar

The collection container directly reads the application container’s log files, processes them, and sends to backend:

apiVersion: v1
kind: Pod
metadata:
  name: java-app-with-sidecar
spec:
  containers:
    - name: app
      image: openjdk:17
      command: ["java", "-jar", "/app/app.jar"]
      volumeMounts:
        - name: logs
          mountPath: /app/logs
    - name: log-collector
      image: fluent/fluent-bit:2.2.0
      volumeMounts:
        - name: logs
          mountPath: /app/logs
          readOnly: true
        - name: config
          mountPath: /fluent-bit/etc/
  volumes:
    - name: logs
      emptyDir: {}
    - name: config
      configMap:
        name: fluent-bit-sidecar-config

Variant 2: Streaming Sidecar

If the application only supports writing logs to files but you want to use DaemonSet mode to collect stdout, you can use Streaming Sidecar—it reads the application’s log files and outputs them to its own stdout, making them available for DaemonSet Agent collection:

apiVersion: v1
kind: Pod
metadata:
  name: app-with-streaming-sidecar
spec:
  containers:
    - name: app
      image: myapp:latest
      # Application logs written to /var/log/app.log
      volumeMounts:
        - name: logs
          mountPath: /var/log
    - name: streaming-sidecar
      image: busybox
      # Continuously output log file contents to stdout
      command: ["sh", "-c", "tail -n+1 -f /var/log/app.log"]
      volumeMounts:
        - name: logs
          mountPath: /var/log
  volumes:
    - name: logs
      emptyDir: {}

My take: Streaming Sidecar is a hack. If you can modify the application, making it output to stdout is the right path. Streaming Sidecar adds an extra container, an extra layer of IO overhead, and a single point of failure for file IO.

Advantages:

Can customize collection rules per Pod
Can collect in-container file logs
Supports multi-tenant log isolation

Disadvantages:

High resource overhead, one extra container per Pod
High operational complexity, managing many Sidecar configurations
Pod startup order and log loss risk

Use cases: Special log formats requiring custom parsing, multi-tenant isolation, Java multi-line stack traces needing special handling.

Architecture 3: Agentless Mode

Agentless mode doesn’t deploy additional Agents, directly using K8s API and container runtime interfaces to collect logs. For example, obtaining logs through kubelet’s API or using kubectl logs.

Advantages: Zero additional resource overhead Disadvantages: Poor real-time performance, no complex processing capability, performance bottleneck in large clusters

This approach is rarely used in production and won’t be covered in detail.

Architecture Comparison

Dimension	DaemonSet	Sidecar	Agentless
Resource overhead	Low (1 per node)	High (1 per Pod)	Minimal
Application intrusion	None	High	None
Customization	Weak	Strong	None
Operations complexity	Low	High	Low
In-container file logs	Not supported	Supported	Not supported
Cluster scale adaptability	Good	Poor	Poor
Recommendation	First choice	Special cases	Not recommended

Fluent Bit Production Configuration

Fluent Bit is a CNCF lightweight log collector written in C, with memory usage typically only 10-50MB. It’s the preferred choice for K8s log collection. The shift from Fluentd to Fluent Bit has been a trend in recent years.

Deploying Fluent Bit as DaemonSet

Complete Fluent Bit DaemonSet deployment configuration:

# fluent-bit-daemonset.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit
rules:
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
      - nodes
      - nodes/proxy
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit
subjects:
  - kind: ServiceAccount
    name: fluent-bit
    namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: kube-system
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         5
        Log_Level     info
        Daemon        off
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020
        Parsers_File  parsers.conf
    
    [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        Parser            cri        # Use cri for containerd, docker for Docker
        Tag               kube.*
        Refresh_Interval  5
        Mem_Buf_Limit    50MB
        Skip_Long_Lines   On
        DB                /var/log/flb_kube.db
        DB.Sync           Normal
    
    [FILTER]
        Name           kubernetes
        Match          kube.*
        Kube_URL       https://kubernetes.default.svc:443
        Kube_CA_File   /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token     /var/run/secrets/kubernetes.io/serviceaccount/token
        Merge_Log      On
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
    
    [OUTPUT]
        Name            es
        Match           *
        Host            elasticsearch.logging.svc
        Port            9200
        Index           k8s-logs
        Type            _doc
        Replace_Dots    On
        Trace_Error     On
        Retry_Limit     5
    
    [OUTPUT]
        Name            stdout
        Match           *
        # For debugging, comment out in production    
  parsers.conf: |
    [PARSER]
        Name        cri
        Format      regex
        Regex       ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
    
    # JSON log parser
    [PARSER]
        Name        json
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
    
    # Multi-line parser (Java stack traces)
    [PARSER]
        Name        multiline_java
        Format      regex
        Regex       ^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?<level>[A-Z]+) (?<message>.*)$    
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: kube-system
  labels:
    k8s-app: fluent-bit
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit
  template:
    metadata:
      labels:
        k8s-app: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:2.2.0
          imagePullPolicy: Always
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 256Mi
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: config
              mountPath: /fluent-bit/etc/
            - name: positions-db
              mountPath: /var/log/
          ports:
            - containerPort: 2020
              name: metrics
              protocol: TCP
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: config
          configMap:
            name: fluent-bit-config
        - name: positions-db
          hostPath:
            path: /var/log/fluent-bit/
            type: DirectoryOrCreate

Key Configuration Details

INPUT section:

Parameter	Description	Production Recommendation
`Path`	Log file path	`/var/log/containers/*.log`
`Parser`	Log format parser	Use `cri` for containerd, `docker` for Docker
`Refresh_Interval`	New file scan interval	5 seconds
`Mem_Buf_Limit`	Memory buffer limit	50MB, pauses reading when exceeded
`Skip_Long_Lines`	Skip overly long lines	On, prevents OOM
`DB`	Position tracking file	Must configure, otherwise restart causes duplicate collection
`DB.Sync`	DB sync strategy	Normal, balances performance and safety

The most important parameter is DB. It records the read position (offset) for each log file. Without it, Fluent Bit starts reading from the beginning on restart, causing massive duplicate data. I’ve seen cases where DB wasn’t configured, and every Pod restart doubled the logs in ES.

FILTER section: The Kubernetes filter calls the K8s API to get Pod metadata (namespace, pod name, labels, annotations) and attaches it to each log entry. This is why you can see which Pod and Deployment a log belongs to in Kibana.

# Control log collection behavior through annotations
# Add annotations to Pods
apiVersion: v1
kind: Pod
metadata:
  name: my-app
  annotations:
    # Fluent Bit parser: specify log format
    fluentbit.io/parser: json
    # Exclude this Pod's logs
    fluentbit.io/exclude: "true"

OUTPUT section: Output target configuration. Besides Elasticsearch, it supports Loki, Kafka, S3, etc.

Fluent Bit Output to Loki

If using Grafana Loki instead of Elasticsearch, change the OUTPUT configuration:

[OUTPUT]
    Name        loki
    Match       kube.*
    Host        loki.logging.svc
    Port        3100
    Labels      job=fluent-bit
    Label_keys  $kubernetes['namespace_name'], $kubernetes['pod_name']
    Line_Format json
    Tenant_ID   ""

Loki’s advantage is being lightweight with low storage costs (only indexes labels, not full text), suitable for small to medium clusters. ES’s advantage is powerful full-text search, suitable for large-scale log retrieval. The selection criteria are discussed in detail later.

Filebeat Production Configuration

Filebeat is Elastic’s official lightweight log collector, written in Go, with the best integration into the ELK ecosystem.

# filebeat-daemonset.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
data:
  filebeat.yml: |
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints:
            enabled: true
          # Auto-discover Pods and attach metadata
          templates:
            - condition:
                contains:
                  kubernetes.labels.app: nginx
              config:
                - module: nginx
                  access:
                    input:
                      type: container
                      stream: stdout
    
    processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          # If Kubelet uses non-standard port, configure here
          kube_client_config:
            kube_config: /etc/kubernetes/kubelet.conf
      - drop_event:
          when:
            # Drop kube-system namespace logs (optional)
            or:
              - contains:
                  kubernetes.namespace: kube-system
    
    output.elasticsearch:
      hosts: ["elasticsearch.logging.svc:9200"]
      index: "k8s-%{[kubernetes.namespace]}-%{+yyyy.MM.dd}"
    
    # Or output to Kafka (high throughput scenarios)
    # output.kafka:
    #   hosts: ["kafka-0.kafka.logging:9092"]
    #   topic: k8s-logs
    #   partition.round_robin:
    #     reachable_only: true
    
    logging.level: info
    monitoring.enabled: true
    monitoring.elasticsearch: true    
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:8.11.0
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi
          volumeMounts:
            - name: config
              mountPath: /usr/share/filebeat/filebeat.yml
              subPath: filebeat.yml
            - name: data
              mountPath: /usr/share/filebeat/data
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: filebeat-config
        - name: data
          hostPath:
            path: /var/lib/filebeat-data
            type: DirectoryOrCreate
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

Fluent Bit vs Filebeat Selection

Dimension	Fluent Bit	Filebeat
Language	C	Go
Memory usage	10-50MB	50-150MB
CPU overhead	Low	Medium
Plugin ecosystem	Rich (CNCF)	Rich (Elastic ecosystem)
Configuration complexity	Medium	Simple
Multi-line log handling	Supported but complex config	Native support
Kafka output	Supported	Better native support
Loki output	Native support	Requires extra config
Monitoring metrics	Built-in Prometheus exporter	Built-in
Community activity	High (CNCF graduated)	High (Elastic maintained)

My selection recommendations:

Using ELK stack → Filebeat, best integration with Elastic ecosystem, works out of the box
Using Loki stack → Fluent Bit, best native support
Extremely resource-constrained (edge nodes) → Fluent Bit, lower memory footprint
Need complex log parsing → Fluent Bit + Fluentd two-tier architecture, Fluent Bit collects, Fluentd aggregates and processes

Backend Storage Selection: Elasticsearch vs Loki

The choice of log backend storage directly impacts operational costs and query capabilities.

Elasticsearch

Elasticsearch (ES) is the de facto standard for log retrieval, with powerful full-text indexing, but high resource consumption.

Suitable scenarios:

Large daily log volume (TB level)
Need full-text search, fuzzy queries
Need complex aggregation analysis
Team has ES operations experience

Resource planning:

Cluster size	Fluent Bit	ES node config	ES nodes	Daily storage
Small (20 nodes)	0.5 core/100MB	4 core/8GB	3	~5GB/day
Medium (100 nodes)	1 core/200MB	8 core/16GB	5	~25GB/day
Large (300+ nodes)	2 core/500MB	16 core/32GB	9+	~75GB/day

ES JVM heap memory should not exceed 50% of physical memory, and at least 2GB must be reserved for system cache (Linux page cache). Many people don’t know this—setting the heap too large actually slows down performance.

Grafana Loki

Loki is a log system built by the Grafana team, designed to be “like Prometheus but for logs.” It only indexes log labels (metadata), not the log content itself, making storage costs an order of magnitude lower than ES.

Suitable scenarios:

Medium log volume (GB to tens of GB/day)
Primarily query logs by labels (namespace, pod, app)
Already have Grafana infrastructure
Limited storage budget

# Loki simplified deployment (single node, for testing)
apiVersion: v1
kind: ConfigMap
metadata:
  name: loki-config
  namespace: logging
data:
  loki.yaml: |
    auth_enabled: false
    server:
      http_listen_port: 3100
    common:
      path_prefix: /loki
      storage:
        filesystem:
          chunks_directory: /loki/chunks
          rules_directory: /loki/rules
      replication_factor: 1
      ring:
        instance_addr: 0.0.0.0
        kvstore:
          store: inmemory
    schema_config:
      configs:
        - from: 2024-01-01
          store: tsdb
          object_store: filesystem
          schema: v13
          index:
            prefix: index_
            period: 24h
    limits_config:
      retention_period: 30d        # Retain logs for 30 days
      max_query_series: 5000
      ingestion_rate_mb: 10        # Write rate limit
      ingestion_burst_size_mb: 20
    compactor:
      working_directory: /loki/compactor
      retention_enabled: true
      delete_request_store: filesystem

ES vs Loki Comparison

Dimension	Elasticsearch	Loki
Indexing method	Full-text index	Label-only index
Storage cost	High	Low (1/10 scale)
Query capability	Full-text search, fuzzy matching	Label filtering + LogQL
Query speed	Fast (indexed)	Medium (requires scanning)
Resource consumption	High	Low
Operations complexity	High	Low
Log volume limit	TB/day	GB/day
Suitable scale	Large scale	Small-medium scale

My recommendations:

Log volume <50GB/day → Loki, cheap storage, simple operations
Log volume 50-500GB/day → Depends on query needs: full-text search use ES, label queries use Loki
Log volume >500GB/day → ES + hot-cold data separation + Kafka buffer

Don’t be fooled by Loki’s “low cost”. Loki is great for querying logs by labels (like searching for logs in a specific namespace), but if you need full-text search (like searching for which logs contain a specific error message), Loki scans large amounts of data and is much slower than ES. The two serve different query patterns—choose based on your actual query needs.

Production Pitfall Guide

Pitfall 1: Log Loss—Pod Logs Disappear After Deletion

When a container is deleted, the log files on the node are also cleaned up. If Fluent Bit hasn’t finished collecting, those logs are permanently lost.

# Check Fluent Bit collection progress
# View position tracking DB
sqlite3 /var/log/flb_kube.db "SELECT * FROM in_tail;"
# Compare file size and read offset

Solutions:

# Solution 1: Configure PreStop hook to wait for Fluent Bit to finish collecting
apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
    - name: app
      image: myapp:latest
      lifecycle:
        preStop:
          exec:
            command: ["sleep", "10"]  # Wait 10 seconds for Fluent Bit to collect

# Solution 2: Configure Graceful Shutdown for Fluent Bit
# Configure terminationGracePeriodSeconds in DaemonSet
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: fluent-bit
          # Configure graceful exit to flush buffer data
          env:
            - name: FLUENT_BIT_FLUSH_INTERVAL
              value: "1"

Pitfall 2: Multi-line Log Merge Failure

When Java applications throw exceptions, stack traces span multiple lines. Without multi-line merging, each line becomes a separate log entry, resulting in fragmented stack traces in Kibana.

Fluent Bit multi-line configuration:

# Add multi-line parser in parsers.conf
[PARSER]
    Name        multiline_java
    Format      regex
    Regex       ^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?<level>\w+) (?<message>.*)$

# Use multi-line in INPUT section of fluent-bit.conf
[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    Parser            cri
    Tag               kube.*
    multiline.parser  java_exception    # Built-in Java multi-line parser
    Refresh_Interval  5

Fluent Bit 2.0+ has built-in common multi-line parsers:

# Built-in multi-line parsers
multiline.parser: java_exception, python, go

Filebeat multi-line configuration is more intuitive:

filebeat.inputs:
  - type: container
    paths:
      - /var/log/containers/*.log
    multiline:
      type: pattern
      pattern: '^\d{4}-\d{2}-\d{2}'  # Lines starting with date are new log entries
      negate: true
      match: after
      flush_pattern: '^\d{4}-\d{2}-\d{2}'
      timeout: 5s

Pitfall 3: Fluent Bit Memory OOM

During log spikes, Fluent Bit memory surges and gets OOM-killed.

# Limit memory and configure backpressure mechanism
[SERVICE]
    Flush         5
    Mem_Buf_Limit 50MB        # Memory buffer limit
    # Pause reading when exceeded, resume after buffer data is flushed

[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    Mem_Buf_Limit    10MB    # INPUT-level buffer limit
    Buffer_Chunk_Size 512KB  # Size of each read chunk
    Buffer_Max_Size   5MB    # Max buffer per file

# K8s resource limits must be reasonable
resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 256Mi    # Don't set too low, easy OOM during log spikes

War story: During a major promotion, log volume surged 10x. Fluent Bit memory limit was set to 128Mi, and it got OOM-killed. The Pod restarted repeatedly, causing massive log loss. After increasing the memory limit to 512Mi and configuring Mem_Buf_Limit for backpressure, it stabilized.

Pitfall 4: Inconsistent Log Formats Causing Parse Failures

Different applications have different log formats—some JSON, some plain text, some key=value. Unified parsing is a headache.

# Specify parser through Pod annotations
apiVersion: v1
kind: Pod
metadata:
  name: json-app
  annotations:
    fluentbit.io/parser: json    # Use JSON parser
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx-app
  annotations:
    fluentbit.io/parser: nginx   # Use Nginx parser

# Corresponding parsers.conf
[PARSER]
    Name        json
    Format      json
    Time_Key    timestamp
    Time_Format %Y-%m-%dT%H:%M:%S.%LZ

[PARSER]
    Name        nginx
    Format      regex
    Regex       ^(?<remote>\S+) - (?<user>\S+) \[(?<time>[^\]]+)\] "(?<method>\S+) (?<path>\S+) (?<protocol>\S+)" (?<status>\d+) (?<size>\d+) "(?<referer>[^"]*)" "(?<agent>[^"]*)"$
    Time_Key    time
    Time_Format %d/%b/%Y:%H:%M:%S %z

Pitfall 5: Improper K8s Log Rotation Configuration

Container logs on nodes without rotation will fill up the disk.

# Check container log rotation configuration
# containerd config file /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd]
  # Max log file line size
  max_container_log_line_size = -1
  
# Or through kubelet configuration
# /var/lib/kubelet/config.yaml
containerLogMaxSize: "100Mi"       # Max 100MB per log file
containerLogMaxFiles: 5             # Keep max 5 log files

# Check node log disk usage
du -sh /var/log/containers/
du -sh /var/log/pods/

# If logs fill the disk, emergency cleanup
# First find the largest log files
find /var/log/containers/ -name "*.log" -size +500M -exec ls -lh {} \;
# Truncate (don't delete, to avoid file handle issues)
truncate -s 0 /var/log/containers/huge-app.log

Pitfall 6: Fluent Bit Position DB Loss

When a DaemonSet Pod is rescheduled to another node, the local position tracking DB is also lost. The new Pod starts reading from the beginning, causing duplicates.

# Solution 1: Mount DB to hostPath
volumes:
  - name: positions-db
    hostPath:
      path: /var/lib/fluent-bit/
      type: DirectoryOrCreate
volumeMounts:
  - name: positions-db
    mountPath: /var/lib/fluent-bit/

# Solution 2: Use persistent path in configuration
[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    DB                /var/lib/fluent-bit/flb_kube.db
    DB.Sync           Normal

Log Monitoring and Alerting

The log collection system itself needs monitoring. Fluent Bit has a built-in Prometheus metrics exporter.

# Fluent Bit built-in metrics port defaults to 2020
curl http://localhost:2020/metrics

# Key metrics:
# fluentbit_input_records_total       → Number of logs collected
# fluentbit_output_proc_records_total → Number of logs output
# fluentbit_output_errors_total       → Number of output errors
# fluentbit_input_bytes_total         → Number of bytes collected

Prometheus alert rule examples:

# fluent-bit-alerts.yaml
groups:
  - name: fluent-bit
    rules:
      - alert: FluentBitDown
        expr: up{job="fluent-bit"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Fluent Bit is down on {{ $labels.node }}"
          description: "Fluent Bit has been down for 2 minutes"
      
      - alert: FluentBitHighErrorRate
        expr: rate(fluentbit_output_errors_total[5m]) > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Fluent Bit high error rate on {{ $labels.node }}"
          description: "Fluent Bit is experiencing high output error rate"
      
      - alert: FluentBitNoLogsCollected
        expr: rate(fluentbit_input_records_total[5m]) == 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "No logs collected by Fluent Bit on {{ $labels.node }}"
          description: "Fluent Bit hasn't collected any logs in the last 10 minutes"
      
      - alert: FluentBitInputOutputGap
        # Large gap between input and output indicates backlog
        expr: |
          sum(rate(fluentbit_input_records_total[5m])) by (node) - 
          sum(rate(fluentbit_output_proc_records_total[5m])) by (node) > 100          
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Fluent Bit log backlog on {{ $labels.node }}"
          description: "Input rate significantly exceeds output rate, logs are backing up"

Log Collection Strategy Selection Framework

Choose the approach based on cluster size and requirements:

Cluster size < 20 nodes?
├─ Yes → DaemonSet + Fluent Bit + Loki
│       Low cost, simple operations, sufficient
│
├─ No → Daily log volume?
│   ├─ < 50GB/day → DaemonSet + Fluent Bit + Loki
│   │              Medium scale, low Loki storage cost
│   │
│   ├─ 50-500GB/day → DaemonSet + Fluent Bit + Kafka + ES
│   │                 Kafka as buffer, ES for search
│   │
│   └─ > 500GB/day → DaemonSet + Fluent Bit + Kafka + ES (hot-cold separation)
│                    ES with hot-warm-cold three-tier data nodes
│
└─ Special log format needs?
    └─ Java stack traces, multi-line logs → Sidecar mode or multi-line parser

Selection checklist:

Cluster node count: ______
Daily log volume: ______ GB
Log format: stdout / file / mixed
Need full-text search: yes / no
Log retention days: ______ days
Budget constraints: ______
Already have Grafana: yes / no
Already have ES cluster: yes / no

Summary

The core decisions in K8s log collection are three: what collection mode, what collection tool, and what backend storage.

A few practical takeaways:

DaemonSet mode covers 90% of scenarios: Unless you have special requirements, don’t use Sidecar. Sidecar’s resource overhead and operational complexity far outweigh its benefits. I’ve seen a team add Sidecars to every Pod—half the containers in the cluster were log collectors, wasting resources enormously
The position tracking DB is a lifeline: Fluent Bit’s DB parameter records the read position. Not configuring it or losing it causes duplicate collection or missed logs. It must be persisted to hostPath
Configure multi-line logs in advance: Java stack traces are the classic multi-line log scenario. Validate multi-line parsing with test data before going live—don’t discover fragmented stack traces during incident response
Implement backpressure for log spikes: During promotions, load testing, and similar scenarios, log volume surges. Configure Mem_Buf_Limit for backpressure—pause reading when limits are exceeded instead of buffering infinitely and causing OOM
Monitor the log collection system itself: Fluent Bit crashes, ES disk fills up, Loki write limits exceeded—these all need alerts. Otherwise the logging system fails silently, and you think logs are being collected when they’ve been broken for hours
Configure container log rotation in advance: kubelet’s containerLogMaxSize and containerLogMaxFiles must be configured, or the disk could fill up with logs and cause node NotReady

Log collection seems like a small task, but in K8s environments it involves many components and decision points. Choose the right approach, configure parameters properly, and set up monitoring to quickly locate issues during troubleshooting.

References & Acknowledgments

This article referenced the following materials during writing. Thanks to the original authors for their contributions:

Filebeat vs Logstash vs Fluent Bit: 三大日志采集器深度对比与选型指南 — Tencent Cloud Developer Community, architecture comparison and performance analysis of Filebeat/Logstash/Fluent Bit
别再纠结DaemonSet还是Sidecar了！手把手教你根据业务场景选对K8S日志采集方案 — CSDN, business scenario selection guide for DaemonSet vs Sidecar modes
从零搭建一套生产可用的K8S日志监控栈:EFK/ELK保姆级配置与避坑指南 — CSDN, EFK/ELK production deployment configuration and performance tuning
Fluentd/FluentBit 接入 — Tencent Cloud Documentation, Fluent Bit deployment methods and Prometheus monitoring integration
如何收集k8s集群日志? — Tencent Cloud Developer Community, comparison of Fluentd/Filebeat/Vector log collection solutions

Overview#

Three Types of K8s Logs#

1. Container Standard Output Logs#

2. In-Container File Logs#

3. Node System Logs#

Comparison of Three Log Collection Architectures#

Architecture 1: DaemonSet Mode#

Architecture 2: Sidecar Mode#

Architecture 3: Agentless Mode#

Architecture Comparison#

Fluent Bit Production Configuration#

Deploying Fluent Bit as DaemonSet#

Key Configuration Details#

Fluent Bit Output to Loki#

Filebeat Production Configuration#

Fluent Bit vs Filebeat Selection#

Backend Storage Selection: Elasticsearch vs Loki#

Elasticsearch#

Grafana Loki#

ES vs Loki Comparison#

Production Pitfall Guide#

Pitfall 1: Log Loss—Pod Logs Disappear After Deletion#

Pitfall 2: Multi-line Log Merge Failure#

Pitfall 3: Fluent Bit Memory OOM#

Pitfall 4: Inconsistent Log Formats Causing Parse Failures#

Pitfall 5: Improper K8s Log Rotation Configuration#

Pitfall 6: Fluent Bit Position DB Loss#

Log Monitoring and Alerting#

Log Collection Strategy Selection Framework#

Summary#

References & Acknowledgments#