Overview

Logs are the eyes of system operations. From kernel messages to application logs, from security auditing to performance analysis, logs run through every aspect of troubleshooting. Modern Linux uses journald as the system logging daemon, combined with logrotate for log rotation, forming a complete log management infrastructure. This article dives deep into journald internals and configuration, advanced journalctl query techniques, log rotation strategies, remote log collection solutions, and practical analysis cases.

journald Internals

Architecture Overview

journald is systemd’s system logging component, replacing the traditional syslog (rsyslog). It receives logs from the kernel, system services, and applications, storing them in a structured binary format.

[Kernel Logs]      [systemd Services]    [Applications]
                                         
                                         
[kmsg]           [sd_journal_print]  [syslog()/stdout]
                                         
     └──────────┬───────┴──────────┬───────┘
                                  
           [journald]         [/dev/log]
                
     ┌──────────┼──────────┐
                         
[Persistent    [Runtime     [Forward to
 Logs]          Logs]        syslog]
/var/log/      /run/log/    /var/log/
journal/       journal/     messages

Log Storage Modes

# View current storage mode
$ cat /etc/systemd/journald.conf | grep Storage
#Storage=auto

# Three modes:
# auto (default): Persist if /var/log/journal exists, otherwise in-memory only
# persistent: Force persistence (auto-creates /var/log/journal)
# volatile: In-memory only (/run/log/journal)
ModeStorage LocationSurvives RebootUse Case
persistent/var/log/journalYesProduction
auto/var/log/journal or /run/log/journalDepends on directoryDefault
volatile/run/log/journalNoTemporary systems/security requirements

Log Size Control

# /etc/systemd/journald.conf

[Journal]
Storage=persistent

# Global log size limits
SystemMaxUse=2G          # Max persistent log usage
SystemKeepFree=4G        # Ensure at least 4G free disk space
SystemMaxFileSize=100M   # Max size per log file
SystemMaxFiles=100       # Keep up to 100 log files

# Runtime log (in-memory) limits
RuntimeMaxUse=200M
RuntimeKeepFree=100M
RuntimeMaxFileSize=20M
RuntimeMaxFiles=20

# Log retention time
MaxRetentionSec=2week    # Keep logs for max 2 weeks
MaxFileSec=1month        # Max 1 month per file

# Forwarding configuration
ForwardToSyslog=no       # Do not forward to syslog (deprecated)
ForwardToKMsg=no
ForwardToConsole=no
ForwardToWall=yes        # Send urgent messages to all terminals

# Log rate limiting
RateLimitIntervalSec=30s
RateLimitBurst=10000     # Max 10000 entries per 30 seconds

Viewing and Managing Log Size

# View log disk usage
$ journalctl --disk-usage
Archived and active journals take up 1.2G in the file system.

# Immediately reduce logs to specified size
$ journalctl --vacuum-size=500M
Vacuumed 123 archived journals (1.2G → 500M)

# Clean by time
$ journalctl --vacuum-time=7d
Vacuumed 45 archived journals older than 7 days

# Clean by file count
$ journalctl --vacuum-files=10
Vacuumed 90 archived journals (keeping 10 files)

# Verify log integrity
$ journalctl --verify

journalctl Query Techniques

Basic Queries

# View all logs
$ journalctl

# View current boot logs
$ journalctl -b

# View previous boot logs
$ journalctl -b -1

# View specific boot logs
$ journalctl --list-boots
IDX  BOOT ID     FIRST ENTRY               LAST ENTRY
-2   abc123...   Mon 2026-07-08 09:00:00   Mon 2026-07-08 18:00:00
-1   def456...   Tue 2026-07-09 09:00:00   Tue 2026-07-09 18:00:00
 0   ghi789...   Wed 2026-07-10 09:00:00   Wed 2026-07-10 15:30:00
$ journalctl -b -2  # View logs from two boots ago

Time Filtering

# Today's logs
$ journalctl --since today

# Last 1 hour
$ journalctl --since "1 hour ago"

# Specific time range
$ journalctl --since "2026-07-10 09:00:00" --until "2026-07-10 12:00:00"

# Relative time
$ journalctl --since "1 hour ago" --until "10 min ago"
$ journalctl --since yesterday
$ journalctl --since "2026-07-01" --until "2026-07-10"

Filtering by Unit

# View specific service logs
$ journalctl -u nginx
$ journalctl -u sshd
$ journalctl -u docker

# Multiple services
$ journalctl -u nginx -u php-fpm

# View service logs since current boot
$ journalctl -u nginx -b

# View service logs from previous failed boot
$ journalctl -u nginx -b -1 --since "1 hour ago"

Filtering by Priority

# Log priorities
# 0: emerg   - Emergency
# 1: alert   - Alert
# 2: crit    - Critical
# 3: err     - Error
# 4: warning - Warning
# 5: notice  - Notice
# 6: info    - Info
# 7: debug   - Debug

# View logs at specified level and above
$ journalctl -p err        # err and above
$ journalctl -p warning    # warning and above
$ journalctl -p 3          # Same as -p err

# Specify level range
$ journalctl -p warning..err  # warning to err

Filtering by Field

# By process ID
$ journalctl _PID=12345

# By user ID
$ journalctl _UID=1000

# By GID
$ journalctl _GID=1000

# By executable
$ journalctl _COMM=nginx
$ journalctl _EXE=/usr/sbin/nginx

# By device
$ journalctl _KERNEL_DEVICE=eth0

# Custom fields (structured logging)
$ journalctl SYSLOG_IDENTIFIER=myapp
$ journalctl MESSAGE_ID=abc123

Output Format Control

# Default format
$ journalctl -u nginx -n 5

# Short format
$ journalctl -u nginx -n 5 -o short

# Precise format (with full timestamps)
$ journalctl -u nginx -n 5 -o short-precise

# JSON format (suitable for script processing)
$ journalctl -u nginx -n 5 -o json

# Pretty JSON format
$ journalctl -u nginx -n 5 -o json-pretty

# Category display
$ journalctl -u nginx -n 5 -o cat
# Shows only message content, no timestamps

# Export format (for backup/migration)
$ journalctl -u nginx --since today -o export > nginx-logs.export

Real-Time Tracking

# Real-time tracking (like tail -f)
$ journalctl -f

# Track specific service
$ journalctl -u nginx -f

# Track multiple services
$ journalctl -u nginx -u php-fpm -f

# Show only error-level real-time logs
$ journalctl -f -p err

Advanced Query Combinations

# Combined query: time + service + level
$ journalctl --since "1 hour ago" -u nginx -p err

# View kernel logs
$ journalctl -k

# View logs for a specific user
$ journalctl _UID=1000 --since today

# View killed processes
$ journalctl --since "1 hour ago" | grep -i "killed\|oom"

# Filter by message content
$ journalctl --since today | grep "connection refused"

# Output as JSON and process with jq
$ journalctl -u nginx --since "1 hour ago" -o json | jq 'select(.MESSAGE | contains("error"))'

journalctl Quick Reference

NeedCommand
Current boot logsjournalctl -b
Previous boot logsjournalctl -b -1
Kernel logsjournalctl -k
Service logsjournalctl -u nginx
Error logsjournalctl -p err
Last 1 hourjournalctl --since "1 hour ago"
Real-time trackingjournalctl -f
Disk usagejournalctl --disk-usage
Clean up logsjournalctl --vacuum-size=500M
JSON outputjournalctl -o json
Message content onlyjournalctl -o cat
Specific PIDjournalctl _PID=12345

Log Rotation (logrotate)

Why logrotate Is Needed

journald has its own log size control, but many applications (like Nginx, MySQL) write log files directly without going through journald. These log files grow continuously and need logrotate to periodically rotate, compress, and clean them.

How logrotate Works

1. Check config files (/etc/logrotate.d/*)
2. Determine if log files meet rotation criteria
3. If criteria are met, perform rotation:
   a. Rename current log (app.log → app.log.1)
   b. Optional: compress old log (app.log.1 → app.log.1.gz)
   c. Delete old logs exceeding retention count
   d. Create new empty log file
   e. Notify application to reopen log file

Main Configuration File

# /etc/logrotate.conf — global configuration

weekly                        # Rotate weekly
rotate 4                      # Keep 4 copies
create                        # Create new file after rotation
compress                      # Compress old logs
dateext                       # Use date as suffix
dateformat -%Y%m%d            # Date format
include /etc/logrotate.d      # Include sub-config directory

Application-Level Configuration Examples

Nginx Log Rotation

# /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    daily
    rotate 30
    missingok
    notifempty
    compress
    delaycompress           # Delay compression by one cycle (for troubleshooting)
    sharedscripts           # Share one script for multiple logs
    postrotate
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 $(cat /var/run/nginx.pid)
        fi
    endscript
}

MySQL Log Rotation

# /etc/logrotate.d/mysql
/var/log/mysql/*.log {
    daily
    rotate 14
    missingok
    create 640 mysql adm
    compress
    delaycompress
    notifempty
    sharedscripts
    postrotate
        mysqladmin flush-logs 2>/dev/null || true
    endscript
}

Application Log Rotation

# /etc/logrotate.d/myapp
/opt/myapp/logs/*.log {
    daily
    rotate 30
    missingok
    notifempty
    compress
    delaycompress
    copytruncate            # Copy then truncate (no service interruption)
    size 100M               # Also rotate when exceeding 100M
    dateext
    dateformat -%Y%m%d-%H%M%S
}

Key Parameter Reference

ParameterDescriptionExample
daily/weekly/monthlyRotation frequencydaily
rotate NNumber of copies to keeprotate 30
sizeSize thresholdsize 100M
compressCompress old logscompress
delaycompressDelay compression by one cycledelaycompress
missingokDon’t error if log missingmissingok
notifemptyDon’t rotate empty filesnotifempty
createCreate new file after rotationcreate 640 root root
copytruncateCopy then truncate original filecopytruncate
dateextUse date suffixdateext
sharedscriptsShare script (run once for multiple logs)sharedscripts
postrotate/endscriptScript to run after rotationpostrotate
prerotate/endscriptScript to run before rotationprerotate

copytruncate vs postrotate

MethodHow It WorksAdvantageDisadvantage
postrotateRename + notify app to reopen fileNo data lossApp must support signals
copytruncateCopy + truncate original fileNo app cooperation neededPossible log loss between copy and truncate

Manual Execution and Debugging

# Manually run all rotations
$ logrotate /etc/logrotate.conf

# Debug mode (dry-run, no actual execution)
$ logrotate -d /etc/logrotate.d/nginx

# Force rotation
$ logrotate -f /etc/logrotate.d/nginx

# View status
$ cat /var/lib/logrotate/status
# Records last rotation time for each log

# Verbose output
$ logrotate -v /etc/logrotate.d/nginx

logrotate Cron Job

# logrotate is typically executed via cron or systemd timer
# Debian/Ubuntu: /etc/cron.daily/logrotate
# RHEL/CentOS: /etc/cron.daily/logrotate

# systemd timer approach
$ systemctl list-timers | grep logrotate
$ systemctl cat logrotate.timer

Structured Logging

Problems with Traditional Logs

# Unstructured log
Jul 10 15:30:00 server nginx: 10.0.0.1 - - [10/Jul/2026:15:30:00 +0800] "GET /api/users HTTP/1.1" 200 1234
# Hard to parse, hard to query, hard to correlate

journald Structured Logging

journald natively supports structured fields. Applications can write structured logs through the following methods:

C Program

#include <systemd/sd-journal.h>

int main() {
    sd_journal_print(LOG_INFO, "User login successful");
    
    // Structured fields
    sd_journal_send(
        "MESSAGE=User %s logged in from %s",
        "PRIORITY=6",
        "USER_ID=12345",
        "USERNAME=admin",
        "SOURCE_IP=10.0.0.1",
        "ACTION=login",
        NULL
    );
    return 0;
}

Python

import logging
import systemd.journal

logger = logging.getLogger('myapp')
logger.addHandler(systemd.journal.JournalHandler())
logger.setLevel(logging.INFO)

# Regular log
logger.info("User login successful")

# Structured log
logger.info("User login", extra={
    'USER_ID': 12345,
    'USERNAME': 'admin',
    'SOURCE_IP': '10.0.0.1',
    'ACTION': 'login'
})

# Query
# journalctl USER_ID=12345
# journalctl ACTION=login

Shell Script

# Write structured logs using systemd-cat
$ echo "Backup completed" | systemd-cat -t myapp -p info

# With structured fields
$ systemd-cat -t myapp -p info << EOF
BACKUP_ID=20260710
STATUS=success
DURATION=3600
FILES=12345
SIZE=10GB
Backup completed successfully
EOF

# Query
$ journalctl -t myapp BACKUP_ID=20260710

Go Program

package main

import (
    "log/syslog"
    "os"
)

func main() {
    writer, _ := syslog.New(syslog.LOG_INFO|syslog.LOG_USER, "myapp")
    logger := log.New(writer, "", 0)
    logger.Println("Application started")
    
    // Or use coreos/go-systemd library
    // to write structured fields directly to journald
}

Log Correlation (Request ID)

# Generate a Request ID at the entry point, trace it through the entire call chain
$ journalctl REQUEST_ID=abc-123-def
# Can trace a request's complete logs across multiple services

# Application implementation
# 1. API gateway generates Request ID
# 2. Pass through HTTP headers to downstream services
# 3. Each service writes Request ID to journald structured fields
# 4. Use journalctl REQUEST_ID=xxx to query the full chain

Remote Log Collection

Solution Comparison

SolutionProtocolReliabilityPerformanceUse Case
rsyslog + TCPTCPHighMediumTraditional approach
journald → rsyslogTCP/UDPMediumMediumTransitional solution
Fluentd/Fluent BitTCP/HTTPHighHighContainerized/cloud-native
Promtail + LokiHTTPHighHighGrafana ecosystem
Filebeat + ELKTCPHighHighELK ecosystem

journald Forwarding to rsyslog

# /etc/systemd/journald.conf
[Journal]
ForwardToSyslog=yes

# /etc/rsyslog.conf or /etc/rsyslog.d/remote.conf
# Forward to remote log server
*.* @@log-server.sre.wang:514    # TCP
# Or
*.* @log-server.sre.wang:514     # UDP

# Forward by facility
local0.*                        @@log-server:514
*.info;mail.none;authpriv.none  @@log-server:514

systemd journal-upload

# Use systemd-journal-upload to send to remote journal server
$ systemctl enable --now systemd-journal-upload

# Configuration
# /etc/systemd/journal-upload.conf
[Upload]
URL=https://journal-collector.sre.wang/upload
# Or use push mode
# KeyFile=/etc/ssl/private/journal-upload.key
# CertificateFile=/etc/ssl/certs/journal-upload.crt
# TrustedCertificateFile=/etc/ssl/certs/ca.crt

Fluent Bit Collecting journald

# /etc/fluent-bit/fluent-bit.conf
[SERVICE]
    Flush         5
    Log_Level     info

[INPUT]
    Name          systemd
    Tag           host.*
    Systemd_Filter _SYSTEMD_UNIT=nginx.service
    Systemd_Filter _SYSTEMD_UNIT=sshd.service
    DB            /var/log/flb_journald.db

[OUTPUT]
    Name          forward
    Match         *
    Host          fluentd.sre.wang
    Port          24224

Log Collection Architecture

[Server A]                    [Server B]                    [Server C]
  journald                      journald                      journald
     │                              │                              │
     ▼                              ▼                              ▼
[Fluent Bit]                  [Fluent Bit]                  [Fluent Bit]
     │                              │                              │
     └──────────┬───────────────────┴──────────────────┬───────────┘
                ▼                                      ▼
          [Fluentd/Loki]                        [Elasticsearch]
                │                                      │
                ▼                                      ▼
           [Grafana/Kibana]                     [Kibana Dashboard]

Log Disk Usage Control

journald Disk Control

# 1. Configure size limits
# /etc/systemd/journald.conf
SystemMaxUse=2G
SystemKeepFree=4G
MaxRetentionSec=2week

# 2. Restart journald
$ systemctl restart systemd-journald

# 3. View current usage
$ journalctl --disk-usage

# 4. Immediate cleanup
$ journalctl --vacuum-size=1G
$ journalctl --vacuum-time=7d

Application Log Disk Control

# Use logrotate to control size
# /etc/logrotate.d/app-logs
/var/log/myapp/*.log {
    daily
    rotate 30
    compress
    delaycompress
    size 500M
    maxsize 2G          # Max 2G per file
    notifempty
    copytruncate
}

# Use a script to monitor log directory size
#!/bin/bash
# /usr/local/bin/log-size-monitor.sh
LOG_DIR="/var/log/myapp"
MAX_SIZE_GB=10

CURRENT_SIZE=$(du -sg $LOG_DIR | awk '{print $1}')
if [ $CURRENT_SIZE -gt $MAX_SIZE_GB ]; then
    # Trigger cleanup
    find $LOG_DIR -name "*.log.*" -mtime +7 -delete
    journalctl --vacuum-size=500M
    echo "Log cleanup triggered: ${CURRENT_SIZE}GB > ${MAX_SIZE_GB}GB" | \
        systemd-cat -t log-monitor -p warning
fi

Disk Space Monitoring

# Use systemd timer for periodic checks
# /etc/systemd/system/log-monitor.service
[Unit]
Description=Log Size Monitor

[Service]
Type=oneshot
ExecStart=/usr/local/bin/log-size-monitor.sh

# /etc/systemd/system/log-monitor.timer
[Unit]
Description=Run Log Size Monitor hourly

[Timer]
OnCalendar=hourly
Persistent=true

[Install]
WantedBy=timers.target

$ systemctl enable --now log-monitor.timer

Practical Log Analysis

Case 1: SSH Brute-Force Analysis

# Count top IPs of failed SSH logins in the last 24 hours
$ journalctl -u sshd --since "24 hours ago" | \
    grep "Failed" | \
    awk '{print $(NF)}' | \
    sort | uniq -c | sort -rn | head -20

# Count attempted usernames
$ journalctl -u sshd --since "24 hours ago" | \
    grep "Failed" | \
    awk '{for(i=1;i<=NF;i++) if($i=="for") print $(i+1)}' | \
    sort | uniq -c | sort -rn | head -20

# Count failures per hour
$ journalctl -u sshd --since "24 hours ago" | \
    grep "Failed" | \
    awk '{print substr($3,1,2)":00"}' | \
    sort | uniq -c

Case 2: Service Troubleshooting

# Nginx crash investigation
# 1. View logs before crash
$ journalctl -u nginx -b -1 --since "30 min ago" -n 100

# 2. Check OOM records
$ journalctl -k --since "1 hour ago" | grep -i "oom\|killed"

# 3. View service exit code
$ systemctl status nginx
$ journalctl -u nginx -n 50 -o json | jq '.[] | {MESSAGE: .MESSAGE, EXIT_STATUS: ._SYSTEMD_UNIT}'

# 4. Correlate multiple service logs
$ journalctl -u nginx -u php-fpm --since "30 min ago" -o short-precise

Case 3: Performance Issue Investigation

# View kernel performance-related logs
$ journalctl -k --since "1 hour ago" | grep -iE "hung|timeout|blocked|slow|delay"

# View I/O errors
$ journalctl -k --since "1 hour ago" | grep -iE "I/O error|read error|write error"

# View memory pressure
$ journalctl -k --since "1 hour ago" | grep -iE "oom|memory|swap|out of"

# View network anomalies
$ journalctl -k --since "1 hour ago" | grep -iE "link.*down|link.*up|carrier|timeout|reset"

Case 4: Security Audit

# View all sudo operations
$ journalctl -t sudo --since today

# View user login/logout
$ journalctl --since today | grep -E "session (opened|closed)"

# View file permission changes
$ journalctl --since today | grep -i "chmod\|chown"

# View service start/stop
$ journalctl --since today | grep -E "Started|Stopped|Failed"

# View all cron job executions
$ journalctl -t CRON --since today

# Export audit logs
$ journalctl --since "2026-07-01" --until "2026-07-10" -o export > audit-july.export

Case 5: Log Alerting

# Use systemd path to monitor log files
# /etc/systemd/system/log-alert.path
[Unit]
Description=Monitor Error Log

[Path]
PathChanged=/var/log/myapp/error.log

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/log-alert.service
[Unit]
Description=Send Alert on Error Log Change

[Service]
ExecStart=/usr/local/bin/send-alert.sh
# Use journalctl for continuous monitoring and alerting
#!/bin/bash
# /usr/local/bin/error-monitor.sh
journalctl -f -p err | while read line; do
    # Send to alerting system
    curl -X POST https://alerts.sre.wang/api/alert \
        -H "Content-Type: application/json" \
        -d "{\"message\": \"$line\"}"
done

Best Practices

Log Configuration Checklist

# /etc/systemd/journald.conf
[Journal]
Storage=persistent
Compress=yes
Seal=yes
SplitMode=uid
RateLimitIntervalSec=30s
RateLimitBurst=10000
SystemMaxUse=2G
SystemKeepFree=4G
SystemMaxFileSize=100M
MaxRetentionSec=2week
MaxFileSec=1month
ForwardToSyslog=no
ForwardToWall=yes
MaxLevelStore=debug
MaxLevelSyslog=info

Log Level Usage Guide

LevelUse CaseExample
emergSystem unusableKernel panic
alertImmediate action requiredDisk full
critCritical errorService crash
errRuntime errorRequest failed
warningPotential problemDisk usage 80%
noticeImportant eventConfiguration change
infoNormal operationRequest completed
debugDebug informationDetailed parameters

Log Writing Best Practices

# 1. Applications should write through systemd logging API
#    rather than writing files directly (gains structured fields, auto-rotation)

# 2. Include sufficient context
logger -t myapp "Request completed" \
    REQUEST_ID=$REQ_ID \
    METHOD=$METHOD \
    PATH=$PATH \
    STATUS=$STATUS \
    DURATION=${DURATION}ms

# 3. Do not log sensitive information
# Bad:  logger "User login: password=abc123"
# Good: logger "User login: user=admin, method=publickey"

# 4. Use log levels sensibly
#    - Default to info level in production
#    - Temporarily switch to debug for troubleshooting
#    - Don't log debug in loops

# 5. Standardize log format
#    Timestamp + Level + Module + Message + Context fields

Summary

Log management is a foundational operations skill — from logs you can discover issues, locate faults, and audit actions. Key takeaways:

  1. journald is the standard logging system for modern Linux: Structured storage, indexed queries, automatic rotation — far superior to traditional syslog.
  2. journalctl is a powerful query tool: Supports filtering by time, service, priority, and fields; -f for real-time tracking; -o json for script processing.
  3. Log size must be controlled: SystemMaxUse and MaxRetentionSec prevent logs from filling the disk; --vacuum-size for emergency cleanup.
  4. logrotate manages application log files: Applications like Nginx and MySQL that write files directly must have logrotate configured.
  5. Structured logging is the trend: Use journald structured fields (REQUEST_ID, USER_ID, etc.) to enable log correlation and fast queries.
  6. Remote log collection is essential for large-scale operations: Fluent Bit + Loki/ELK enables centralized log management.
  7. Log analysis requires a toolchain: journalctl + grep/awk/jq combinations cover most analysis needs.
  8. Log security cannot be overlooked: Don’t log sensitive information, control access permissions, and regularly audit login and operation logs.

The golden rule of log management: logs are data with a cost. Every log entry consumes storage, I/O, and CPU — only log what’s valuable, and ensure logs are searchable, correlatable, and alertable.