Overview
Logs are the eyes of system operations. From kernel messages to application logs, from security auditing to performance analysis, logs run through every aspect of troubleshooting. Modern Linux uses journald as the system logging daemon, combined with logrotate for log rotation, forming a complete log management infrastructure. This article dives deep into journald internals and configuration, advanced journalctl query techniques, log rotation strategies, remote log collection solutions, and practical analysis cases.
journald Internals
Architecture Overview
journald is systemd’s system logging component, replacing the traditional syslog (rsyslog). It receives logs from the kernel, system services, and applications, storing them in a structured binary format.
[Kernel Logs] [systemd Services] [Applications]
│ │ │
▼ ▼ ▼
[kmsg] [sd_journal_print] [syslog()/stdout]
│ │ │
└──────────┬───────┴──────────┬───────┘
▼ ▼
[journald] [/dev/log]
│
┌──────────┼──────────┐
▼ ▼ ▼
[Persistent [Runtime [Forward to
Logs] Logs] syslog]
/var/log/ /run/log/ /var/log/
journal/ journal/ messages
Log Storage Modes
# View current storage mode
$ cat /etc/systemd/journald.conf | grep Storage
#Storage=auto
# Three modes:
# auto (default): Persist if /var/log/journal exists, otherwise in-memory only
# persistent: Force persistence (auto-creates /var/log/journal)
# volatile: In-memory only (/run/log/journal)
| Mode | Storage Location | Survives Reboot | Use Case |
|---|---|---|---|
persistent | /var/log/journal | Yes | Production |
auto | /var/log/journal or /run/log/journal | Depends on directory | Default |
volatile | /run/log/journal | No | Temporary systems/security requirements |
Log Size Control
# /etc/systemd/journald.conf
[Journal]
Storage=persistent
# Global log size limits
SystemMaxUse=2G # Max persistent log usage
SystemKeepFree=4G # Ensure at least 4G free disk space
SystemMaxFileSize=100M # Max size per log file
SystemMaxFiles=100 # Keep up to 100 log files
# Runtime log (in-memory) limits
RuntimeMaxUse=200M
RuntimeKeepFree=100M
RuntimeMaxFileSize=20M
RuntimeMaxFiles=20
# Log retention time
MaxRetentionSec=2week # Keep logs for max 2 weeks
MaxFileSec=1month # Max 1 month per file
# Forwarding configuration
ForwardToSyslog=no # Do not forward to syslog (deprecated)
ForwardToKMsg=no
ForwardToConsole=no
ForwardToWall=yes # Send urgent messages to all terminals
# Log rate limiting
RateLimitIntervalSec=30s
RateLimitBurst=10000 # Max 10000 entries per 30 seconds
Viewing and Managing Log Size
# View log disk usage
$ journalctl --disk-usage
Archived and active journals take up 1.2G in the file system.
# Immediately reduce logs to specified size
$ journalctl --vacuum-size=500M
Vacuumed 123 archived journals (1.2G → 500M)
# Clean by time
$ journalctl --vacuum-time=7d
Vacuumed 45 archived journals older than 7 days
# Clean by file count
$ journalctl --vacuum-files=10
Vacuumed 90 archived journals (keeping 10 files)
# Verify log integrity
$ journalctl --verify
journalctl Query Techniques
Basic Queries
# View all logs
$ journalctl
# View current boot logs
$ journalctl -b
# View previous boot logs
$ journalctl -b -1
# View specific boot logs
$ journalctl --list-boots
IDX BOOT ID FIRST ENTRY LAST ENTRY
-2 abc123... Mon 2026-07-08 09:00:00 Mon 2026-07-08 18:00:00
-1 def456... Tue 2026-07-09 09:00:00 Tue 2026-07-09 18:00:00
0 ghi789... Wed 2026-07-10 09:00:00 Wed 2026-07-10 15:30:00
$ journalctl -b -2 # View logs from two boots ago
Time Filtering
# Today's logs
$ journalctl --since today
# Last 1 hour
$ journalctl --since "1 hour ago"
# Specific time range
$ journalctl --since "2026-07-10 09:00:00" --until "2026-07-10 12:00:00"
# Relative time
$ journalctl --since "1 hour ago" --until "10 min ago"
$ journalctl --since yesterday
$ journalctl --since "2026-07-01" --until "2026-07-10"
Filtering by Unit
# View specific service logs
$ journalctl -u nginx
$ journalctl -u sshd
$ journalctl -u docker
# Multiple services
$ journalctl -u nginx -u php-fpm
# View service logs since current boot
$ journalctl -u nginx -b
# View service logs from previous failed boot
$ journalctl -u nginx -b -1 --since "1 hour ago"
Filtering by Priority
# Log priorities
# 0: emerg - Emergency
# 1: alert - Alert
# 2: crit - Critical
# 3: err - Error
# 4: warning - Warning
# 5: notice - Notice
# 6: info - Info
# 7: debug - Debug
# View logs at specified level and above
$ journalctl -p err # err and above
$ journalctl -p warning # warning and above
$ journalctl -p 3 # Same as -p err
# Specify level range
$ journalctl -p warning..err # warning to err
Filtering by Field
# By process ID
$ journalctl _PID=12345
# By user ID
$ journalctl _UID=1000
# By GID
$ journalctl _GID=1000
# By executable
$ journalctl _COMM=nginx
$ journalctl _EXE=/usr/sbin/nginx
# By device
$ journalctl _KERNEL_DEVICE=eth0
# Custom fields (structured logging)
$ journalctl SYSLOG_IDENTIFIER=myapp
$ journalctl MESSAGE_ID=abc123
Output Format Control
# Default format
$ journalctl -u nginx -n 5
# Short format
$ journalctl -u nginx -n 5 -o short
# Precise format (with full timestamps)
$ journalctl -u nginx -n 5 -o short-precise
# JSON format (suitable for script processing)
$ journalctl -u nginx -n 5 -o json
# Pretty JSON format
$ journalctl -u nginx -n 5 -o json-pretty
# Category display
$ journalctl -u nginx -n 5 -o cat
# Shows only message content, no timestamps
# Export format (for backup/migration)
$ journalctl -u nginx --since today -o export > nginx-logs.export
Real-Time Tracking
# Real-time tracking (like tail -f)
$ journalctl -f
# Track specific service
$ journalctl -u nginx -f
# Track multiple services
$ journalctl -u nginx -u php-fpm -f
# Show only error-level real-time logs
$ journalctl -f -p err
Advanced Query Combinations
# Combined query: time + service + level
$ journalctl --since "1 hour ago" -u nginx -p err
# View kernel logs
$ journalctl -k
# View logs for a specific user
$ journalctl _UID=1000 --since today
# View killed processes
$ journalctl --since "1 hour ago" | grep -i "killed\|oom"
# Filter by message content
$ journalctl --since today | grep "connection refused"
# Output as JSON and process with jq
$ journalctl -u nginx --since "1 hour ago" -o json | jq 'select(.MESSAGE | contains("error"))'
journalctl Quick Reference
| Need | Command |
|---|---|
| Current boot logs | journalctl -b |
| Previous boot logs | journalctl -b -1 |
| Kernel logs | journalctl -k |
| Service logs | journalctl -u nginx |
| Error logs | journalctl -p err |
| Last 1 hour | journalctl --since "1 hour ago" |
| Real-time tracking | journalctl -f |
| Disk usage | journalctl --disk-usage |
| Clean up logs | journalctl --vacuum-size=500M |
| JSON output | journalctl -o json |
| Message content only | journalctl -o cat |
| Specific PID | journalctl _PID=12345 |
Log Rotation (logrotate)
Why logrotate Is Needed
journald has its own log size control, but many applications (like Nginx, MySQL) write log files directly without going through journald. These log files grow continuously and need logrotate to periodically rotate, compress, and clean them.
How logrotate Works
1. Check config files (/etc/logrotate.d/*)
2. Determine if log files meet rotation criteria
3. If criteria are met, perform rotation:
a. Rename current log (app.log → app.log.1)
b. Optional: compress old log (app.log.1 → app.log.1.gz)
c. Delete old logs exceeding retention count
d. Create new empty log file
e. Notify application to reopen log file
Main Configuration File
# /etc/logrotate.conf — global configuration
weekly # Rotate weekly
rotate 4 # Keep 4 copies
create # Create new file after rotation
compress # Compress old logs
dateext # Use date as suffix
dateformat -%Y%m%d # Date format
include /etc/logrotate.d # Include sub-config directory
Application-Level Configuration Examples
Nginx Log Rotation
# /etc/logrotate.d/nginx
/var/log/nginx/*.log {
daily
rotate 30
missingok
notifempty
compress
delaycompress # Delay compression by one cycle (for troubleshooting)
sharedscripts # Share one script for multiple logs
postrotate
if [ -f /var/run/nginx.pid ]; then
kill -USR1 $(cat /var/run/nginx.pid)
fi
endscript
}
MySQL Log Rotation
# /etc/logrotate.d/mysql
/var/log/mysql/*.log {
daily
rotate 14
missingok
create 640 mysql adm
compress
delaycompress
notifempty
sharedscripts
postrotate
mysqladmin flush-logs 2>/dev/null || true
endscript
}
Application Log Rotation
# /etc/logrotate.d/myapp
/opt/myapp/logs/*.log {
daily
rotate 30
missingok
notifempty
compress
delaycompress
copytruncate # Copy then truncate (no service interruption)
size 100M # Also rotate when exceeding 100M
dateext
dateformat -%Y%m%d-%H%M%S
}
Key Parameter Reference
| Parameter | Description | Example |
|---|---|---|
daily/weekly/monthly | Rotation frequency | daily |
rotate N | Number of copies to keep | rotate 30 |
size | Size threshold | size 100M |
compress | Compress old logs | compress |
delaycompress | Delay compression by one cycle | delaycompress |
missingok | Don’t error if log missing | missingok |
notifempty | Don’t rotate empty files | notifempty |
create | Create new file after rotation | create 640 root root |
copytruncate | Copy then truncate original file | copytruncate |
dateext | Use date suffix | dateext |
sharedscripts | Share script (run once for multiple logs) | sharedscripts |
postrotate/endscript | Script to run after rotation | postrotate |
prerotate/endscript | Script to run before rotation | prerotate |
copytruncate vs postrotate
| Method | How It Works | Advantage | Disadvantage |
|---|---|---|---|
postrotate | Rename + notify app to reopen file | No data loss | App must support signals |
copytruncate | Copy + truncate original file | No app cooperation needed | Possible log loss between copy and truncate |
Manual Execution and Debugging
# Manually run all rotations
$ logrotate /etc/logrotate.conf
# Debug mode (dry-run, no actual execution)
$ logrotate -d /etc/logrotate.d/nginx
# Force rotation
$ logrotate -f /etc/logrotate.d/nginx
# View status
$ cat /var/lib/logrotate/status
# Records last rotation time for each log
# Verbose output
$ logrotate -v /etc/logrotate.d/nginx
logrotate Cron Job
# logrotate is typically executed via cron or systemd timer
# Debian/Ubuntu: /etc/cron.daily/logrotate
# RHEL/CentOS: /etc/cron.daily/logrotate
# systemd timer approach
$ systemctl list-timers | grep logrotate
$ systemctl cat logrotate.timer
Structured Logging
Problems with Traditional Logs
# Unstructured log
Jul 10 15:30:00 server nginx: 10.0.0.1 - - [10/Jul/2026:15:30:00 +0800] "GET /api/users HTTP/1.1" 200 1234
# Hard to parse, hard to query, hard to correlate
journald Structured Logging
journald natively supports structured fields. Applications can write structured logs through the following methods:
C Program
#include <systemd/sd-journal.h>
int main() {
sd_journal_print(LOG_INFO, "User login successful");
// Structured fields
sd_journal_send(
"MESSAGE=User %s logged in from %s",
"PRIORITY=6",
"USER_ID=12345",
"USERNAME=admin",
"SOURCE_IP=10.0.0.1",
"ACTION=login",
NULL
);
return 0;
}
Python
import logging
import systemd.journal
logger = logging.getLogger('myapp')
logger.addHandler(systemd.journal.JournalHandler())
logger.setLevel(logging.INFO)
# Regular log
logger.info("User login successful")
# Structured log
logger.info("User login", extra={
'USER_ID': 12345,
'USERNAME': 'admin',
'SOURCE_IP': '10.0.0.1',
'ACTION': 'login'
})
# Query
# journalctl USER_ID=12345
# journalctl ACTION=login
Shell Script
# Write structured logs using systemd-cat
$ echo "Backup completed" | systemd-cat -t myapp -p info
# With structured fields
$ systemd-cat -t myapp -p info << EOF
BACKUP_ID=20260710
STATUS=success
DURATION=3600
FILES=12345
SIZE=10GB
Backup completed successfully
EOF
# Query
$ journalctl -t myapp BACKUP_ID=20260710
Go Program
package main
import (
"log/syslog"
"os"
)
func main() {
writer, _ := syslog.New(syslog.LOG_INFO|syslog.LOG_USER, "myapp")
logger := log.New(writer, "", 0)
logger.Println("Application started")
// Or use coreos/go-systemd library
// to write structured fields directly to journald
}
Log Correlation (Request ID)
# Generate a Request ID at the entry point, trace it through the entire call chain
$ journalctl REQUEST_ID=abc-123-def
# Can trace a request's complete logs across multiple services
# Application implementation
# 1. API gateway generates Request ID
# 2. Pass through HTTP headers to downstream services
# 3. Each service writes Request ID to journald structured fields
# 4. Use journalctl REQUEST_ID=xxx to query the full chain
Remote Log Collection
Solution Comparison
| Solution | Protocol | Reliability | Performance | Use Case |
|---|---|---|---|---|
| rsyslog + TCP | TCP | High | Medium | Traditional approach |
| journald → rsyslog | TCP/UDP | Medium | Medium | Transitional solution |
| Fluentd/Fluent Bit | TCP/HTTP | High | High | Containerized/cloud-native |
| Promtail + Loki | HTTP | High | High | Grafana ecosystem |
| Filebeat + ELK | TCP | High | High | ELK ecosystem |
journald Forwarding to rsyslog
# /etc/systemd/journald.conf
[Journal]
ForwardToSyslog=yes
# /etc/rsyslog.conf or /etc/rsyslog.d/remote.conf
# Forward to remote log server
*.* @@log-server.sre.wang:514 # TCP
# Or
*.* @log-server.sre.wang:514 # UDP
# Forward by facility
local0.* @@log-server:514
*.info;mail.none;authpriv.none @@log-server:514
systemd journal-upload
# Use systemd-journal-upload to send to remote journal server
$ systemctl enable --now systemd-journal-upload
# Configuration
# /etc/systemd/journal-upload.conf
[Upload]
URL=https://journal-collector.sre.wang/upload
# Or use push mode
# KeyFile=/etc/ssl/private/journal-upload.key
# CertificateFile=/etc/ssl/certs/journal-upload.crt
# TrustedCertificateFile=/etc/ssl/certs/ca.crt
Fluent Bit Collecting journald
# /etc/fluent-bit/fluent-bit.conf
[SERVICE]
Flush 5
Log_Level info
[INPUT]
Name systemd
Tag host.*
Systemd_Filter _SYSTEMD_UNIT=nginx.service
Systemd_Filter _SYSTEMD_UNIT=sshd.service
DB /var/log/flb_journald.db
[OUTPUT]
Name forward
Match *
Host fluentd.sre.wang
Port 24224
Log Collection Architecture
[Server A] [Server B] [Server C]
journald journald journald
│ │ │
▼ ▼ ▼
[Fluent Bit] [Fluent Bit] [Fluent Bit]
│ │ │
└──────────┬───────────────────┴──────────────────┬───────────┘
▼ ▼
[Fluentd/Loki] [Elasticsearch]
│ │
▼ ▼
[Grafana/Kibana] [Kibana Dashboard]
Log Disk Usage Control
journald Disk Control
# 1. Configure size limits
# /etc/systemd/journald.conf
SystemMaxUse=2G
SystemKeepFree=4G
MaxRetentionSec=2week
# 2. Restart journald
$ systemctl restart systemd-journald
# 3. View current usage
$ journalctl --disk-usage
# 4. Immediate cleanup
$ journalctl --vacuum-size=1G
$ journalctl --vacuum-time=7d
Application Log Disk Control
# Use logrotate to control size
# /etc/logrotate.d/app-logs
/var/log/myapp/*.log {
daily
rotate 30
compress
delaycompress
size 500M
maxsize 2G # Max 2G per file
notifempty
copytruncate
}
# Use a script to monitor log directory size
#!/bin/bash
# /usr/local/bin/log-size-monitor.sh
LOG_DIR="/var/log/myapp"
MAX_SIZE_GB=10
CURRENT_SIZE=$(du -sg $LOG_DIR | awk '{print $1}')
if [ $CURRENT_SIZE -gt $MAX_SIZE_GB ]; then
# Trigger cleanup
find $LOG_DIR -name "*.log.*" -mtime +7 -delete
journalctl --vacuum-size=500M
echo "Log cleanup triggered: ${CURRENT_SIZE}GB > ${MAX_SIZE_GB}GB" | \
systemd-cat -t log-monitor -p warning
fi
Disk Space Monitoring
# Use systemd timer for periodic checks
# /etc/systemd/system/log-monitor.service
[Unit]
Description=Log Size Monitor
[Service]
Type=oneshot
ExecStart=/usr/local/bin/log-size-monitor.sh
# /etc/systemd/system/log-monitor.timer
[Unit]
Description=Run Log Size Monitor hourly
[Timer]
OnCalendar=hourly
Persistent=true
[Install]
WantedBy=timers.target
$ systemctl enable --now log-monitor.timer
Practical Log Analysis
Case 1: SSH Brute-Force Analysis
# Count top IPs of failed SSH logins in the last 24 hours
$ journalctl -u sshd --since "24 hours ago" | \
grep "Failed" | \
awk '{print $(NF)}' | \
sort | uniq -c | sort -rn | head -20
# Count attempted usernames
$ journalctl -u sshd --since "24 hours ago" | \
grep "Failed" | \
awk '{for(i=1;i<=NF;i++) if($i=="for") print $(i+1)}' | \
sort | uniq -c | sort -rn | head -20
# Count failures per hour
$ journalctl -u sshd --since "24 hours ago" | \
grep "Failed" | \
awk '{print substr($3,1,2)":00"}' | \
sort | uniq -c
Case 2: Service Troubleshooting
# Nginx crash investigation
# 1. View logs before crash
$ journalctl -u nginx -b -1 --since "30 min ago" -n 100
# 2. Check OOM records
$ journalctl -k --since "1 hour ago" | grep -i "oom\|killed"
# 3. View service exit code
$ systemctl status nginx
$ journalctl -u nginx -n 50 -o json | jq '.[] | {MESSAGE: .MESSAGE, EXIT_STATUS: ._SYSTEMD_UNIT}'
# 4. Correlate multiple service logs
$ journalctl -u nginx -u php-fpm --since "30 min ago" -o short-precise
Case 3: Performance Issue Investigation
# View kernel performance-related logs
$ journalctl -k --since "1 hour ago" | grep -iE "hung|timeout|blocked|slow|delay"
# View I/O errors
$ journalctl -k --since "1 hour ago" | grep -iE "I/O error|read error|write error"
# View memory pressure
$ journalctl -k --since "1 hour ago" | grep -iE "oom|memory|swap|out of"
# View network anomalies
$ journalctl -k --since "1 hour ago" | grep -iE "link.*down|link.*up|carrier|timeout|reset"
Case 4: Security Audit
# View all sudo operations
$ journalctl -t sudo --since today
# View user login/logout
$ journalctl --since today | grep -E "session (opened|closed)"
# View file permission changes
$ journalctl --since today | grep -i "chmod\|chown"
# View service start/stop
$ journalctl --since today | grep -E "Started|Stopped|Failed"
# View all cron job executions
$ journalctl -t CRON --since today
# Export audit logs
$ journalctl --since "2026-07-01" --until "2026-07-10" -o export > audit-july.export
Case 5: Log Alerting
# Use systemd path to monitor log files
# /etc/systemd/system/log-alert.path
[Unit]
Description=Monitor Error Log
[Path]
PathChanged=/var/log/myapp/error.log
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/log-alert.service
[Unit]
Description=Send Alert on Error Log Change
[Service]
ExecStart=/usr/local/bin/send-alert.sh
# Use journalctl for continuous monitoring and alerting
#!/bin/bash
# /usr/local/bin/error-monitor.sh
journalctl -f -p err | while read line; do
# Send to alerting system
curl -X POST https://alerts.sre.wang/api/alert \
-H "Content-Type: application/json" \
-d "{\"message\": \"$line\"}"
done
Best Practices
Log Configuration Checklist
# /etc/systemd/journald.conf
[Journal]
Storage=persistent
Compress=yes
Seal=yes
SplitMode=uid
RateLimitIntervalSec=30s
RateLimitBurst=10000
SystemMaxUse=2G
SystemKeepFree=4G
SystemMaxFileSize=100M
MaxRetentionSec=2week
MaxFileSec=1month
ForwardToSyslog=no
ForwardToWall=yes
MaxLevelStore=debug
MaxLevelSyslog=info
Log Level Usage Guide
| Level | Use Case | Example |
|---|---|---|
| emerg | System unusable | Kernel panic |
| alert | Immediate action required | Disk full |
| crit | Critical error | Service crash |
| err | Runtime error | Request failed |
| warning | Potential problem | Disk usage 80% |
| notice | Important event | Configuration change |
| info | Normal operation | Request completed |
| debug | Debug information | Detailed parameters |
Log Writing Best Practices
# 1. Applications should write through systemd logging API
# rather than writing files directly (gains structured fields, auto-rotation)
# 2. Include sufficient context
logger -t myapp "Request completed" \
REQUEST_ID=$REQ_ID \
METHOD=$METHOD \
PATH=$PATH \
STATUS=$STATUS \
DURATION=${DURATION}ms
# 3. Do not log sensitive information
# Bad: logger "User login: password=abc123"
# Good: logger "User login: user=admin, method=publickey"
# 4. Use log levels sensibly
# - Default to info level in production
# - Temporarily switch to debug for troubleshooting
# - Don't log debug in loops
# 5. Standardize log format
# Timestamp + Level + Module + Message + Context fields
Summary
Log management is a foundational operations skill — from logs you can discover issues, locate faults, and audit actions. Key takeaways:
- journald is the standard logging system for modern Linux: Structured storage, indexed queries, automatic rotation — far superior to traditional syslog.
- journalctl is a powerful query tool: Supports filtering by time, service, priority, and fields;
-ffor real-time tracking;-o jsonfor script processing. - Log size must be controlled:
SystemMaxUseandMaxRetentionSecprevent logs from filling the disk;--vacuum-sizefor emergency cleanup. - logrotate manages application log files: Applications like Nginx and MySQL that write files directly must have logrotate configured.
- Structured logging is the trend: Use journald structured fields (REQUEST_ID, USER_ID, etc.) to enable log correlation and fast queries.
- Remote log collection is essential for large-scale operations: Fluent Bit + Loki/ELK enables centralized log management.
- Log analysis requires a toolchain:
journalctl+grep/awk/jqcombinations cover most analysis needs. - Log security cannot be overlooked: Don’t log sensitive information, control access permissions, and regularly audit login and operation logs.
The golden rule of log management: logs are data with a cost. Every log entry consumes storage, I/O, and CPU — only log what’s valuable, and ensure logs are searchable, correlatable, and alertable.