systemd Service Management Deep Guide

systemd Architecture Overview

systemd is the de facto standard init system for modern Linux distributions, having replaced SysVinit as the default init in most mainstream distributions since 2015. It is not merely a “service starter” but a complete system and service manager.

Core Unit Types

systemd manages system resources through units, with each unit type corresponding to a specific resource type:

Unit Type	Extension	Purpose
service	`.service`	System services (daemons)
socket	`.socket`	IPC sockets (supports socket activation)
timer	`.timer`	Scheduled tasks (replaces cron)
target	`.target`	Service groups (similar to traditional runlevels)
mount	`.mount`	Filesystem mount points
device	`.device`	Kernel devices
path	`.path`	File path monitoring (triggers service when a file appears)
slice	`.slice`	cgroup resource allocation hierarchy

Mapping Targets to Traditional Runlevels

# View the current default target
systemctl get-default
# Typical output: multi-user.target (corresponds to runlevel 3, multi-user command-line mode)

# Switch to graphical interface (corresponds to runlevel 5)
sudo systemctl isolate graphical.target

# Runlevel to target mapping:
# runlevel 0 → poweroff.target
# runlevel 1 → rescue.target
# runlevel 3 → multi-user.target
# runlevel 5 → graphical.target
# runlevel 6 → reboot.target

cgroup Integration

systemd is the primary consumer of cgroup v1/v2. Each service automatically creates a cgroup, with a path format like:

# cgroup v2 path example
/sys/fs/cgroup/system.slice/nginx.service/

# View cgroup information for a service
systemctl status nginx

# The output will include:
#    CGroup: /system.slice/nginx.service
#            ├─4567 "nginx: master process /usr/sbin/nginx"
#            └─4568 "nginx: worker process"

For the complete architecture and configuration reference of systemd, see the systemd official documentation.

Deep Dive into Service Unit Files

Unit files use an INI-style format, divided into three main sections. Below is a section-by-section analysis using a complete Nginx service file as an example:

[Unit] Section

[Unit]
Description=Nginx HTTP Server
Documentation=man:nginx(8)
Documentation=https://nginx.org/en/docs/
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target
Conflicts=apache2.service

Field	Description
`Description`	Human-readable description of the service, shown in `systemctl status` output
`Documentation`	Documentation references, supports `man:`, `http:`, `info:` prefixes
`After`	Current unit starts after the specified units (ordering only, no dependency)
`Before`	Current unit starts before the specified units
`Requires`	Hard dependency: if the dependency fails to start, the current unit also fails
`Wants`	Soft dependency: attempts to start the dependency; failure does not affect the current unit
`Requisite`	Hard dependency (no wait): the dependency must already be running, otherwise fails immediately
`Conflicts`	Mutual exclusion: the specified unit cannot run simultaneously with the current unit
`PartOf`	When the dependency unit is stopped/restarted, the current unit follows suit
`BindsTo`	Strongest binding: when the dependency stops, the current unit also stops, and does not auto-restart

[Service] Section

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/usr/sbin/nginx -s reload
ExecStop=/usr/sbin/nginx -s stop
Restart=on-failure
RestartSec=5s
TimeoutStartSec=30
TimeoutStopSec=30
KillMode=mixed
KillSignal=SIGQUIT
LimitNOFILE=65535

The Type field determines how systemd judges whether a service has started successfully — this is the most error-prone area:

Type Value	Start Success Signal	Use Case
`simple`	ExecStart returns immediately	Foreground daemons
`forking`	The forked child of ExecStart exits, parent returns	Traditional daemons (nginx, sshd)
`oneshot`	ExecStart finishes execution	One-time scripts
`notify`	Service sends `READY=1` via `sd_notify()`	Services supporting systemd notification
`idle`	Waits until all jobs are done before starting	Console services that should not slow down boot

Key start/stop commands:

ExecStartPre: Preparation commands run before the main process starts (e.g., config check)
ExecStartPost: Commands run after the main process starts
ExecStopPre: Commands run before stopping
ExecStop: Stop command (if omitted, KillSignal is sent)
ExecStopPost: Cleanup commands run after stopping

Restart policies:

Value	Restart Condition
`no`	Do not restart (default)
`on-success`	Restart only when exit code is 0
`on-failure`	Restart on non-zero exit code or killed by signal
`on-abnormal`	Restart when killed by signal or timeout
`on-watchdog`	Restart on watchdog timeout
`always`	Always restart

[Install] Section

[Install]
WantedBy=multi-user.target
Alias=nginx.service

Field	Description
`WantedBy`	Creates a `.wants` symlink to the specified target when `systemctl enable` is run
`RequiredBy`	Creates a `.requires` symlink when `systemctl enable` is run
`Alias`	Service alias
`Also`	Also enables the specified other units

Dependency Management: Requires/Wants/After/Before Differences

This is the most confusing concept in systemd. The key distinction is that dependencies and ordering are two independent dimensions.

Dependencies (Requires / Wants)

Dependencies determine “should this be started alongside”:

# Hard dependency: network must start successfully, otherwise this service fails
Requires=network.target

# Soft dependency: prefer network to start, but if it fails, this service still starts
Wants=network.target

Ordering (After / Before)

Ordering determines “who goes first” but does not establish a dependency:

# This service starts after network.target, but does not depend on it
# If network.target is not in the start list, this service still starts
After=network.target

Common Pitfalls

Pitfall 1: Only specifying After without Wants/Requires

# Incorrect: assuming this will start network
[Unit]
After=network.target
# In reality, network.target may not be pulled in at all

# Correct: declare both dependency and ordering
[Unit]
Wants=network.target
After=network.target

Pitfall 2: Requires does not guarantee ordering

# Incorrect: Requires ensures the dependency, but does not guarantee network starts first
[Unit]
Requires=network.target
# This service might start before network is fully ready

# Correct
[Unit]
Requires=network.target
After=network.target

Pitfall 3: Transitivity of Requires

If A Requires B and B fails to start, systemd will attempt to start all units that B depends on. This can cause cascading failures. For “try to start but don’t affect me” scenarios, Wants is safer.

Verifying Dependency Chains

# View the dependency tree of a service
systemctl list-dependencies nginx.service

# View reverse dependencies (who depends on nginx)
systemctl list-dependencies --reverse nginx.service

# View startup ordering
systemctl list-dependencies --after nginx.service
systemctl list-dependencies --before nginx.service

Resource Control: CPU/Memory/BlockIO Limits

systemd implements service-level resource control through cgroups, without the need to manually configure cgroups.

cgroup v1 vs v2

# Check the current cgroup version
stat -fc %T /sys/fs/cgroup/
# Output cgroup2fs → cgroup v2
# Output tmpfs → cgroup v1

CPU Limits

[Service]
# cgroup v2 syntax
CPUQuota=200%          # Limit to at most 2 CPU cores
CPUWeight=500          # CPU weight (default 100, range 1-10000)

# cgroup v1 syntax (backward compatible)
CPUAccounting=true
CPUQuota=200%
CPUShares=512

Memory Limits

[Service]
# cgroup v2
MemoryMax=2G           # Hard limit: exceeding triggers OOM
MemoryHigh=1536M       # Soft limit: memory reclaim begins above this
MemorySwapMax=512M     # Swap usage limit

# cgroup v1 (some parameter names differ)
MemoryLimit=2G

BlockIO Limits

[Service]
# cgroup v2
IOWeight=500                        # IO weight (1-10000, default 100)
IOReadBandwidthMax=/dev/sda 10M     # Read bandwidth limit
IOWriteBandwidthMax=/dev/sda 5M     # Write bandwidth limit

# cgroup v1
BlockIOWeight=500
BlockIOReadBandwidth=/dev/sda 10485760

Dynamically Adjusting Resource Limits

systemd supports runtime modification of resource limits without restarting the service:

# Temporarily adjust nginx memory limit to 1G
systemctl set-property nginx.service MemoryMax=1G

# Make it permanent (written to config file)
systemctl set-property nginx.service MemoryMax=1G --runtime=no

# View current resource usage
systemctl show nginx.service -p CPUUsageNSec -p MemoryCurrent -p CPUQuotaPerSecUSec

journalctl Log Management in Practice

systemd’s logging system, journald, is a structured logging system that supports indexing, filtering, and persistence.

Basic Queries

# View logs for a specific service
journalctl -u nginx.service

# View today's logs
journalctl --since today

# View the last 1 hour
journalctl --since "1 hour ago"

# Follow logs in real time (similar to tail -f)
journalctl -u nginx.service -f

# View logs from the previous boot
journalctl -b -1

# View logs from the current boot
journalctl -b 0

Advanced Filtering

# Filter by priority (0-7, 0=emerg, 7=debug)
journalctl -p err              # Show error and above only
journalctl -p warning          # Show warning and above only

# By process PID
journalctl _PID=4567

# By executable file path
journalctl /usr/sbin/nginx

# Combined filtering (AND relation)
journalctl -u nginx.service -p err --since "1 hour ago"

# Filter by log field (structured log fields)
journalctl SYSLOG_FACILITY=10

# JSON output format (suitable for script processing)
journalctl -u nginx.service -o json | python3 -m json.tool

Log Persistence

By default, journald stores logs in /run/log/journal/ (in memory), which are lost on reboot. Production environments must configure persistence:

# Create the persistent log directory
sudo mkdir -p /var/log/journal/
sudo systemd-tmpfiles --create --prefix /var/log/journal

# Restart journald
sudo systemctl restart systemd-journald

Configuration file /etc/systemd/journald.conf:

[Journal]
Storage=persistent          # Persist to disk
Compress=yes                # Compress old logs
SystemMaxUse=2G             # Maximum 2G for persistent logs
SystemMaxFileSize=100M      # Maximum 100M per log file
MaxRetentionSec=30day       # Retain logs for 30 days
ForwardToSyslog=no          # Do not forward to syslog (avoid duplication)

# Restart to apply configuration
sudo systemctl restart systemd-journald

# Verify disk usage
journalctl --disk-usage
# Example output: Archived and active journals take up 1.2G in the file system.

Remote Log Collection

systemd natively supports sending logs to a remote server via systemd-journal-remote:

# Server-side installation
sudo apt install -y systemd-journal-remote
sudo systemctl enable --now systemd-journal-remote.service

# Client-side forwarding configuration
sudo apt install -y systemd-journal-remote

# Edit /etc/systemd/journal-upload.conf
# [Upload]
# URL=https://log-collector.internal:19532

# Start client-side log forwarding
sudo systemctl enable --now systemd-journal-upload.service

Practical: Writing a Production-Grade systemd Service File

Below is a production-grade application service configuration example that comprehensively applies all the concepts covered in this article:

[Unit]
Description=MyApp Production Service
Documentation=https://wiki.internal/myapp/deployment
After=network-online.target postgresql.service redis.service
Wants=network-online.target
Requires=postgresql.service redis.service

[Service]
Type=notify
NotifyAccess=main

User=myapp
Group=myapp

WorkingDirectory=/opt/myapp
EnvironmentFile=/etc/myapp/env
Environment=GIN_MODE=release
Environment=APP_ENV=production

# Start commands
ExecStartPre=/opt/myapp/bin/myapp migrate
ExecStart=/opt/myapp/bin/myapp server
ExecReload=/bin/kill -HUP $MAINPID

# Restart policy
Restart=on-failure
RestartSec=5s
StartLimitInterval=60
StartLimitBurst=3
TimeoutStartSec=60
TimeoutStopSec=30
TimeoutAbortSec=30

# Process management
KillMode=mixed
KillSignal=SIGTERM
SendSIGKILL=yes
WatchdogSec=60

# Resource limits
CPUQuota=300%
MemoryMax=4G
MemoryHigh=3G
LimitNOFILE=100000
LimitNPROC=65535
TasksMax=4096

# Security hardening
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/myapp /var/log/myapp
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=true
LockPersonality=true
RestrictRealtime=true
RestrictSUIDSGID=true
RemoveIPC=true
CapabilityBoundingSet=
AmbientCapabilities=

# Logging
SyslogIdentifier=myapp
LogLevelMax=info

[Install]
WantedBy=multi-user.target

Key design points of this configuration:

Type=notify + WatchdogSec: The application proactively reports readiness via sd_notify(), and systemd also monitors heartbeats, automatically restarting if the application hangs.
StartLimitBurst=3: Limits restarts to at most 3 times within 60 seconds, preventing crash loops.
KillMode=mixed: The main process receives SIGTERM, child processes receive SIGKILL, ensuring graceful shutdown.
Security hardening: A series of Protect* and Restrict* directives implement service sandboxing, making escape difficult even if the application is compromised.
ReadWritePaths: Precisely declares writable paths under ProtectSystem=strict, minimizing the filesystem attack surface.

Deployment:

# Install the service file
sudo cp myapp.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable myapp.service
sudo systemctl start myapp.service

# Verify status
systemctl status myapp.service
journalctl -u myapp.service -f

Summary

systemd is more than just an init system — it is a complete framework for Linux service management. Mastering unit file structure, the distinction between dependencies and ordering, cgroup resource control, and journalctl log management is fundamental for operations engineers.

In production environments, every custom service should have resource limits and security hardening configured — this is not optional, but a basic infrastructure requirement. systemd provides a rich set of directives to achieve these goals; the key is understanding the meaning of each field and combining them correctly.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

systemd official documentation — freedesktop.org, referenced for systemd official documentation

systemd Architecture Overview#

Core Unit Types#

Mapping Targets to Traditional Runlevels#

cgroup Integration#

Deep Dive into Service Unit Files#

[Unit] Section#

[Service] Section#

[Install] Section#

Dependency Management: Requires/Wants/After/Before Differences#

Dependencies (Requires / Wants)#

Ordering (After / Before)#

Common Pitfalls#

Verifying Dependency Chains#

Resource Control: CPU/Memory/BlockIO Limits#

cgroup v1 vs v2#

CPU Limits#

Memory Limits#

BlockIO Limits#

Dynamically Adjusting Resource Limits#

journalctl Log Management in Practice#

Basic Queries#

Advanced Filtering#

Log Persistence#

Remote Log Collection#

Practical: Writing a Production-Grade systemd Service File#

Summary#

References & Acknowledgments#