Overview 3 AM. Your alarm goes off. You log into the server and see a single line on the screen: Kernel panic - not syncing: Fatal exception. Then the system reboots. When you finally get back in, the crash scene is completely gone — no logs, no core dump, no call trace. You stare at systemd-logind: System is going down with nothing to work with. If you’ve been in ops for a few years, you’ve been there....
Overview 3 AM, woken up by an alert. You log into the server and find a critical configuration file has been modified, but last shows no one logged in during that window, and bash_history has nothing. You know something happened, but you don’t know who did it or how. This is when you need auditd—the audit system built into the Linux kernel. It’s like an airplane’s black box, recording every critical action on the system: who executed what command, which files were accessed, what configurations were changed, and when privilege escalation occurred....
Overview 2 AM. Your phone rings. The order system is timing out across the board — CPU is fine, memory is fine, disk I/O is fine. Restarting services does nothing. Rolling back does nothing. You stare at the dashboard; every metric is green. Only the users are screaming. Nine times out of ten, it is a network-layer problem. And what you need is a pair of eyes that can actually see the packets....
Overview You’re running a high-frequency trading system online. P99 latency sits at 2ms normally, but occasionally spikes to 20ms. CPU usage isn’t high, memory is sufficient, network is fine. After investigation, you discover the CPU scheduler migrated a critical thread to another core, L3 cache missed entirely, and latency jumped 10x. This kind of problem can’t be solved by adding resources. The issue is “sharing” — all processes share CPU cores, the scheduler distributes freely, and nobody knows which threads are latency-sensitive....
Overview eBPF (Extended Berkeley Packet Filter) is one of the most revolutionary technologies in the Linux kernel over the past decade. It allows developers to safely and efficiently run custom programs in kernel space — without modifying kernel source code or loading kernel modules. From network packet filtering to syscall tracing, from performance analysis to security auditing, eBPF has become the cornerstone of modern observability and network data planes. This article starts with the fundamental principles of eBPF, progressively covering development environment setup, program type selection, a practical comparison between BCC and libbpf+CO-RE, and finally a complete production-grade eBPF tool development workflow....
Overview The Linux firewall has evolved from ipfwadm → ipchains → iptables → nftables. All are based on the Netfilter framework, but the higher-level syntax and management approach have continuously improved. This article starts from the Netfilter architecture and dives into iptables’ five chains and four tables, nftables’ advantages and usage, NAT/port forwarding, connection tracking, and production performance optimization. Netfilter Framework Architecture Overview Netfilter is a packet processing framework in the Linux kernel that implements packet filtering, address translation, connection tracking, and other functions by mounting hooks at key positions in the kernel network stack....
eBPF: The Kernel Programmability Revolution Traditional performance analysis tools fall into two categories: “overview” tools like top, vmstat, and iostat that tell you what’s happening at the system level but lack detail; and “tracing” tools like strace and gdb that show detail but incur massive overhead, making them impractical for production. eBPF (Extended Berkeley Packet Filter) changed everything. It allows you to run sandboxed programs in the kernel without modifying kernel source code or loading kernel modules....
Overview In today’s era of cloud-native and containerized technologies, Linux cgroups (control groups) serve as the kernel-level foundation for resource isolation and limiting. From Docker container memory limits to Kubernetes Pod CPU Requests/Limits, the underlying mechanism relies on cgroups. However, cgroup v1’s multi-hierarchy architecture, inconsistent controller behavior, and confusing thread model have caused numerous operational headaches in production. cgroup v2, as a complete reimagining of v1, adopts a unified hierarchy architecture that fundamentally addresses v1’s design flaws....
Overview The process scheduler is a core component of the operating system kernel — it determines which process runs on which CPU and for how long. Linux has used CFS (Completely Fair Scheduler) as its default scheduler since 2.6.23, and after years of evolution, introduced EEVDF (Earliest Eligible Virtual Deadline First) to replace CFS in the 6.6 kernel. This article provides an in-depth analysis of CFS principles, nice/cgroup CPU control, real-time scheduling, CPU affinity, and other core topics, with production tuning experience....
Overview Memory is one of the most precious resources in a Linux system. Understanding how the kernel manages memory not only helps you troubleshoot OOM and memory leak issues in production, but also enables better decisions in capacity planning and performance tuning. This article starts from the virtual memory model and covers core topics including Page Cache, Swap policies, OOM Killer principles, cgroup v2 memory limits, slab/shmem tuning, with multiple production case studies....