Linux

Overview 3 AM, you get paged — the production database can’t write, Nginx won’t start, SSH is sluggish. You log in and df -h shows the root partition at 100%. Every ops engineer has been there. Disk full is one of the most frequent production incidents — based on real-world operations statistics, disk-related alerts account for roughly 15%-20% of all alerts. But the 100% you see in df -h may not be the full picture:...

Overview One production incident made me take time synchronization seriously. In the middle of the night, the business team reported mass HTTPS request failures with SSL_ERROR_BAD_CERTIFICATE. We spent two hours troubleshooting—combed through network topology, firewall rules, certificate configs—nothing. Finally, someone happened to glance at the system clock: it was 12 seconds ahead of real time. Those 12 seconds made the server think client certificates hadn’t reached their notBefore time yet, so it rejected every handshake....

Overview It’s 2 AM and alerts are firing. Your service can’t connect to the database, curl times out, but ping to the IP works fine. Is it a network issue or an application issue? Nine times out of ten, it’s DNS resolution gone wrong. DNS resolution is the most fundamental yet most overlooked piece of Linux infrastructure. Nobody thinks about it until it breaks—and when it does, the entire system acts like it suddenly has amnesia: every domain becomes unrecognizable....

Overview 3 AM. Your alarm goes off. You log into the server and see a single line on the screen: Kernel panic - not syncing: Fatal exception. Then the system reboots. When you finally get back in, the crash scene is completely gone — no logs, no core dump, no call trace. You stare at systemd-logind: System is going down with nothing to work with. If you’ve been in ops for a few years, you’ve been there....

Overview 3 AM, woken up by an alert. You log into the server and find a critical configuration file has been modified, but last shows no one logged in during that window, and bash_history has nothing. You know something happened, but you don’t know who did it or how. This is when you need auditd—the audit system built into the Linux kernel. It’s like an airplane’s black box, recording every critical action on the system: who executed what command, which files were accessed, what configurations were changed, and when privilege escalation occurred....

Overview You’re running a high-frequency trading system online. P99 latency sits at 2ms normally, but occasionally spikes to 20ms. CPU usage isn’t high, memory is sufficient, network is fine. After investigation, you discover the CPU scheduler migrated a critical thread to another core, L3 cache missed entirely, and latency jumped 10x. This kind of problem can’t be solved by adding resources. The issue is “sharing” — all processes share CPU cores, the scheduler distributes freely, and nobody knows which threads are latency-sensitive....

Overview Linux Namespaces and cgroups: The Foundation of Container Technology is an essential skill in SRE operations. In production environments, mastering these techniques can significantly improve system stability and operational efficiency. Why Linux Namespaces and cgroups Matters As systems grow in scale and complexity, traditional operations approaches struggle to meet the demands of modern distributed systems. Linux Namespaces and cgroups helps operations teams: Rapid Problem Resolution: Systematic tools and methods reduce troubleshooting time Improved System Visibility: Establish comprehensive monitoring and observability Proactive Fault Prevention: Identify and fix potential risks before they cause outages Resource Optimization: Allocate and schedule resources efficiently Core Concepts and Principles Basic Concepts The core of Linux Namespaces and cgroups lies in establishing standardized processes and automated toolchains....

Overview Nginx Performance Tuning: From Configuration to Kernel Parameters is an essential skill in SRE operations. In production environments, mastering these techniques can significantly improve system stability and operational efficiency. Why Nginx Performance Tuning Matters As systems grow in scale and complexity, traditional operations approaches struggle to meet the demands of modern distributed systems. Nginx Performance Tuning helps operations teams: Rapid Problem Resolution: Systematic tools and methods reduce troubleshooting time Improved System Visibility: Establish comprehensive monitoring and observability Proactive Fault Prevention: Identify and fix potential risks before they cause outages Resource Optimization: Allocate and schedule resources efficiently Core Concepts and Principles Basic Concepts The core of Nginx Performance Tuning lies in establishing standardized processes and automated toolchains....

Overview Linux System Call Tracing: strace and ltrace Debugging is an essential skill in SRE operations. In production environments, mastering these techniques can significantly improve system stability and operational efficiency. Why Linux System Call Tracing Matters As systems grow in scale and complexity, traditional operations approaches struggle to meet the demands of modern distributed systems. Linux System Call Tracing helps operations teams: Rapid Problem Resolution: Systematic tools and methods reduce troubleshooting time Improved System Visibility: Establish comprehensive monitoring and observability Proactive Fault Prevention: Identify and fix potential risks before they cause outages Resource Optimization: Allocate and schedule resources efficiently Core Concepts and Principles Basic Concepts The core of Linux System Call Tracing lies in establishing standardized processes and automated toolchains....

Overview The Linux firewall has evolved from ipfwadm → ipchains → iptables → nftables. All are based on the Netfilter framework, but the higher-level syntax and management approach have continuously improved. This article starts from the Netfilter architecture and dives into iptables’ five chains and four tables, nftables’ advantages and usage, NAT/port forwarding, connection tracking, and production performance optimization. Netfilter Framework Architecture Overview Netfilter is a packet processing framework in the Linux kernel that implements packet filtering, address translation, connection tracking, and other functions by mounting hooks at key positions in the kernel network stack....