Pods OOM Killed at 3 AM: From working_set Blind Spots to a Memory Warning System in 5 Production Decisions

Overview 3 AM. The alert group exploded. Five pods were OOMKilled and restarted within 10 minutes. The on-call engineer opened Grafana, and every pod’s working_set was still 200MB below its limit—80% utilization. Monitoring looked fine. The pods died anyway. This is not an isolated case. When I was doing a Kubernetes migration for a new-energy logistics platform, I dealt with this exact pattern—“monitoring is green but pods die”—more than 10 times....

October 9, 2026 · 21 mins · 4427 words · Xu Baojin

Linux Memory Management Mechanisms and Tuning in Practice

Overview Memory is one of the most precious resources in a Linux system. Understanding how the kernel manages memory not only helps you troubleshoot OOM and memory leak issues in production, but also enables better decisions in capacity planning and performance tuning. This article starts from the virtual memory model and covers core topics including Page Cache, Swap policies, OOM Killer principles, cgroup v2 memory limits, slab/shmem tuning, with multiple production case studies....

May 9, 2024 · 13 mins · 2564 words · ABao