Pods OOM Killed at 3 AM: From working_set Blind Spots to a Memory Warning System in 5 Production Decisions
Overview 3 AM. The alert group exploded. Five pods were OOMKilled and restarted within 10 minutes. The on-call engineer opened Grafana, and every pod’s working_set was still 200MB below its limit—80% utilization. Monitoring looked fine. The pods died anyway. This is not an isolated case. When I was doing a Kubernetes migration for a new-energy logistics platform, I dealt with this exact pattern—“monitoring is green but pods die”—more than 10 times....