Kubernetes

Overview 2 AM. A ride-hailing platform’s payment service does a blue-green switch. The on-call engineer runs kubectl rollout to flip the Service selector, shifting traffic from blue to green. Everything looks fine — Pods are Ready, health checks pass, response latency looks normal. Three minutes later, the DBA chat explodes. Database connections spike from 200 to 1,200, the connection pool is maxed out, and five downstream services all time out. Root cause: the green environment’s app version updated the ORM framework, and the default pool size jumped from 20 to 100....

Overview The night before a major promotional event, all Pods in the test environment were stuck in Pending. kubectl top nodes showed node CPU usage at 35% and memory under 50%. The Deployment YAML had reasonable requests and limits. The problem? Not a resource shortage — the “ledger” was maxed out. A namespace’s ResourceQuota had hit its ceiling, and the admission controller was rejecting all new Pod creation requests. This incident forced me to re-examine the entire chain of K8s namespace-level resource governance....

Overview 3 AM. The alert group exploded. Five pods were OOMKilled and restarted within 10 minutes. The on-call engineer opened Grafana, and every pod’s working_set was still 200MB below its limit—80% utilization. Monitoring looked fine. The pods died anyway. This is not an isolated case. When I was doing a Kubernetes migration for a new-energy logistics platform, I dealt with this exact pattern—“monitoring is green but pods die”—more than 10 times....

Overview 2:47 AM, the alert chat exploded. A payment service’s P99 latency spiked from 80ms to 1200ms after a new version went live. The on-call engineer hit rollback immediately. kubectl rollout undo executed successfully, old version Pods came back up, traffic shifted back. The result? The service went completely down — because the new version had run a database migration script that changed the status column in the orders table from varchar(20) to tinyint....

Overview Migrating 120+ microservices from traditional VM deployments to Kubernetes sounds like a standard project: write Dockerfiles, build a cluster, migrate services, shift traffic. In planning meetings, everyone thinks three months is enough. In practice, you discover that the real challenges aren’t about “how to get services running” — that’s just helm install — but about pitfalls nobody warned you about: circular dependencies in service graphs, in-memory state loss during stateful migration, CNI performance cliffs at scale, DNS caching traps during canary traffic shifts, ConfigMap bloat, and monitoring metric gaps after migration....

Overview Netflix says chaos engineering reduced major production incidents by 70%. Many teams have seen this number, but those who have actually run fault injection in production are rare. The reason is simple: fear. Inject a network partition and Pods freeze—what then? Simulate disk full and the database crashes—then what? Manually injecting faults at 3 AM and scrambling to roll back—that’s not a drill, that’s manufacturing an incident. During a K8s migration for a new-energy logistics platform with 120+ microservices, I needed to validate disaster recovery capabilities before the full cutover....

Overview 1 AM. You get an alert: GitLab CI pipeline queue has 47 pending jobs. The dev chat explodes—“code pushed 40 minutes ago, still not running"“is the Runner down?““just add more machines!” You check the console: all 3 Runners are at capacity, each running 10 concurrent jobs, 30 slots fully occupied. Add machines? Docker Machine executor needs 3 minutes to spin up a new EC2, then another 2 minutes to become Ready....

Overview “Fully managed control plane” — this phrase makes many teams think migrating to an ACK managed cluster means they can kick back and relax. In reality, managed only takes care of etcd, kube-apiserver, kube-controller-manager, and kube-scheduler. Data plane problems? Still all yours. I led a migration of 120+ microservices from a self-hosted K8s cluster to ACK Managed Cluster Pro. The migration took 3 months with zero-downtime cutover — but the first month after switching, 3 AM alerts never stopped....

Overview You manage a K8s cluster where the API Server has --anonymous-auth=true, etcd certificates are world-readable at 644, and the kubelet --read-only-port=10255 is still open. Each of these looks like a “minor issue” on its own, but an attacker who gets one foothold can pivot through the entire cluster. This isn’t hypothetical — the CNCF 2025 annual survey showed that over 90% of production K8s clusters have at least one CIS Benchmark-level configuration deficiency....

Overview Production blows up, you open Kibana to search logs, and discover the critical Pod was OOM-killed 3 minutes ago—the old logs vanished with the container. This is extremely common in K8s environments: Pods are ephemeral, and logs can’t disappear with them. K8s log collection is fundamentally different from traditional VM environments. In a traditional setup, logs sit quietly in /var/log/ and you just SSH in to read them. In K8s, Pods can be scheduled to any node at any time, destroyed and recreated at will, with logs scattered across the entire cluster....