Database Connection Pool Blew Up Mid-Traffic-Shift: 6 Production-Grade Decisions for Blue-Green and Canary Deployment with Traffic Governance

Overview 2 AM. A ride-hailing platform’s payment service does a blue-green switch. The on-call engineer runs kubectl rollout to flip the Service selector, shifting traffic from blue to green. Everything looks fine — Pods are Ready, health checks pass, response latency looks normal. Three minutes later, the DBA chat explodes. Database connections spike from 200 to 1,200, the connection pool is maxed out, and five downstream services all time out. Root cause: the green environment’s app version updated the ORM framework, and the default pool size jumped from 20 to 100....

October 17, 2026 · 23 mins · 4898 words · Xu Baojin

Rollback Made It Worse: 5-Layer CI/CD Auto-Rollback Architecture and 4 Production Pitfalls

Overview 2:47 AM, the alert chat exploded. A payment service’s P99 latency spiked from 80ms to 1200ms after a new version went live. The on-call engineer hit rollback immediately. kubectl rollout undo executed successfully, old version Pods came back up, traffic shifted back. The result? The service went completely down — because the new version had run a database migration script that changed the status column in the orders table from varchar(20) to tinyint....

October 3, 2026 · 22 mins · 4596 words · Xu Baojin