14-Day Change Freeze Still Blew Up P0: 5 Engineering Decisions from Manual Freeze to Dynamic Change Windows
Overview Fourteen days before Double 11 one year, an e-commerce platform initiated a full change freeze—all non-emergency changes were frozen, CI/CD pipeline gates were closed, and only a security patch channel remained. The ops team breathed a sigh of relief. This time, it would be stable. At 2 AM on the day of the big sale, the payment pipeline P0 alert blew up. The root cause wasn’t new code. It was a configuration change deployed 3 days before the freeze—a rate-limiting rule’s whitelist had the wrong IP range....