3 Days of Approval, Still Crashed: Replacing Manual CAB with Error Budget Gates Cut Change-Related Incidents by 60%

Overview 1:47 AM. Phone buzzes. The alert group explodes with 200+ messages. Order success rate drops from 99.8% to 92%. The on-call engineer restarts the service, rolls back the version, and fights for 40 minutes to restore service. Next day’s postmortem: root cause was a one-line database connection pool config change pushed during the afternoon release. This change went through the full approval process — developer submission, test verification, manager sign-off, SRE review....

August 6, 2026 · 23 mins · 4818 words · Xu Baojin

Change Management: Canary Release and Rollback Strategies

The Role of Change Management in SRE Google SRE identified an iron rule: approximately 70% of production incidents are directly caused by changes. Whether it’s code deployment, configuration modification, infrastructure adjustment, or dependency upgrades, every change injects uncertainty into the system. Change management is therefore not bureaucratic red tape — it’s the first line of defense in SRE reliability engineering. The core objectives of change management can be summarized in three points:...

February 11, 2025 · 9 mins · 1902 words · XuBaojin