Waiting on Three Approvals to Fight a Fire at 3 AM: 6 Engineering Decisions for Production Break-Glass Access and Audit Trails
Overview At 3:17 AM, my phone rang during a shift at an e-commerce platform. Payment success rate had dropped from 99.7% to 89%, and the order pipeline was starting to back up. The on-call engineer quickly pinned down the root cause: during the previous canary rollout, a single network ACL rule had been deleted by mistake, so payment callback traffic could no longer reach the backend cluster. The fix itself was trivial — add the rule back, or revert that config commit....