3 Untracked Resources at 2 AM: 6 Engineering Decisions for Terraform Drift Detection and Remediation
Overview At 2 AM on a Wednesday, my on-call phone buzzed during a shift at a mobility platform. The alert channel lit up with a Prometheus notification — the production ALB (Application Load Balancer) showed abnormal traffic distribution. One backend ECS instance had its weight suddenly set to 0, even though the machine was alive and healthy. Investigation revealed: someone had manually modified the ALB listener rule through the cloud console the previous day, setting one ECS instance’s weight from 50 to 0....