Infrastructure as Code

Overview At 2 AM on a Wednesday, my on-call phone buzzed during a shift at a mobility platform. The alert channel lit up with a Prometheus notification — the production ALB (Application Load Balancer) showed abnormal traffic distribution. One backend ECS instance had its weight suddenly set to 0, even though the machine was alive and healthy. Investigation revealed: someone had manually modified the ALB listener rule through the cloud console the previous day, setting one ECS instance’s weight from 50 to 0....

Overview Monday stand-up, our platform engineering team at a mobility platform was reviewing last week’s incidents: 17 state lock conflicts, two production environments broken by a “helpful” colleague running apply manually, and one set of credentials sitting in plaintext in CI logs. The root cause wasn’t a single operator error—it was our self-hosted Terraform backend (Git repo + S3 state bucket + DynamoDB lock + hand-made CI scripts) that had stopped holding up once the team grew from 5 to 30 engineers....

Overview A team has been running Terraform for three years. Their main module has grown to 3,000 lines of HCL, with deeply nested variable definitions, multiple providers, and for_each loops that make code review a nightmare. Every change is like defusing a bomb—the pull request gets reviewed five times, tested in three environments, and the final plan output scrolls for 200+ lines before anyone clicks apply. The last guy who understood the module structure left two months ago, and the handover doc is an abandoned Notion page....

Overview Last year I was setting up an IaC system for an e-commerce client. When I took over, their Terraform codebase looked roughly like this: three environments (dev/staging/prod), each with its own copy of the configuration, totaling over 3,000 lines—90% of which was copy-paste. Changing a VPC CIDR meant editing three files in sync. Miss one, and you get environment drift. The worst part: a security group rule was missing in prod for three months without anyone noticing....

Overview At 3 AM, your phone vibrates. An alert message: a major cloud provider’s East China region storage service is experiencing widespread unavailability. Your core business is fully deployed in this region, with the primary database there too. The outage has lasted 12 minutes, customers are complaining, and your boss is asking “how long to recover” in the group chat. You know the truth: the cross-region disaster recovery plan was reviewed six months ago but got cut due to “high costs....