Terraform

Overview Monday stand-up, our platform engineering team at a mobility platform was reviewing last week’s incidents: 17 state lock conflicts, two production environments broken by a “helpful” colleague running apply manually, and one set of credentials sitting in plaintext in CI logs. The root cause wasn’t a single operator error—it was our self-hosted Terraform backend (Git repo + S3 state bucket + DynamoDB lock + hand-made CI scripts) that had stopped holding up once the team grew from 5 to 30 engineers....

Overview A team has been running Terraform for three years. Their main module has grown to 3,000 lines of HCL, with deeply nested variable definitions, multiple providers, and for_each loops that make code review a nightmare. Every change is like defusing a bomb—the pull request gets reviewed five times, tested in three environments, and the final plan output scrolls for 200+ lines before anyone clicks apply. The last guy who understood the module structure left two months ago, and the handover doc is an abandoned Notion page....

Overview 2 AM, the alert channel exploded. A web server that went live two hours ago was at 100% CPU. SSH in — Nginx wasn’t installed, but an old Apache was squatting on port 80. Checked the deploy logs: Terraform had run terraform apply ten minutes ago, rebuilding the machine (someone changed instance_type), but Ansible’s configuration playbook never followed — fresh machine, all software configs gone. This isn’t an isolated incident....

Overview Last year I was setting up an IaC system for an e-commerce client. When I took over, their Terraform codebase looked roughly like this: three environments (dev/staging/prod), each with its own copy of the configuration, totaling over 3,000 lines—90% of which was copy-paste. Changing a VPC CIDR meant editing three files in sync. Miss one, and you get environment drift. The worst part: a security group rule was missing in prod for three months without anyone noticing....

Overview I took over a project last year where a former colleague’s Terraform code was running fine. One Monday morning, he changed a single line of instance_type. terraform plan looked clean. apply succeeded. Then the system blew up. The problem wasn’t a syntax error — the instance_type referenced an AMI that didn’t exist in the target availability zone. plan passed because it only checks “is this logically valid,” not “does this resource actually exist on the cloud....

Overview 2 AM, my phone exploded. The ops channel for a ride-hailing project was flooding with alerts: RDS failover failed, EIP binding status abnormal, security group rules missing. After a long investigation, the root cause was someone manually modifying a security group rule through the console. Terraform’s state file was out of sync with the actual cloud state, and a routine terraform apply “corrected” the manually modified resources back to their old configuration—overwriting an emergency hotfix made on the production server....

Overview Monitoring as Code: Managing Alert Rules with Terraform and YAML is an essential skill in SRE operations. In production environments, mastering these techniques can significantly improve system stability and operational efficiency. Why Monitoring as Code Matters As systems grow in scale and complexity, traditional operations approaches struggle to meet the demands of modern distributed systems. Monitoring as Code helps operations teams: Rapid Problem Resolution: Systematic tools and methods reduce troubleshooting time Improved System Visibility: Establish comprehensive monitoring and observability Proactive Fault Prevention: Identify and fix potential risks before they cause outages Resource Optimization: Allocate and schedule resources efficiently Core Concepts and Principles Basic Concepts The core of Monitoring as Code lies in establishing standardized processes and automated toolchains....

Manually logging into cloud consoles to create servers, databases, and networks — this approach barely works when resources are few, but once environments grow complex, you end up unable to modify, clean up, or explain your infrastructure. Infrastructure as Code (IaC) uses code to describe infrastructure, making resource creation, modification, and destruction versionable, reviewable, and reusable. Terraform is currently the most popular IaC tool. This article covers everything from concepts to practice....