3 Untracked Resources at 2 AM: 6 Engineering Decisions for Terraform Drift Detection and Remediation

Overview

At 2 AM on a Wednesday, my on-call phone buzzed during a shift at a mobility platform. The alert channel lit up with a Prometheus notification — the production ALB (Application Load Balancer) showed abnormal traffic distribution. One backend ECS instance had its weight suddenly set to 0, even though the machine was alive and healthy. Investigation revealed: someone had manually modified the ALB listener rule through the cloud console the previous day, setting one ECS instance’s weight from 50 to 0. But this operation bypassed the Terraform pipeline entirely — the Terraform state file still recorded weight 50.

What sent chills down my spine: I ran terraform plan that afternoon, and the output read No changes. Your infrastructure matches the configuration. — the state file said everything was fine, but the actual cloud configuration had already diverged. The state file was lying to me.

This is configuration drift: the gap between what Terraform declares as the desired state and what’s actually running in the cloud — and Terraform doesn’t know about it. Verizon’s data security report indicates that 68% of cloud security incidents stem from misconfiguration and configuration drift — not zero-day exploits, not vulnerability exploitation, but “what’s running doesn’t match what’s in the code.”

This isn’t an isolated case. Across multiple enterprise clients I’ve worked with, drift is almost an inevitable phase after IaC adoption — as long as someone can bypass Terraform and operate directly in the cloud console, drift will happen. The question isn’t “how to prevent drift” (you can’t fully prevent it), but “how to detect it, how to fix it, and how to govern it.”

This article documents my complete engineering decisions for handling drift over 9 years of operations and IaC practice, covering:

• Three types of drift — not all “changes” warrant alarm
• The exit code mechanism of terraform plan -detailed-exitcode and its pitfalls
• The complementary relationship between driftctl (now Snyk IaC) and terraform plan
• A complete CI/CD pipeline configuration with embedded drift detection
• Three remediation strategies: reconcile, import, rebuild — and when to choose which
• 6 production pitfalls: from “eternal diff” to remediation race conditions
• A drift governance framework: detection frequency, alert thresholds, remediation ownership, prevention strategies

Let me be clear about my stance: drift detection is not as simple as “run a plan and see if there are changes.” It involves tool selection, pipeline orchestration, alerting strategy, remediation workflow, and team collaboration. If you just stuff terraform plan into a cron job and send alerts to a group chat, you’ll be drowned in false positives and eventually mute the alerts — I’ve seen this happen to too many teams.

What Is Drift: When Terraform’s State File Lies to You

Let’s nail this concept down.

Terraform’s working principle in one sentence: you write code declaring “what I want,” and Terraform makes the cloud match what’s in the code. It relies on something called a state file to record “what the cloud looked like after the last apply.” On the next plan, Terraform compares the state file with the code and tells you “what differences need to be synced between the code and the state.”

Here’s the critical part: terraform plan compares code with the state file, not code with the actual cloud state.

The state file records “what Terraform thought the cloud looked like at the last apply,” not “what the cloud actually looks like right now.” If someone modifies a configuration directly in the cloud console after an apply — adding a security group rule, changing an instance type, removing a tag — Terraform’s state file doesn’t update, and terraform plan won’t see these changes.

A diagram to illustrate:

┌──────────────────────────────────────────────────────────────────┐
│                    Terraform Workflow                           │
│                                                                 │
│  Code (.tf)  ←──plan──→  State File  ←──apply──→  Cloud Actual  │
│      │                      │                      │           │
│      │           plan compares these two          apply changes these two
│      │                      │                      │           │
│      └──────────────────────┘                      │           │
│                 ↑                                   │           │
│                 │                  ┌─────────────────┘          │
│                 │                  │  Someone manually changed  │
│                 │                  │  cloud config (console/CLI)│
│                 │                  └───────────┬──────────────┘
│                 │                              │
│           state doesn't know ←── gap is here ←─┘
│                                              ↑
│                                    This is "drift"
└──────────────────────────────────────────────────────────────────┘

terraform plan tells you “the difference between code and state,” but not “the difference between state and the actual cloud.” To discover the latter, Terraform needs to actually read the cloud resources — this is called refresh — and only then can it see drift.

But refresh has a cost: it makes one API call per resource. At scale, a single refresh can take 30 minutes. This is why drift detection isn’t “just run a plan” — it’s a systematic engineering effort involving performance, permissions, and pipeline orchestration.

For state file management and disaster recovery, I covered S3 + DynamoDB production-grade configuration in detail in State File Lost: Terraform State Disaster Recovery. This article focuses on drift detection and remediation — another dimension on top of state management.

Three Types of Drift: Not All “Changes” Warrant Alarm

The community categorizes drift into three types. Understanding the differences determines your detection strategy and alert thresholds.

Type 1: Introduced Drift

This is the most common and most dangerous type. Someone bypassed Terraform and changed configuration directly in the cloud console.

Typical scenario: A developer troubleshooting an issue at 2 AM directly widens the inbound rule of a security group in the console to quickly restore service. Or an ops colleague manually changes the ASG’s desired capacity in the console to scale up. After the change, nobody goes back and syncs it to the Terraform code.

The characteristic of this type: the change was intentional, but the process was wrong. The person wasn’t being malicious — they took a shortcut to solve an urgent problem. The problem is that after the shortcut, code and reality diverge — the next Terraform apply might overwrite these manual changes, or it might not (depending on whether the resource is managed by Terraform).

Type 2: Emergent Drift

This type isn’t caused by humans — it’s the cloud platform’s own changes.

Typical scenario: AWS updates the default configuration of a managed service (e.g., RDS default parameter group), or a cloud platform feature changes, causing Terraform to read attributes that differ from the last apply. Your code didn’t change, nobody manually operated, but terraform plan reports “there are changes.”

The characteristic: nobody is at fault, but the change needs to be perceived and decided upon — either update the code to adapt to the new defaults, or explicitly declare in the code to keep the old values. Emergent drift is most easily misclassified as a “false positive” because it wasn’t caused by human action, and teams tend to ignore it. But the consequence of ignoring it: one day during apply, a critical parameter gets “silently” changed, and service behavior shifts.

Type 3: Pseudo Drift

The most headache-inducing type — it’s not real drift at all, but terraform plan reports a difference.

Typical scenario: A Terraform provider version upgrade changes a default value or calculation method for an attribute, causing plan to show “needs updating” even though the cloud resource hasn’t changed. Or a provider has a bug where a specific attribute reports “needs change” on every plan — the community calls this eternal diff.

The Cloudflare Terraform provider has long had eternal diff issues. The proxied attribute of cloudflare_record reports “needs to change from false to false” on every plan in certain scenarios, and cloudflare_zone_settings_override’s security_header has similar problems. This isn’t real drift, but your detection pipeline treats it as drift, continuously alerting until someone manually ignores it — and “manually ignoring” is exactly the beginning of drift detection failure.

Comparison of the three types:

When I deployed drift detection at a logistics platform, I initially mixed all three types into one alert stream. The first week brought 47 alerts, 39 of which were pseudo drift. The team muted the alert channel in the second week. It took two more weeks to build a pseudo drift whitelist and alert tiering before detection became effective again. I’ll cover this lesson in detail in the pitfalls section below.

Detecting Drift: From terraform plan to Specialized Toolchains

There are two layers of need for drift detection: discovering differences and classifying differences. terraform plan can discover differences but isn’t good at classifying. Specialized tools fill the classification gap.

Method 1: terraform plan -detailed-exitcode

Terraform has a not-so-obvious parameter -detailed-exitcode that changes the plan command’s exit code behavior:

Note: exit code 2 can’t distinguish “differences from code changes” and “differences from drift.” If you check out new code and run plan, exit code 2 might just be because the code has changes. To detect drift, you need to run plan without modifying the code — i.e., first terraform apply to sync code and state, then run plan again to see if there are “unexpected” differences.

A more reliable approach in practice is a two-stage CI pipeline:

#!/bin/bash
set -euo pipefail

# Stage 1: Normal plan (apply code changes)
cd "${TF_WORKSPACE}"
terraform init -input=false
terraform plan -input=false -out=tfplan

# Stage 2: Drift detection (plan again after apply)
terraform apply -input=false -auto-approve tfplan

# refresh + plan, if there are differences here it's drift
terraform plan -detailed-exitcode -input=false -out=drift_plan 2>/dev/null
EXIT_CODE=$?

if [ $EXIT_CODE -eq 2 ]; then
    echo "DRIFT_DETECTED"
    # Output diff details for alerting
    terraform show -json drift_plan > /tmp/drift_report.json
    # Send alert
    curl -s -X POST "${ALERT_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"event\": \"terraform_drift\", \"workspace\": \"${TF_WORKSPACE}\", \"report\": $(cat /tmp/drift_report.json)}"
    exit 0  # Don't block CI, just alert
elif [ $EXIT_CODE -eq 0 ]; then
    echo "NO_DRIFT"
    exit 0
else
    echo "PLAN_ERROR"
    exit 1
fi

The key in this script is the second plan after apply — at this point code changes are already synced, and if plan still reports differences, those differences come from the gap between the actual cloud state and the state file — i.e., drift.

But this approach has a problem: apply actually changes cloud resources. In production, you might not want to trigger an apply just to detect drift. The alternative is to use only terraform refresh (called terraform apply -refresh-only in newer versions):

#!/bin/bash
set -euo pipefail

# Only refresh state, make no actual changes
terraform apply -refresh-only -input=false -auto-approve

# After refresh, run plan to check for drift
terraform plan -detailed-exitcode -input=false -out=drift_plan 2>/dev/null
EXIT_CODE=$?

if [ $EXIT_CODE -eq 2 ]; then
    echo "DRIFT_DETECTED"
    terraform show -json drift_plan > /tmp/drift_report.json
    exit 0
fi

The -refresh-only mode only updates the attribute values recorded in the state file, making no actual changes. This is currently the officially recommended drift detection method in Terraform. But note: refresh makes one provider API call per resource — at scale, performance becomes an issue. I tested on a Workspace managing 800+ resources, and a single refresh took 12 minutes. If run daily, that’s acceptable; for hourly alerting, 12 minutes is too long.

Method 2: driftctl (now Snyk IaC)

driftctl is an open-source tool specifically for IaC drift detection, acquired by Snyk in 2023 and integrated as snyk iac describe. The core difference from terraform plan: driftctl can distinguish “resources managed by code” from “resources that exist in the cloud but aren’t in the code.”

terraform plan can only detect changes to resources already managed by Terraform. But if someone creates a brand-new security group in the console — a resource not in the Terraform state at all — terraform plan can’t see it, because the state doesn’t record it. driftctl catches this: it does a full comparison between the Terraform state and actual cloud resources, finding “resources in the cloud but not in the code.”

Usage (Snyk IaC example):

# Install
npm install -g snyk

# Configure cloud credentials
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."

# Detect drift
snyk iac describe --to "aws+tf" --tfstatefile ./terraform.tfstate

Exit codes:

driftctl/Snyk IaC’s output is better suited for drift alerting than terraform plan because it explicitly labels each drift’s type:

Found 3 drifts:
  - [AWS] aws_security_group.untracked: sg-0abc123 (not in Terraform state)
  - [AWS] aws_security_group_rule.changed: ingress rule on sg-0xyz456
      - from_port: 443 -> 8080
  - [AWS] aws_instance.deleted: i-0def789 (in state but not in cloud)

untracked (exists in cloud but not in state), changed (attributes changed), deleted (in state but not in cloud) — these three categories directly correspond to the drift scenarios I described earlier, and alerts can be tiered by type.

Method 3: HCP Terraform Built-in Drift Detection

If you use HCP Terraform (formerly Terraform Cloud), the Standard tier and above includes built-in drift detection. It periodically (configurable from 5 minutes to 24 hours) performs refresh + plan on each Workspace, and alerts via Slack/Email/Webhook when drift is detected.

I covered the decision to migrate to HCP Terraform in Self-hosted Terraform Backend Broke at 30 People. From a drift detection perspective, HCP Terraform’s advantage is out-of-the-box — no need to build pipelines, manage credential rotation, or write alert scripts. The disadvantage is per-Workspace billing; if your infrastructure is split across 50+ Workspaces, the cost adds up.

Comparison of Three Detection Methods

My recommendation: If your infrastructure is under 200 resources and single-cloud, terraform apply -refresh-only + plan -detailed-exitcode is sufficient — lowest cost, fewest dependencies. If you exceed 200 resources or have multi-cloud needs, driftctl/Snyk IaC’s untracked detection is essential — at this scale, someone will bypass IaC to operate in the console, and plan simply can’t see those newly created resources. If you’re already on HCP Terraform with no more than 20 Workspaces, the built-in feature is the most hassle-free.

Embedding Drift Detection in CI: A Reproducible Pipeline

With the detection method chosen, the next step is embedding it into CI/CD. Here’s a complete pipeline configuration I’ve validated in production.

Three design principles: don’t block deployments (drift detection is informational, shouldn’t gate normal releases), tiered alerting (introduced drift = immediate, pseudo drift = whitelist), traceable (detection results archived for postmortem analysis).

Using GitLab CI as an example (GitHub Actions structure is similar — swap script for steps):

# .gitlab-ci.yml drift detection stage
drift-detection:
  stage: verify
  image: hashicorp/terraform:1.9
  rules:
    # Only run on schedule, not in release pipeline
    - if: $CI_PIPELINE_SOURCE == "schedule"
  variables:
    TF_WORKSPACE: "production"
    ALERT_WEBHOOK: "${ALERT_WEBHOOK_URL}"
  script:
    - terraform init -input=false
    # Only refresh state, no actual changes
    - terraform apply -refresh-only -input=false -auto-approve
    # Detect drift
    - terraform plan -detailed-exitcode -input=false -out=drift_plan 2>/dev/null || true
    - |
      EXIT_CODE=$?
      if [ $EXIT_CODE -eq 2 ]; then
        # Generate JSON format drift report
        terraform show -json drift_plan > drift_report.json
        # Parse drift types
        python3 scripts/drift_analyze.py drift_report.json
        # Send tiered alerts
        python3 scripts/drift_alert.py --report drift_report.json --webhook $ALERT_WEBHOOK
      elif [ $EXIT_CODE -eq 0 ]; then
        echo "No drift detected."
      else
        echo "Plan execution error, exit code: $EXIT_CODE"
        exit 1
      fi      
  artifacts:
    paths:
      - drift_report.json
    expire_in: 30 days

Key points:

Why use schedule trigger instead of every commit: Drift detection’s purpose is to find “code didn’t change but cloud did” — running it on code commits is pointless. Use a scheduled task 2-4 times daily to catch drift without wasting CI resources. I tried hourly — alerts were too frequent, team fatigue was severe. Adjusting to every 6 hours, combined with immediate alerting for introduced drift, worked best.

Why use -refresh-only instead of plan -refresh=false: By default, terraform plan automatically refreshes. If you use terraform plan -refresh=false, it compares the cached values in the state file with the code — effectively turning off drift detection entirely. -refresh-only is the correct approach — refresh state only, no changes.

Core logic of drift_analyze.py: This script parses the output of terraform show -json, classifies each resource change by type (known resource changed = introduced drift, computed attribute changed = possible pseudo drift), then determines alert severity. Simplified version:

#!/usr/bin/env python3
"""Drift report analyzer: parse terraform show -json output, classify drift, generate alerts"""
import json
import sys

# Pseudo drift whitelist (known provider issues)
PSEUDO_DRIFT_WHITELIST = {
    "cloudflare_record.proxied": "Cloudflare provider eternal diff",
    "cloudflare_zone_settings_override.security_header": "Cloudflare provider eternal diff",
    "aws_launch_template.default_version": "AWS provider computed attribute",
}

def analyze_drift(report_path):
    with open(report_path) as f:
        report = json.load(f)
    
    drifts = []
    if not report.get("resource_changes"):
        return {"total": 0, "real_drifts": 0, "pseudo_drifts": 0, "items": []}
    
    for rc in report["resource_changes"]:
        # Only focus on resources with actual changes
        if rc.get("change", {}).get("actions") == ["no-op"]:
            continue
        
        addr = rc["address"]
        change = rc.get("change", {})
        before = change.get("before", {})
        after = change.get("after", {})
        
        # Check if it's pseudo drift (whitelist match)
        is_pseudo = False
        for key, reason in PSEUDO_DRIFT_WHITELIST.items():
            resource_type, attr = key.rsplit(".", 1)
            if addr.startswith(resource_type):
                if attr in before and attr in after:
                    if before[attr] == after[attr]:
                        is_pseudo = True
                        drifts.append({
                            "resource": addr,
                            "type": "pseudo",
                            "reason": reason,
                            "severity": "low",
                        })
                        break
        
        if not is_pseudo:
            drifts.append({
                "resource": addr,
                "type": "introduced" if before else "emergent",
                "before": before,
                "after": after,
                "severity": "high",
            })
    
    real_count = sum(1 for d in drifts if d["type"] != "pseudo")
    pseudo_count = sum(1 for d in drifts if d["type"] == "pseudo")
    
    return {
        "total": len(drifts),
        "real_drifts": real_count,
        "pseudo_drifts": pseudo_count,
        "items": drifts,
    }

if __name__ == "__main__":
    result = analyze_drift(sys.argv[1])
    print(json.dumps(result, indent=2, ensure_ascii=False))
    # Flag alert quality degradation when pseudo drift exceeds 80%
    if result["total"] > 0:
        pseudo_ratio = result["pseudo_drifts"] / result["total"]
        if pseudo_ratio > 0.8:
            print("WARNING: Pseudo drift ratio > 80%, check whitelist coverage")
        sys.exit(0)

Alert strategy in drift_alert.py:

#!/usr/bin/env python3
"""Drift alert sender: tiered by type, introduced drift pushed immediately, pseudo drift in daily summary"""
import json
import sys
import os
import requests
from datetime import datetime

def send_alert(report, webhook_url):
    result = json.loads(open(report).read()) if isinstance(report, str) else report
    
    real_drifts = [d for d in result["items"] if d["type"] != "pseudo"]
    pseudo_drifts = [d for d in result["items"] if d["type"] == "pseudo"]
    
    if real_drifts:
        # Introduced/emergent drift: push immediately
        severity = "P2"
        title = f"[Drift Alert] {len(real_drifts)} real drifts detected"
        color = "#FF4444"
    elif pseudo_drifts:
        # Pure pseudo drift: daily summary, no immediate push
        severity = "P3"
        title = f"[Drift Daily] {len(pseudo_drifts)} pseudo drifts (whitelisted)"
        color = "#FFA500"
    else:
        return  # No drift, no alert
    
    # Build Webhook message (DingTalk/Feishu example)
    msg = {
        "msgtype": "markdown",
        "markdown": {
            "title": title,
            "text": f"### {title}\n\n"
                    f"**Detection time**: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n\n"
                    f"**Workspace**: {os.environ.get('TF_WORKSPACE', 'unknown')}\n\n"
                    f"**Real drifts**: {len(real_drifts)}\n\n"
                    f"**Pseudo drifts**: {len(pseudo_drifts)}\n\n"
                    + "\n".join(
                        f"- `{d['resource']}` ({d['type']})"
                        for d in result["items"][:10]
                    )
        }
    }
    
    requests.post(webhook_url, json=msg, timeout=10)
    print(f"Alert sent: {title}")

if __name__ == "__main__":
    report_file = sys.argv[sys.argv.index("--report") + 1]
    webhook = sys.argv[sys.argv.index("--webhook") + 1]
    send_alert(report_file, webhook)

This pipeline ran for 4 months at a mobility project with these results: the first week found 23 introduced drifts (mostly developers temporarily modifying security groups in the console), dropping to 5 in the second week and 1-2 in the third. Team trust in drift detection went from “here’s another noise alert” to “this alert is worth reading.” The turning point was the pseudo drift whitelist — after covering 8 known eternal diffs, the real drift alert signal-to-noise ratio improved from 1:10 to 8:1.

Three Drift Remediation Strategies

Once drift is detected, what do you do? There are three remediation strategies, and choosing the wrong one can be costlier than the drift itself.

Strategy 1: Reconcile (align back to code) — terraform apply

The most direct remediation: run terraform apply to let Terraform change the actual cloud state “back” to what the code declares.

terraform apply -auto-approve

When to use: the drift is unreasonable and should be rolled back. For example, someone changed a security group inbound rule from port 443 to 0-65535 — this change should be reverted.

When NOT to use: the manual change is legitimate and should be preserved. For example, someone changed the ASG desired capacity from 4 to 8 because traffic genuinely increased. If you run apply to change it back to 4, the service might not handle the load. I’ve seen this happen — drift detection found the ASG desired was inconsistent, auto-ran apply to revert it, and it hit a traffic peak: P99 latency went from 200ms to 1200ms.

Rule of thumb: Reconcile is suitable for configuration-type resources (security group rules, IAM policies, tags) but NOT for capacity-type resources (instance count, disk size, bandwidth). Configuration drift is typically “should be rolled back,” while capacity drift is typically “understand the reason before deciding.”

Strategy 2: Import (bring into management) — terraform import

Someone created a new resource in the console that’s outside Terraform management. You can’t simply apply (apply won’t touch resources not in state). You need to first terraform import to bring the resource into state, then update the code to align it with the actual configuration.

# Someone manually created a security group in the console
# First import it into state
terraform import aws_security_group.manual_created sg-0abc123456

# Then run plan to see differences
terraform plan
# Plan will show differences between code and the imported resource
# Update code based on plan output to match actual configuration
# Or decide to change the actual configuration back to what code declares

When to use: the manually created resource is legitimate and should be kept, but the IaC process was skipped. This is especially common when a team is new to Terraform — developers aren’t used to managing infrastructure through code and create resources directly in the console.

When NOT to use: the manually created resource is redundant and should be deleted. For example, someone created a temporary security group during testing and forgot to remove it. In this case, delete it directly in the cloud console — don’t import it into Terraform and then delete via code, which is a roundabout path that adds state file complexity.

Strategy 3: Rebuild (recreate from code)

The most extreme remediation: remove the drifted resource from Terraform (terraform destroy or terraform state rm), then recreate it from code.

When to use: the resource state is severely corrupted, and refresh and import can’t fix it. For example, the state file and actual cloud state differ too much, or the resource has been manually modified beyond recognition, making patching more expensive than starting over.

When NOT to use: production resources currently serving traffic. Rebuild means the resource is destroyed and then recreated — service will be interrupted. Unless you have a clear disaster recovery plan and change window, don’t use this strategy on production resources.

Decision tree for the three strategies:

Drift detected
    │
    ├── Is it a managed resource with changed attributes?
    │       ├── Is the change legitimate?
    │       │       ├── Yes → Import/Update code (Strategy 2)
    │       │       └── No → Reconcile/apply rollback (Strategy 1)
    │       └── Not sure?
    │               └── Manual review before deciding
    │
    └── Is it a new resource in cloud but not in state?
            ├── Should it be kept?
            │       ├── Yes → Import + update code (Strategy 2)
            │       └── No → Delete directly in cloud
            └── In state but gone from cloud?
                    ├── Was it accidentally deleted?
                    │       ├── Yes → Rebuild from code (Strategy 3)
                    │       └── No → Remove from state (state rm)

This decision tree isn’t theoretical — it was distilled from 40+ drift incidents. Walking through it each time at least prevents mistakes like “rolling back a legitimate manual change” or “importing a redundant resource into IaC, adding complexity.”

Production Pitfalls: 6 Traps

This section has the highest information density in the entire article. Each pitfall comes from a real production environment, each with a fix.

Pitfall 1: Cloudflare Provider’s Eternal Diff

Symptom: The drift detection pipeline alerts every 6 hours about cloudflare_record resource changes, but after terraform apply, the next detection still reports the same differences. Plan output shows the proxied attribute needs to change from false to false — the value didn’t change but Terraform thinks it needs updating.

Root cause: The Cloudflare provider’s proxied attribute has a type mapping inconsistency between the API return value and the Terraform schema. The provider reads a boolean from the API, but the type changes during serialization to the state file, and the next plan triggers a “change” due to type mismatch.

Fix: Two paths. First, upgrade the provider version — the community fixed some eternal diff issues in version 4.x and later. Second, if residuals remain after upgrade, add the corresponding resource+attribute combination to the drift analysis script’s whitelist, letting the analyzer know this is pseudo drift. The PSEUDO_DRIFT_WHITELIST in drift_analyze.py above serves this purpose.

Lesson: Not all providers are as mature as AWS. Niche providers (Cloudflare, Fastly, Akamai) have eternal diffs as the norm. Drift detection pipelines must account for whitelist mechanisms from the design phase, or they’ll be overwhelmed by pseudo drift.

Pitfall 2: Refresh Causing API Throttling

Symptom: Running terraform apply -refresh-only on a Workspace managing 800+ resources fails halfway through with RateExceededError — AWS API is throttled. Terraform doesn’t do API rate control by default.

Root cause: Refresh makes one Describe API call per resource. 800 resources sent in one burst exceed AWS’s API throttling limit (N requests per second).

Fix: Add rate limiting in the provider configuration:

provider "aws" {
  region = "us-east-1"
  
  # Limit API call rate
  max_retries = 5
  
  # In Terraform 1.9+, can use rate_limiting config block
  dynamic "endpoints" {}
}

A more practical solution is to adjust the drift detection execution strategy: don’t refresh all resources at once — batch by Workspace. Split 800 resources into 4 Workspaces (200 each), staggering each Workspace’s refresh by 15 minutes. This drops per-Workspace API calls to 200, avoiding throttling.

Pitfall 3: Remediation Race Condition

Symptom: Drift detection finds a manually modified security group rule, CI auto-runs terraform apply to fix it. But immediately after apply, running detection again reports drift — because someone modified the configuration between apply and detection.

Root cause: No lock mechanism between drift detection and drift remediation. Multiple operators (CI’s auto-remediation + human manual operation) modify the same resource in the same time window, and the later execution overwrites the earlier one.

Fix: Don’t trigger remediation automatically. After detecting drift, only alert — let the on-call SRE confirm and manually execute the fix. Auto-remediation sounds appealing, but in a multi-person production environment, auto-remediation + human manual operation = endless tug-of-war.

I tried auto-remediation in one project — detect drift and auto-apply. It caused an incident in the first week: a developer manually changed the ASG for emergency scaling, CI detected the drift and auto-reverted it, the developer changed it back, CI reverted it again… 3 rounds of tug-of-war until ASG quota was exhausted and new Pods couldn’t start. After that, I turned off auto-remediation and switched to “detect → alert → manual confirmation → manual fix” flow. Slower, but safe.

Pitfall 4: State Lock Causing False Positives

Symptom: The drift detection pipeline reports Error acquiring the state lock, then CI exits with code 1 (execution error), which gets treated as “detection failed” and ignored. In reality, an incomplete Terraform operation held the state lock, preventing detection from running.

Root cause: A stale lock remained in the DynamoDB lock table — a CI timeout left the lock unreleased, and subsequent detections couldn’t acquire it. I covered DynamoDB lock pitfalls in detail in State File Lost.

Fix: Add lock status checking and stale lock handling in the drift detection script:

# Check lock status
LOCK_ID=$(aws dynamodb get-item \
    --table-name terraform-locks \
    --key '{"LockID": {"S": "'${STATE_KEY}'"}}' \
    --projection-expression "LockID,Info" \
    --query 'Item.Info.S' --output text 2>/dev/null || echo "")

if [ -n "$LOCK_ID" ]; then
    # Lock exists, check lock age
    LOCK_AGE=$(echo "$LOCK_ID" | python3 -c "
import sys, json
lock = json.load(sys.stdin)
created = lock.get('created', '')
print(created)
" 2>/dev/null || echo "unknown")
    
    echo "State is locked (age: $LOCK_AGE). Skipping drift detection."
    # Send alert for SRE to handle stale lock
    curl -s -X POST "${ALERT_WEBHOOK}" \
        -d "{\"event\": \"state_locked\", \"lock_age\": \"$LOCK_AGE\"}"
    exit 0  # Don't block CI
fi

Pitfall 5: Large State File Plan Performance

Symptom: A 5MB state file (~600 resources), terraform plan -detailed-exitcode takes 18 minutes. The drift detection CI times out (default 10 minutes), and detection never completes.

Root cause: Plan execution time scales linearly with resource count. Terraform does diff calculations for each resource during plan — 600 resources × 10-30 attributes each = massive JSON diff operations. Plus the API calls during refresh, total time = API latency × resource count + diff computation × resource count.

Fix: Three approaches.

First, split Workspaces. Break 600 resources into 5-6 Workspaces by business domain (100-120 each), reducing per-Workspace plan time to 2-3 minutes. This is the most fundamental solution — I covered layered design principles in Terraform Module Layered Design, and the same thinking applies here.

Second, skip refresh and use driftctl instead. driftctl doesn’t need to run a full terraform plan — it uses SDK batch resource listing, which is 2-3x faster than Terraform’s per-resource refresh.

Third, increase CI timeout. Simple but effective — bump the drift detection CI timeout from 10 to 30 minutes. If detection runs every 6 hours, 30 minutes of execution time is perfectly acceptable.

Pitfall 6: Alert Fatigue — The Silent Killer of Drift Detection

Symptom: In the first week after drift detection went live, the alert channel received 47 drift alerts. In the second week, the team muted the channel. In the third week, someone asked “is drift detection still running?” — it was, but nobody was looking.

Root cause: All three drift types mixed into one alert stream. 39 were pseudo drift (Cloudflare eternal diff + provider computed attributes), 7 were emergent drift (AWS updated RDS default parameter group), and only 1 was introduced drift (someone manually modified a security group). Signal-to-noise ratio of 1:46 — alerts completely failed.

Fix: This is actually the most time-consuming part of the entire drift detection engineering effort — not building the pipeline is hard, making alerts effective is hard. Three steps:

1. Build a pseudo drift whitelist: Run detection for a week, collect all “still reports differences after apply” resource+attribute combinations, add them to the whitelist. The PSEUDO_DRIFT_WHITELIST in drift_analyze.py above serves this purpose.
2. Tiered alerting: Introduced drift = immediate push (DingTalk/Feishu/Slack), emergent drift = daily summary (one email per day), pseudo drift = silent logging (store logs only, no alerts).
3. Regularly clean the whitelist: Check the whitelist quarterly to see if providers have fixed eternal diffs that can be removed. Whitelists aren’t set-and-forget — providers upgrade, bugs get fixed, whitelists must evolve.

Drift Governance Framework: From Detection to Prevention

Detection and remediation are “firefighting.” Governance is “fire prevention.” A mature IaC drift governance system should have four layers.

Layer 1: Detection Frequency Strategy

Different environments use different frequencies:

This frequency wasn’t pulled from thin air — tested at a mobility project: dropping production from hourly to every 6 hours reduced alert fatigue by 70%, but median drift detection time (from introduction to detection) went from 35 minutes to 4 hours. A 4-hour detection delay is acceptable for configuration-type drift (config changes typically don’t cause immediate failures), but for security risks like widened security groups, 4 hours is indeed too long. The final solution: full detection every 6 hours + hourly detection for security group/IAM resources only.

Layer 2: Drift Alert Thresholds

Not all drift is worth alerting. Two thresholds:

• P2 immediate alert: Security group rule changes, IAM policy changes, network ACL changes — drift involving security/compliance must be pushed immediately
• P3 daily summary: Tag changes, instance type changes, ASG capacity changes — drift not involving security but needing awareness, summarized in a daily email

The drift_analyze.py above already implements this tiering through the severity field. The key is that the alert template should let the on-call SRE see at a glance “this is a security issue or a configuration issue,” not a pile of JSON.

Layer 3: Remediation Ownership and Workflow

Once drift is detected, who fixes it? Ownership must be clear:

• Introduced drift: The person who introduced it fixes it. Use CloudTrail / audit logs to identify the operator, notify them to fix or initiate a change rollback process
• Emergent drift: IaC team owns it. Emergent drift requires code updates or confirming to keep old values — this is IaC maintenance work
• Pseudo drift: IaC team owns it. Requires provider upgrade or whitelist update

In 5000 Alerts Nobody Reads, I covered the importance of alert ownership — drift alerts are the same: alerts without clear ownership equal no alerts.

Layer 4: Prevention Strategy

The best drift is the drift that never happened. Prevention measures, ranked by effectiveness:

1. Converge console permissions (most effective): Most developers shouldn’t have direct permission to modify resources in the console. This sounds like permissions management rather than an IaC issue, but in practice it’s the first line of defense against drift. Across the clients I maintain, the one with the most restricted console permissions has 1/5 the introduced drift frequency of others.

2. Sentinel policy-as-code: If using HCP Terraform, Sentinel can block non-compliant changes at the plan stage. For example, a Sentinel policy can forbid setting a security group inbound rule to 0.0.0.0/0. Sentinel doesn’t directly prevent drift, but it limits what Terraform apply can do, indirectly reducing the need for “apply then manually change.”

3. Change window mechanism: Only allow IaC operations during change windows; out-of-window changes must go through an emergency process. I covered this approach in Using Error Budget Gates to Replace Manual CAB — using policy gates instead of manual approval, but the core logic is the same: make changes predictable and auditable.

4. Read-only infrastructure console: The most extreme approach — developers get read-only access to the console, all changes must go through Terraform. This is mandatory in some compliance-heavy scenarios (finance, healthcare), but most internet companies can’t achieve it — developers need quick console access for troubleshooting. A pragmatic approach is to differentiate by resource type: core infrastructure (network, security groups, IAM) is read-only, application-layer resources (ECS, RDS parameters) are writable.

Tool Selection Decision Table

Condensing the above into a decision table for easy reference:

A more detailed cost comparison:

My clear recommendation: don’t jump straight to HCP Terraform. Start with a 2-person-day CI + plan solution, run it for two weeks, collect the pseudo drift baseline, validate the alerting strategy, and get the team used to the concept of “drift detection.” If the solution works but CI performance can’t keep up (resource count exceeds 200), then add driftctl. If driftctl isn’t enough or the team exceeds 30 people and needs approval flows, then consider HCP Terraform. Iterate gradually — don’t buy the most expensive solution on day one.

Summary

Back to the opening scenario — 3 untracked resources at 2 AM. The actual handling process: the drift detection pipeline had alerted 6 hours earlier, but because those 3 resources weren’t in Terraform’s management scope, terraform plan didn’t find them — driftctl’s untracked detection did. The on-call colleague received a P2 alert and within 15 minutes confirmed the resource source: a temporary security group created by a developer through CLI during an emergency fix the previous day, forgotten to be removed. After assessment, the decision was to “delete directly in the cloud” (because the resource was redundant), then the team’s operating procedures were updated to include “sync IaC after emergency changes” in the runbook.

The entire process from detection to remediation: 47 minutes. No incident escalation, no alert noise.

This case confirmed a few things for me:

The value of drift detection isn’t in the tool itself, but in the process. Whether the tool is terraform plan, driftctl, or HCP built-in, they’re all just detection means. What truly determines effectiveness is alert tiering (introduced/emergent/pseudo handled separately), remediation ownership (who fixes it), and prevention strategy (console permission convergence). A team with a CI script but good processes is safer than a team with HCP Enterprise but all alerts muted.

Pseudo drift governance is the make-or-break factor. If you do one thing, build the pseudo drift whitelist first. The biggest fear in drift detection isn’t “missing drift” — it’s “50 alerts, 48 of which are false, the team mutes alerts, and the 2 real ones get missed too.”

Auto-remediation is both temptation and trap. Auto-applying on drift detection sounds appealing — but as long as humans operate cloud resources, auto-remediation will conflict with manual operations. The pragmatic approach is alert → manual confirmation → manual fix. One step slower, but no errors.

Finally, drift detection is the litmus test for IaC maturity. A team’s attitude toward drift detection — whether they take it seriously or treat it as noise — directly reflects their understanding depth of IaC. If you’re still in the “just started Terraform, everything is beautiful” phase, you might not have encountered drift yet. Once your infrastructure exceeds 200+ resources and the team surpasses 10 people, drift will come. Set up the detection pipeline in advance, so you won’t be caught off guard.

References & Acknowledgments

The following resources were referenced during the writing of this article. Thanks to the original authors for their contributions:

1. Verizon Data Breach Investigations Report (DBIR) — Verizon, 2024-2025 annual data security report, referenced for the statistic that 68% of cloud security incidents stem from misconfiguration and drift
2. Terraform documentation: plan -detailed-exitcode — HashiCorp official documentation, referenced for the plan command exit code mechanism
3. Snyk IaC describe (formerly driftctl) — Snyk official documentation, referenced for driftctl/Snyk IaC drift detection capabilities and exit codes
4. HCP Terraform: Drift Detection — HashiCorp official documentation, referenced for HCP Terraform built-in drift detection configuration and triggers
5. Terraform refresh-only mode — HashiCorp official documentation, referenced for -refresh-only mode usage and considerations
6. Cloudflare Terraform Provider GitHub Issues — Cloudflare provider open-source repository, referenced for eternal diff issue discussions and fix progress
7. Google SRE Book: Change Management — Google SRE team, referenced for change management and configuration consistency methodology