Overview

2 AM, the alert channel exploded. A web server that went live two hours ago was at 100% CPU. SSH in — Nginx wasn’t installed, but an old Apache was squatting on port 80. Checked the deploy logs: Terraform had run terraform apply ten minutes ago, rebuilding the machine (someone changed instance_type), but Ansible’s configuration playbook never followed — fresh machine, all software configs gone.

This isn’t an isolated incident. Using Terraform to provision cloud instances and Ansible for configuration management has become the standard combo for ops teams in 2026. But “standard” doesn’t mean “done right” — too many teams stack these tools together without a clear collaboration design, resulting in infrastructure being “managed twice”: Terraform builds resources once, Ansible configures them again, neither side knows what the other is doing. Configuration drift, state inconsistency, broken pipelines — all of it follows.

This article breaks down 5 core decisions in Terraform + Ansible collaboration. Each comes from real production environment lessons, not theoretical speculation. If your team is using or planning to use this combo, these decision points will save you at least half a year of trial and error.

1. Responsibility Boundary: What Terraform Owns, What Ansible Owns

Let’s address the most fundamental yet most overlooked question first: where exactly do you draw the line between these two tools?

Terraform’s strength is infrastructure orchestration — creating cloud instances, VMs, networks, storage, load balancers, security groups, database instances, K8s clusters. It’s declarative: you tell it “I want 3 4C8G instances in this availability zone,” and it creates them. It manages the full resource lifecycle: create, modify, destroy.

Ansible’s strength is system configuration and application delivery — OS initialization, account permissions, kernel parameters, firewall rules, package installation, middleware deployment, config file rendering, service management, cron jobs. It’s procedural (though designed for idempotency): you tell it “install Nginx first, then copy config, then start the service,” and it executes in order.

Each tool has its own domain, and the line seems clear. But in practice, the boundary is often blurry. Here are common “gray areas”:

Configuration ItemTerraform Can DoAnsible Can DoWho Should Own It
Create cloud instance✅ Native❌ Not suitableTerraform
Install Nginx✅ remote-exec✅ NativeAnsible
Security group rules✅ Native⚠️ Possible but not recommendedTerraform
Render Nginx config files⚠️ local-exec hack✅ Jinja2 templatesAnsible
Disk mount and format⚠️ Possible but crude✅ Fine-grained controlAnsible
Create DNS records✅ Native⚠️ Possible but not recommendedTerraform
Users and SSH keys✅ Possible✅ NativeAnsible

Decision 1: Don’t Use Terraform Provisioners for Complex Configuration

Terraform provides two provisioners — remote-exec and local-exec — that can execute commands or scripts after resource creation. Many teams start by writing long remote-exec blocks inside Terraform, using shell scripts to install software, configure firewalls, tune kernel parameters.

This approach works in testing but is a ticking time bomb in production:

# ❌ Bad example: doing complex configuration in Terraform
resource "alicloud_instance" "web" {
  instance_type   = "ecs.g6.large"
  # ... other config
  
  provisioner "remote-exec" {
    inline = [
      "yum install -y nginx",
      "systemctl enable nginx",
      "systemctl start nginx",
      "echo 'net.ipv4.tcp_tw_reuse = 1' >> /etc/sysctl.conf",
      "sysctl -p",
      # ... 50 more lines
    ]
  }
}

What’s wrong with this?

First, provisioners only execute on resource creation. If you later change instance_type causing a rebuild, the provisioner runs again. But if you just modify an unrelated parameter (like adding a tag), Terraform doesn’t trigger a rebuild — the provisioner doesn’t run — and your configuration is now out of sync.

Second, provisioner execution is outside Terraform’s state management. Terraform’s core mechanism is “declarative + state tracking”: you say you want 3 instances, it ensures 3 exist. But the commands run by provisioners aren’t in state. Terraform doesn’t know if these commands ran, succeeded, or if the configuration is still effective. Drift goes undetected.

Third, provisioners turn Terraform into a procedural tool. Terraform’s value lies in being declarative — you describe the “end state” and the tool figures out how to get there. Provisioners force imperative logic into this model, breaking it.

The correct approach: Terraform handles “resource creation complete,” then exposes necessary information (IPs, hostnames, role tags) through outputs, handing off to Ansible for configuration.

# ✅ Correct: Terraform only creates resources, outputs to Ansible
resource "alicloud_instance" "web" {
  count          = 3
  instance_type  = "ecs.g6.large"
  # ... other config
}

output "web_instance_ips" {
  value = alicloud_instance.web[*].public_ip
}

output "web_instance_roles" {
  value = alicloud_instance.web[*].tags.Role
}

Field Experience: During a K8s migration at a ride-hailing project, we initially used Terraform provisioners for node initialization. Every terraform plan showed dozens of provisioner changes, making plan reviews unreadable. After moving all initialization logic to Ansible, Terraform only managed ECS and security groups, and terraform plan output shrank from 200+ lines to 30. Review efficiency doubled. (Related: Getting Started with Terraform IaC)

Practical Rules for Drawing the Boundary

I implemented a rule in my team that worked well:

“Cloud resource lifecycle belongs to Terraform. OS through application layer belongs to Ansible. Data flows between the two layers only via outputs and dynamic inventory.”

Specifically:

  • Terraform owns: cloud instances, VPCs, subnets, security groups, SLBs, RDS instances, DNS records, object storage buckets
  • Ansible owns: OS initialization, kernel parameters, users and permissions, software installation, config files, service management, application deployment
  • Handoff point: Terraform output exposes IPs and metadata → Ansible dynamic inventory consumes this data

With this rule in place, the boundary becomes clear. When team members ask “who should manage this configuration,” they first check which layer it belongs to — cloud resource or OS — and the answer comes naturally.

2. Dynamic Inventory: Making Terraform Outputs Auto-Become Ansible Inventory

With the boundary drawn, the next question is: Terraform provisions 50 cloud instances, but how does Ansible know their IPs and groupings?

The most basic approach is manually maintaining Ansible’s hosts file. After Terraform creates machines, someone copies IPs into hosts, then runs Ansible. This works with few machines, but once Terraform does elastic scaling, static inventory is useless — new machines are added but hosts isn’t updated, Ansible misses them.

Decision 2: Use Terraform State as the Data Source for Ansible Dynamic Inventory

Ansible natively supports Dynamic Inventory. The concept: before executing a playbook, Ansible calls an external script or plugin that pulls current host lists and grouping info from a data source.

There are three data source options:

Data SourceImplementationProsCons
Terraform StateUse terraform-inventory or custom scriptReal-time accurate, fully synced with TFRequires state file access
Cloud APIUse Ansible cloud plugins (e.g., alicloud/aws_ec2)Independent of TFGrouping depends on tags, may mismatch TF
TF Output + Local FileTF output to JSON file, Ansible reads itSimple, few dependenciesNot real-time, needs manual trigger

I recommend using Terraform State as the data source, because: Terraform State is the “source of truth” for infrastructure. It precisely records what resources exist and their attributes. Using State for inventory means Ansible sees exactly what Terraform manages — no “exists in cloud but missing from inventory” or “in inventory but deleted from cloud.”

There are two implementation approaches.

Approach 1: terraform-inventory tool

This is an open-source tool that reads Terraform State files directly and generates Ansible dynamic inventory:

# Install terraform-inventory
# https://github.com/adammck/terraform-inventory
wget https://github.com/adammck/terraform-inventory/releases/download/v0.2.1/terraform-inventory_0.2.1_linux_amd64.tar.gz
tar xzf terraform-inventory_0.2.1_linux_amd64.tar.gz
mv terraform-inventory /usr/local/bin/

# Usage: specify the directory containing Terraform state
ansible-playbook -i /path/to/terraform/envs/prod site.yml

terraform-inventory reads the state file and generates Ansible groups based on Terraform resource type and name. For example, alicloud_instance.web generates a host group called web.

Approach 2: Custom Dynamic Inventory Script

If terraform-inventory’s grouping logic doesn’t meet your needs, you can write your own. The script only needs to implement two parameters: --list returns all hosts and groups, --host <hostname> returns a single host’s variables.

#!/usr/bin/env python3
"""Terraform State → Ansible Dynamic Inventory Script"""
import json
import subprocess
import sys

def get_state():
    """Read Terraform State"""
    result = subprocess.run(
        ["terraform", "show", "-json"],
        capture_output=True,
        text=True,
        cwd="/path/to/terraform/envs/prod"
    )
    return json.loads(result.stdout)

def build_inventory(state):
    """Build Ansible inventory from state"""
    inventory = {"_meta": {"hostvars": {}}, "all": {"children": []}}
    
    for resource in state.get("values", {}).get("root_module", {}).get("resources", []):
        if resource["type"] == "alicloud_instance":
            # Group by tags.Role
            role = resource.get("values", {}).get("tags", {}).get("Role", "ungrouped")
            if role not in inventory:
                inventory[role] = {"hosts": []}
                inventory["all"]["children"].append(role)
            
            ip = resource["values"]["public_ip"]
            hostname = resource["values"]["instance_name"]
            
            inventory[role]["hosts"].append(hostname)
            inventory["_meta"]["hostvars"][hostname] = {
                "ansible_host": ip,
                "instance_id": resource["values"]["id"],
                "instance_type": resource["values"]["instance_type"]
            }
    
    return inventory

if __name__ == "__main__":
    if len(sys.argv) == 2 and sys.argv[1] == "--list":
        state = get_state()
        inventory = build_inventory(state)
        print(json.dumps(inventory, indent=2))
    elif len(sys.argv) == 3 and sys.argv[1] == "--host":
        # Single host variable query
        print(json.dumps({}))

The key design: use Terraform resource tags for grouping. In Terraform, tag each machine with Role=web or Role=db. The script reads state, groups by Role, and Ansible can directly use hosts: web or hosts: db to target playbooks.

Pitfall Warning: The dynamic inventory script reads Terraform State files. If your State is stored in a remote backend (e.g., OSS/S3), the script needs access. In production, I recommend storing State in a remote backend with proper permissions — enabling both team collaboration and preventing local State file loss. See Related: Lost Your State File? for more State management practices.

Benchmark: Dynamic Inventory vs Static Inventory

Tested in a 50-instance environment:

MetricStatic Inventory (Manual)Dynamic Inventory (TF State)
Inventory update timeManual 5-15 min0 (automatic)
Machine miss rate~8% (human error)0%
Post-scaling activation timeRequires manual interventionEffective on next playbook run
Grouping consistencyDepends on humans, error-proneFully consistent with Terraform

Another benefit of dynamic inventory: automatic adaptation during elastic scaling. When Terraform scales count = 5 → count = 8, Ansible’s next run automatically includes the 3 new machines — no manual intervention needed.

3. Orchestration Order and Triggers: Who Goes First, What If Something Fails Mid-Way

With boundary and inventory solved, the next question is: how do you orchestrate Terraform and Ansible in time? Who executes first, how do they pass data, how do you handle failures?

Decision 3: Use a Two-Stage Pipeline to Isolate Terraform and Ansible, with Verification Gates in Between

The most intuitive approach is triggering Ansible directly from Terraform — using a local-exec provisioner to call ansible-playbook after terraform apply. This seems simple, but has three fatal flaws in production:

  1. Uncontrolled error propagation: Terraform apply succeeded, but Ansible playbook failed. Resources are already created but unconfigured — half-baked state. Terraform’s provisioner failure marks the entire apply as failed, but resources are already created — you can’t roll back.

  2. Excessive execution time: Terraform apply takes minutes (waiting for cloud API), plus Ansible configuring 50 machines could take 10-20 minutes. One command waits 25 minutes — any interruption causes state inconsistency.

  3. Debugging difficulty: Terraform and Ansible output mix together. When something breaks, you can’t tell which tool caused it.

The correct approach is splitting Terraform and Ansible into two pipeline stages, connected by a verification gate:

Stage 1: Terraform Plan → Review → Terraform Apply
         Verification Gate: Check resources actually exist (API query)
Stage 2: Ansible Dynamic Inventory Refresh → Ansible Playbook
         Verification Gate: Check key services are running (health check)

The core of this design is isolation + verification. Terraform only builds resources, then enters a verification gate — using cloud API to confirm instance status is Running and SSH port is reachable. Only after verification passes does Ansible begin. After Ansible configures, another verification gate checks whether key ports are listening and services are healthy.

Here’s a GitLab CI implementation of this two-stage pipeline:

# .gitlab-ci.yml - Terraform + Ansible two-stage pipeline
stages:
  - terraform
  - verify_infra
  - configure
  - verify_app

variables:
  TF_DIR: "terraform/envs/prod"
  ANSIBLE_DIR: "ansible"

# Stage 1: Terraform
terraform_plan:
  stage: terraform
  script:
    - cd $TF_DIR
    - terraform init
    - terraform plan -out=tfplan
  artifacts:
    paths:
      - $TF_DIR/tfplan
  only:
    - main

terraform_apply:
  stage: terraform
  script:
    - cd $TF_DIR
    - terraform init
    - terraform apply -auto-approve tfplan
    # Output host info to file for later stages
    - terraform output -json > ../../$ANSIBLE_DIR/tf_outputs.json
  dependencies:
    - terraform_plan
  only:
    - main
  when: manual  # Requires manual confirmation before apply

# Verification gate: check infrastructure readiness
verify_infra:
  stage: verify_infra
  script:
    - python3 scripts/verify_infra.py --tf-output $ANSIBLE_DIR/tf_outputs.json
  only:
    - main

# Stage 2: Ansible
ansible_configure:
  stage: configure
  script:
    - cd $ANSIBLE_DIR
    # Refresh dynamic inventory
    - ansible-inventory -i inventory_terraform.py --list > /dev/null
    # Execute playbook
    - ansible-playbook -i inventory_terraform.py site.yml --limit "tag_Role_web"
  only:
    - main

# Verification gate: check application health
verify_app:
  stage: verify_app
  script:
    - python3 scripts/verify_app.py --tf-output $ANSIBLE_DIR/tf_outputs.json
  only:
    - main

Key points in this pipeline design:

Manual confirmation gate: terraform_apply has when: manual — plan must be manually confirmed before apply executes. This is because Terraform apply is irreversible — once resources are created, they incur costs. When building our Go CI/CD scheduling engine, we cut deployment time from 1.5h to 5min, but critical operations (like infrastructure changes) always retained manual confirmation. Speed and safety aren’t opposites — what’s fast is execution speed, what’s safe is the confirmation process.

Verification scripts: verify_infra.py queries the cloud API to confirm each instance is Running and SSH port is reachable. verify_app.py checks HTTP health endpoints return 200.

# scripts/verify_infra.py
"""Verify infrastructure readiness"""
import json
import socket
import sys
import time

def check_ssh_reachable(ip, port=22, timeout=5):
    """Check if SSH port is reachable"""
    try:
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        sock.settimeout(timeout)
        result = sock.connect_ex((ip, port))
        sock.close()
        return result == 0
    except Exception:
        return False

def main():
    with open(sys.argv[2]) as f:
        outputs = json.load(f)
    
    ips = outputs.get("web_instance_ips", {}).get("value", [])
    all_ready = True
    
    for ip in ips:
        # Wait up to 120 seconds
        for attempt in range(24):
            if check_ssh_reachable(ip):
                print(f"[OK] {ip}:22 SSH reachable")
                break
            time.sleep(5)
        else:
            print(f"[FAIL] {ip}:22 SSH unreachable after 120s timeout")
            all_ready = False
    
    if not all_ready:
        print("Infrastructure verification failed, blocking Ansible stage")
        sys.exit(1)
    
    print("All instances SSH reachable, infrastructure ready")

if __name__ == "__main__":
    main()

Pitfall Warning: There’s a delay between cloud instance creation and SSH reachability. Alibaba Cloud ECS typically takes 30-60 seconds to complete initialization, AWS EC2 60-90 seconds. If you run Ansible directly without a verification gate, the first batch of SSH connections will fail and the playbook will error out. The verification gate’s value is waiting until machines are truly ready before configuring.

4. Configuration Drift Governance: What Happens When Both Tools Modify the Same Machine

This is the trickiest problem in Terraform + Ansible collaboration.

Consider this scenario: Terraform created a cloud instance and configured a security group rule — opening ports 80 and 443. Ansible did system initialization on this machine, including an nftables firewall rule — also managing port access.

Now someone files an ops ticket: “temporarily open port 8080 for testing.” An ops engineer SSHes in, nftables add rule to open 8080. Two days later, Terraform runs plan — security group hasn’t changed (because 8080 is an in-host nftables rule, outside Terraform’s scope), so it does nothing. But when Ansible runs the initialization playbook next, since 8080 isn’t in the Ansible template, it gets deleted. The tester finds the port closed again, comes back to ops.

This is configuration drift. The root cause is unclear resource ownership.

Decision 4: Build a Resource Ownership Matrix, Clarify Who Manages Each Configuration Type

The key to solving configuration drift isn’t tools — it’s ownership rules. My approach is maintaining a “resource ownership matrix”:

Configuration LayerOwnerPermitted Change PathDrift Detection Method
Cloud instances/network/storageTerraformModify TF code → PR → applyterraform plan
Security group rulesTerraformModify TF code → PR → applyterraform plan
OS kernel parametersAnsibleModify Ansible role → PR → playbookansible --check
Users and SSH keysAnsibleModify Ansible role → PR → playbookansible --check
Firewall rules (in-host)AnsibleModify Ansible role → PR → playbookansible --check
Application config filesAnsibleModify Ansible template → PR → playbookansible --check

The core principle of this table: each configuration item has exactly one owner. No “both Terraform and Ansible can manage this” situations. If both tools claim to manage the same configuration, drift is only a matter of time.

Specific drift detection methods:

Terraform side: Run terraform plan periodically (e.g., daily at midnight). If plan shows changes, the actual resources have deviated from the declared state — someone made manual changes, or Ansible modified something it shouldn’t have.

# Drift detection script (cron or CI scheduled task)
#!/bin/bash
cd /path/to/terraform/envs/prod
terraform init -input=false
terraform plan -detailed-exitcode -out=/dev/null
EXIT_CODE=$?
# 0: no changes, 1: error, 2: drift detected
if [ $EXIT_CODE -eq 2 ]; then
    echo "⚠️ Terraform drift detected!"
    terraform plan -no-color > /tmp/drift_report.txt
    # Send to alert channel
    curl -X POST "https://oapi.dingtalk.com/robot/send?access_token=$DINGTALK_TOKEN" \
      -H "Content-Type: application/json" \
      -d "{\"msgtype\":\"text\",\"text\":{\"content\":\"Terraform drift alert: $ENV environment\n$(cat /tmp/drift_report.txt | head -50)\"}}"
fi

Ansible side: Use ansible-playbook --check for dry-run detection. --check mode only detects without modifying, reporting which tasks have states that don’t match expectations. If it reports changes, the actual configuration has deviated from what the playbook declares.

# Ansible drift detection
ansible-playbook -i inventory_terraform.py site.yml --check --diff
# --check: don't actually execute changes, just detect
# --diff: show expected differences

Field Experience: At an e-commerce platform, we pushed a rule — all temporary changes must go through Ansible ad-hoc commands with git commit history, no direct SSH manual edits. Ops initially found it cumbersome, but after a month, configuration drift incidents dropped from 3-4 per week to 0. The key is making “change via tools” more convenient than “change via SSH” — wrapping common Ansible roles into one-click scripts that are faster than SSHing in and typing manually. This approach is discussed in more detail in Related: IaC Testing Pipeline.

What About Configurations Both Tools Can Manage?

Some configurations are genuinely “gray areas” — both Terraform and Ansible have the capability. SSH key distribution, for example: Terraform can manage it via cloud-init or alicloud_key_pair resources, and Ansible can manage it via the authorized_key module.

My recommendation: at the OS-perceptible layer, prefer Ansible; at the cloud API-perceptible layer, prefer Terraform.

SSH keys are a good example. Cloud platform SSH key binding (e.g., Alibaba Cloud ECS key pair) belongs to Terraform — it binds keys to instances via cloud API. But the in-instance authorized_keys file belongs to Ansible — it operates the file directly via SSH connection. These two layers don’t conflict; each manages its own.

5. CI/CD Integration: From Manual Scripts to Automated Pipelines

The previous decisions solved “how to collaborate,” but the ultimate goal is making the entire process automated — developer pushes code, CI/CD automatically runs Terraform plan, human review, apply, Ansible configuration, verification — all without SSHing into any machine.

Decision 5: Bring the TF + Ansible Collaboration Pipeline into the IaC Testing Framework

Many teams’ TF + Ansible pipelines lack testing. Terraform code changes go directly to apply; Ansible playbook changes go directly to execution. When something breaks, they roll back — but “rollback” in the infrastructure domain is a heavy operation.

A complete testing framework should include four layers:

Layer 1: Code Linting
├── Terraform: terraform fmt + terraform validate + tflint
├── Ansible: ansible-lint + yamllint
└── General: shellcheck (for shell scripts)

Layer 2: Plan Verification
├── Terraform: terraform plan (run in CI, output for human review)
└── Ansible: ansible-playbook --check (dry-run, detect expected changes)

Layer 3: Integration Testing
├── Execute full apply + playbook in test environment
├── Verify service availability
└── Auto-destroy test environment after tests pass

Layer 4: Production Deployment
├── Manual plan confirmation
├── apply + playbook
├── Health checks
└── Auto-rollback on failure

Here’s the Terraform code linting stage in CI:

# .gitlab-ci.yml - IaC linting stage
lint_terraform:
  stage: lint
  image: hashicorp/terraform:1.6
  script:
    - cd terraform/envs/prod
    - terraform fmt -check -recursive
    - terraform init -backend=false
    - terraform validate
  only:
    - merge_requests

lint_ansible:
  stage: lint
  image: ansible/ansible:latest
  script:
    - cd ansible
    - ansible-lint site.yml roles/
    - yamllint -d relaxed .
  only:
    - merge_requests

The integration testing stage concept: execute the full Terraform apply + Ansible playbook in a test environment, verify service availability, then auto-destroy test resources. This stage typically takes 5-10 minutes (depending on cloud resource creation speed and Ansible playbook complexity), but catches 90% of integration issues before merging to main.

# scripts/integration_test.py
"""Integration test: build temp env → configure → verify → destroy"""
import subprocess
import sys
import json

def run(cmd, cwd=None):
    """Execute command, returns (returncode, stdout, stderr)"""
    result = subprocess.run(
        cmd, shell=True, cwd=cwd,
        capture_output=True, text=True, timeout=600
    )
    return result.returncode, result.stdout, result.stderr

def main():
    test_dir = "terraform/envs/test"
    
    # 1. Build test environment
    code, out, err = run("terraform apply -auto-approve", cwd=test_dir)
    if code != 0:
        print(f"TF apply failed: {err}")
        sys.exit(1)
    
    try:
        # 2. Get test environment host info
        code, out, _ = run("terraform output -json", cwd=test_dir)
        outputs = json.loads(out)
        
        # 3. Ansible configuration
        code, out, err = run(
            "ansible-playbook -i inventory_terraform.py site.yml",
            cwd="ansible"
        )
        if code != 0:
            print(f"Ansible execution failed: {err}")
            sys.exit(1)
        
        # 4. Verify service availability
        ips = outputs.get("test_instance_ips", {}).get("value", [])
        for ip in ips:
            code, _, _ = run(f"curl -sf http://{ip}:80/healthz")
            if code != 0:
                print(f"Health check failed: {ip}")
                sys.exit(1)
        
        print("✅ Integration test passed")
    
    finally:
        # 5. Destroy test environment regardless of success/failure
        run("terraform destroy -auto-approve", cwd=test_dir)

if __name__ == "__main__":
    main()

Rollback Strategy

Infrastructure rollback is far more complex than application rollback. Application rollback is switching image versions, but after Terraform apply creates new resources and modifies configurations, rollback means returning to the previous state.

I recommend a layered rollback strategy:

Failure ScopeRollback MethodExecution TimeImpact Scope
Ansible config errorRevert Ansible code → re-run playbook2-5 minutesOnly OS config
Terraform resource config errorRevert TF code → terraform apply5-15 minutesMay involve resource recreation
Resource accidentally deletedRestore from State → terraform apply10-30 minutesMay lose data
Catastrophic failureRestore from backup + re-apply + configure30+ minutesFull recovery

Key principle: try Ansible rollback first, then Terraform if that doesn’t work. Because Ansible rollback is fast and has a small blast radius. Only touch Terraform when the infrastructure itself has issues.

War Story: During CI/CD platform development for a project, we hit an Ansible playbook template variable reference error that caused Nginx config file rendering failures on 20 machines. Thanks to the pipeline’s --diff output, we spotted the problem immediately. Rollback was simple — git revert that commit, re-run the pipeline, all 20 machines recovered within 5 minutes. Without a layered rollback strategy, we might have touched Terraform (rebuilding instances), which would take 30+ minutes and potentially lose data.

6. Performance Benchmarks and Production Considerations

Tool Performance Comparison

Benchmarked in a 50-instance environment:

OperationTerraformAnsibleNotes
Create 50 ECS4-6 minLimited by cloud API rate
Configure 50 hosts8-12 minLimited by SSH connections and task count
Drift detection15-30 sec3-5 minTF plan is fast, Ansible check is slow
Config rollback5-15 min2-5 minAnsible rollback is faster
Cleanup/destroy3-5 minTF destroy limited by API rate

Ansible Large-Scale Performance Optimization

When host count exceeds 100, Ansible’s default config becomes a bottleneck. Here are three key optimizations:

1. Adjust fork count

# ansible.cfg
[defaults]
forks = 50  # Default is 5, recommend 30-50 for 50+ hosts

Fork count controls how many hosts Ansible connects simultaneously. Default 5 is too conservative — 50 machines means 10 batches per task. Setting to 50 covers all in one round. But don’t set it unlimited — each fork process consumes memory. 200 forks for 200 machines might exhaust the Ansible controller’s memory.

2. Enable SSH Multiplexing

# ansible.cfg
[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=300s
pipelining = True

SSH multiplexing (ControlMaster) lets Ansible reuse SSH connections to the same host, avoiding re-establishing connections for each task. pipelining = True reduces temp file transfers by sending module code directly through the SSH pipe. Together, these can cut 50-machine playbook execution time by 40-50%.

3. Optimize Output with Callback Plugin

# ansible.cfg
[defaults]
stdout_callback = yaml  # Default is "default", yaml format is more readable

State File Security

Terraform State files contain complete infrastructure configuration — IP addresses, instance IDs, and potentially passwords and keys. This is the highest security risk file in Terraform + Ansible collaboration.

Production environments must:

  1. Remote State: Use OSS/S3 + DynamoDB (for locking) as backend, never leave State on local disk
  2. Encrypted Storage: Enable backend server-side encryption (SSE-S3 or SSE-KMS)
  3. Access Control: State file read/write permissions only for CI/CD service accounts and select SREs
  4. Sensitive Variable Masking: Mark sensitive outputs with sensitive = true to avoid exposing in logs
# Terraform backend configuration
terraform {
  backend "oss" {
    bucket   = "sre-tfstate-prod"
    prefix   = "terraform/prod"
    key      = "terraform.tfstate"
    region   = "cn-hangzhou"
    encrypt  = true
    acl      = "private"
  }
}

# Mark sensitive outputs
output "db_password" {
  value     = alicloud_rds_instance.main.password
  sensitive = true  # Won't show in terraform output
}

Security Reminder: Sensitive data on the Ansible side must also be encrypted. Use Ansible Vault to encrypt variable files containing passwords and keys. Never store them in plaintext — even in CI variables, use the principle of least privilege to restrict who can see them. (Related: Ansible Vault Password Management)

Summary

Terraform + Ansible collaboration isn’t as simple as “putting two tools together.” The core is getting five decision points right:

  1. Responsibility Boundary: Terraform owns cloud resource lifecycle, Ansible owns OS through application layer. Don’t use Terraform provisioners for complex configuration — it’s outside state management, drift goes undetected.

  2. Dynamic Inventory: Use Terraform State as the data source for Ansible dynamic inventory, making inventory auto-sync. Avoid manually maintaining hosts files — once elastic scaling kicks in, static inventory is dead.

  3. Two-Stage Pipeline: Split Terraform and Ansible into two pipeline stages with verification gates between them. Don’t use provisioners to trigger Ansible from within Terraform — error propagation is uncontrolled, debugging is hard.

  4. Configuration Drift Governance: Build a resource ownership matrix where each configuration item has exactly one owner. Terraform runs periodic plan for drift detection, Ansible runs periodic --check for deviation detection.

  5. IaC Testing Framework: Four layers — code linting, plan verification, integration testing, production deployment. Before merging to main, run a full apply + playbook in a test environment first.

The shared logic across these five decisions: isolation + ownership + automated verification. Two tools each manage their own domain, connected through an automated pipeline, with verification gates as guardrails. Don’t chase “one tool does everything” — Terraform trying to manage config turns into writing shell scripts in HCL, Ansible trying to manage infrastructure turns into calling cloud APIs in YAML. Use each tool for its strength, keep boundaries clear, and collaboration truly delivers its power.

One final thought: the essence of tool collaboration is team collaboration. The boundary between Terraform and Ansible is really the boundary between the “infrastructure team” and “configuration management team.” If these two teams work in silos without a shared repository and pipeline, even the best tool design gets shattered by organizational walls. When pushing IaC collaboration, start with unified repository, unified pipeline, clear ownership rules — tools are secondary, process and rules are fundamental.

References & Acknowledgments

The following resources were referenced during the writing of this article. Thanks to the original authors for their contributions:

  1. Ansible vs. Terraform — Red Hat Ansible official blog, articulating Ansible’s role as an orchestrator working alongside Terraform
  2. HashiCorp and Red Hat, better together — HashiCorp official blog, discussing the division of labor and integration directions between Terraform and Ansible for Day 1 and Day 2 operations
  3. Working with dynamic inventory — Ansible Community Documentation — Ansible official documentation on dynamic inventory mechanisms and plugins
  4. Terraform vs Ansible - Infrastructure as Code Showdown — Comparing Terraform and Ansible differences in security compliance and CI/CD integration, with GitOps workflow case studies
  5. IaC Dual Engine: Terraform + Ansible Complete Best Practices — CSDN blog post providing concrete implementation of Terraform outputs injected into Ansible Inventory
  6. Say Goodbye to Handwritten Inventory: Terraform and Ansible Integration on Azure — CSDN blog post on dynamic inventory auto-grouping practices in cloud environments
  7. tads-boilerplate — GitHub open-source project, a complete Terraform + Ansible + Docker Swarm boilerplate