Overview

Ansible has a low barrier to entry — a YAML file and a few lines of yum install can get things running. But when you face hundreds of servers, multiple environments, complex dependencies, and strict change audit requirements, the gap between “it runs” and “it’s production-ready” is an entire engineering discipline. This article distills the core practices of production-grade Playbooks into an actionable guide, covering the full chain from structure organization to performance optimization.

Reference: Ansible Official Best Practices

I. Playbook Structure Optimization

1.1 Directory Layout

A production-grade Ansible project should follow a standard directory structure. This isn’t an Ansible requirement — it’s a consensus formed after many teams hit the same pitfalls:

production-project/
├── ansible.cfg              # Project-level configuration
├── inventory/
   ├── production/
      ├── hosts.ini        # Production host inventory
      └── group_vars/
          ├── all.yml      # Variables shared across all environments
          ├── web.yml      # Web group-specific variables
          └── db.yml       # DB group-specific variables
   └── staging/
       ├── hosts.ini
       └── group_vars/
           └── all.yml
├── roles/
   ├── nginx/
      ├── defaults/
         └── main.yml     # Default variables (low priority)
      ├── vars/
         └── main.yml     # Role-internal variables (high priority)
      ├── tasks/
         └── main.yml
      ├── handlers/
         └── main.yml
      ├── templates/
         └── nginx.conf.j2
      ├── files/
      └── meta/
          └── main.yml
   └── postgresql/
├── playbooks/
   ├── site.yml             # Main entry point
   ├── web.yml
   └── db.yml
└── requirements.yml          # Galaxy dependencies

1.2 Entry Point Design

site.yml is the main entry point for the entire Playbook and should provide an at-a-glance view of the architecture:

---
# site.yml - Main deployment entry point
- name: Deploy web tier
  import_playbook: playbooks/web.yml

- name: Deploy database tier
  import_playbook: playbooks/db.yml

- name: Deploy cache tier
  import_playbook: playbooks/cache.yml

- name: Deploy monitoring tier
  import_playbook: playbooks/monitoring.yml

Each sub-Playbook is responsible only for its own tier:

---
# playbooks/web.yml
- name: Configure web servers
  hosts: web
  become: true
  roles:
    - role: common
      tags: [common, base]
    - role: nginx
      tags: [nginx, web]
    - role: logrotate
      tags: [logrotate]

Key principle: Keep entry points flat, use import_playbook to organize the flow; each role does one thing only; use tags to control execution scope.

1.3 import vs include

Ansible provides two ways to include tasks, with significantly different behavior:

Featureimport_* (static)include_* (dynamic)
Parse timeAt Playbook parse timeWhen execution reaches that point
Variable availabilityOnly variables known at parse timeCan use runtime dynamic variables
LoopsNot supportedSupported
ConditionalsApplied to all sub-tasksApplied to the include statement itself
TagsInherited by all sub-tasksOnly applied to the include statement
# Static import: determined at compile time, suitable for fixed structures
- import_tasks: install.yml

# Dynamic include: determined at runtime, suitable for conditional branching
- include_tasks: "{{ ansible_os_family | lower }}_setup.yml"

II. Variable Management

2.1 Variable Priority

Ansible has up to 22 levels of variable priority from low to high. Understanding the core levels covers 90% of scenarios:

PriorityVariable SourceTypical Use
1 (low)role defaultsSafe default values for roles
2inventory group_varsEnvironment-level shared configuration
3inventory host_varsSingle-host special configuration
4play varsPlaybook-level variables
5role varsRole-internal enforced values
6extra vars (-e)Command-line overrides, highest priority

Best practice: Put “values that may be overridden” in defaults/, and “values required for the role to function properly” in vars/. Default values should be safe enough that “the role runs without any variables passed.”

2.2 Variable Layered Organization

# inventory/production/group_vars/all.yml
# === Global shared variables ===
---
ntp_servers:
  - ntp1.aliyun.com
  - ntp2.aliyun.com
dns_servers:
  - 223.5.5.5
  - 8.8.8.8
sysctl_config:
  net.core.somaxconn: 65535
  vm.swappiness: 10
  net.ipv4.tcp_max_syn_backlog: 65535

# inventory/production/group_vars/web.yml
# === Web tier-specific variables ===
---
nginx_worker_processes: auto
nginx_worker_connections: 10240
nginx_keepalive_timeout: 65
upstream_backends:
  - { name: app1, host: 10.0.1.11, port: 8080 }
  - { name: app2, host: 10.0.1.12, port: 8080 }

# inventory/production/host_vars/web-01.yml
# === Single-host variables ===
---
ansible_host: 10.0.1.21
nginx_worker_processes: 4  # Override group-level config

2.3 Variable Validation

Use the assert module to validate variables before execution, preventing errors from propagating downstream:

---
# roles/nginx/tasks/main.yml
- name: Validate required variables
  assert:
    that:
      - nginx_worker_processes is defined
      - nginx_worker_processes | int >= 1
      - nginx_port in [80, 443, 8080, 8443]
      - upstream_backends | length > 0
    fail_msg: "Nginx role required variables are missing or invalid, check group_vars configuration"
    success_msg: "Variable validation passed"
  tags: [validate]

III. Role Design

3.1 Single Responsibility per Role

Good roles are like Unix tools — they do one thing and do it well. Compare these two designs:

# ❌ Bad design: one role does everything
roles/
  └── lamp/
      └── tasks/main.yml    # Install Linux+Apache+MySQL+PHP all crammed here

# ✅ Good design: split into composable atomic roles
roles/
  ├── common/               # Base initialization
  ├── nginx/                # Web server
  ├── php-fpm/              # PHP runtime
  ├── mysql/                # Database
  └── composer/             # Dependency management

3.2 Role Meta Dependency Declaration

Declare role dependencies through meta/main.yml — Ansible will execute them in order automatically:

---
# roles/php-fpm/meta/main.yml
dependencies:
  - role: common
    tags: [common]
  - role: repo-remi
    when: ansible_os_family == "RedHat"

Note: Role dependencies execute before the current role. Avoid circular dependencies and complex conditional logic in meta.

3.3 Idempotency Design

Idempotency is Ansible’s core advantage, but it’s not automatic — you must deliberately ensure it when writing tasks:

---
# ❌ Not idempotent: appends every time
- name: Configure hosts
  shell: echo "10.0.1.10 app-server" >> /etc/hosts

# ✅ Idempotent: using the lineinfile module
- name: Ensure hosts entry exists
  lineinfile:
    path: /etc/hosts
    line: "10.0.1.10 app-server"
    state: present

# ✅ Idempotent: using blockinfile for multi-line config blocks
- name: Ensure hosts config block exists
  blockinfile:
    path: /etc/hosts
    marker: "# {mark} ANSIBLE MANAGED BLOCK"
    block: |
      10.0.1.10 app-server
      10.0.1.11 db-server
      10.0.1.12 cache-server      
    state: present

IV. Conditionals and Loops

4.1 Conditional Logic

---
# Based on OS family
- name: Install Nginx (RedHat family)
  yum:
    name: nginx
    state: present
  when: ansible_os_family == "RedHat"

- name: Install Nginx (Debian family)
  apt:
    name: nginx
    state: present
    update_cache: true
  when: ansible_os_family == "Debian"

# Based on variable flags
- name: Configure high-availability parameters
  sysctl:
    name: "{{ item.key }}"
    value: "{{ item.value }}"
    sysctl_set: true
  loop: "{{ sysctl_config | dict2items }}"
  when: enable_ha_tuning | default(false) | bool

# Based on command result
- name: Check if config rebuild is needed
  command: nginx -t
  register: nginx_test
  changed_when: false
  failed_when: false

- name: Reload Nginx configuration
  systemd:
    name: nginx
    state: reloaded
  when: nginx_test.rc == 0

4.2 Loop Patterns

---
# Basic loop
- name: Create multiple users
  user:
    name: "{{ item.name }}"
    groups: "{{ item.groups }}"
    shell: "{{ item.shell | default('/bin/bash') }}"
  loop:
    - { name: deploy, groups: www-data }
    - { name: monitor, groups: www-data, shell: /sbin/nologin }
    - { name: backup, groups: backup }

# Dictionary loop
- name: Create data directories
  file:
    path: "/data/{{ item.key }}"
    state: directory
    owner: "{{ item.value.owner }}"
    mode: "{{ item.value.mode }}"
  loop: "{{ data_dirs | dict2items }}"

# Filtered loop
- name: Alert on servers exceeding disk threshold
  debug:
    msg: "Disk alert: {{ item.mount }} usage {{ item.use_percent }}"
  loop: "{{ ansible_mounts }}"
  when: item.use_percent | regex_replace('%','') | int > 80

# Throttled loop (Ansible 2.12+)
- name: Batch download files
  get_url:
    url: "{{ item }}"
    dest: "/tmp/{{ item | basename }}"
  loop: "{{ file_list }}"
  throttle: 5  # Limit concurrency to 5

V. Error Handling

5.1 block/rescue/always

This is Ansible’s try-catch-finally — essential for production:

---
- name: Database migration (with rollback)
  block:
    - name: Backup current database
      command: "pg_dump {{ db_name }} > /backup/{{ db_name }}_{{ ansible_date_time.epoch }}.sql"
      register: backup_result

    - name: Execute database migration
      command: "psql -d {{ db_name }} -f /tmp/migration.sql"
      register: migrate_result

  rescue:
    - name: Migration failed, restore database
      command: "psql -d {{ db_name }} < /backup/{{ db_name }}_{{ ansible_date_time.epoch }}.sql"
      when: backup_result is succeeded

    - name: Send failure alert
      mail:
        to: ops@example.com
        subject: "[CRITICAL] Database migration failed - {{ inventory_hostname }}"
        body: "Migration script execution failed, auto-rollback performed. Check /tmp/migration.sql"

  always:
    - name: Clean up temporary files
      file:
        path: /tmp/migration.sql
        state: absent

5.2 failed_when Custom Failure Conditions

---
- name: Run health check script
  command: /opt/app/health_check.sh
  register: health_result
  changed_when: false
  failed_when:
    - health_result.rc != 0
    - "'CRITICAL' in health_result.stdout"

- name: Check application startup log
  shell: "tail -100 {{ app_log_dir }}/startup.log"
  register: startup_log
  changed_when: false
  failed_when: startup_log.rc != 0 or "'FATAL' in startup_log.stdout"

- name: Build project
  command: make build
  register: build_result
  failed_when: build_result.rc != 0 or "'error' in build_result.stderr | lower"

5.3 changed_when Controlling Change Status

Non-Ansible module commands are marked as changed by default — you need to control this manually:

---
- name: Check if config needs updating
  shell: diff /tmp/nginx.conf.new /etc/nginx/nginx.conf
  register: config_diff
  changed_when: false
  failed_when: false

- name: Update Nginx config
  copy:
    src: /tmp/nginx.conf.new
    dest: /etc/nginx/nginx.conf
    remote_src: true
  when: config_diff.rc != 0
  notify: reload nginx

VI. Dynamic Inventory

6.1 Why Dynamic Inventory

Static hosts files work when physical servers are fixed. In cloud or K8s environments, nodes are added and removed constantly — static files can’t keep up. Dynamic Inventory queries cloud APIs in real time to return host lists.

6.2 AWS Dynamic Inventory

# ansible.cfg
[defaults]
inventory = inventory/aws_ec2.yml
host_key_checking = False
---
# inventory/aws_ec2.yml
plugin: aws_ec2
regions:
  - cn-north-1
  - cn-northwest-1
keyed_groups:
  - key: tags.Environment
    prefix: env
  - key: tags.Role
    prefix: role
  - key: tags.Stack
    prefix: stack
filters:
  instance-state-name: running
  tag:Project: production-platform
compose:
  ansible_host: private_ip_address
  ansible_user: "'ec2-user'"
host_vars:
  ansible_python_interpreter: /usr/bin/python3

Verify:

# List all hosts
ansible-inventory -i inventory/aws_ec2.yml --list

# View by group
ansible-inventory -i inventory/aws_ec2.yml --graph

# View host details
ansible-inventory -i inventory/aws_ec2.yml --host 10.0.1.21

6.3 Custom Dynamic Inventory Script

When existing plugins don’t meet your needs, you can write a custom script in any language — it just needs to output JSON:

#!/usr/bin/env python3
"""Custom dynamic inventory: fetch host list from CMDB API"""
import json
import requests
import sys

def get_inventory():
    resp = requests.get(
        "https://cmdb.internal/api/v1/hosts",
        headers={"Authorization": "Bearer ${CMDB_TOKEN}"},
        params={"env": "production", "status": "active"},
        timeout=10
    )
    hosts = resp.json()

    inventory = {"_meta": {"hostvars": {}}}
    
    for host in hosts:
        group = host.get("role", "ungrouped")
        if group not in inventory:
            inventory[group] = {"hosts": [], "vars": {}}
        
        inventory[group]["hosts"].append(host["hostname"])
        inventory["_meta"]["hostvars"][host["hostname"]] = {
            "ansible_host": host["ip"],
            "ansible_port": host.get("ssh_port", 22),
            "datacenter": host["datacenter"],
        }
    
    return inventory

if __name__ == "__main__":
    # Support --list and --host parameters
    if len(sys.argv) == 2 and sys.argv[1] == "--list":
        print(json.dumps(get_inventory(), indent=2))
    elif len(sys.argv) == 3 and sys.argv[1] == "--host":
        print(json.dumps({}))

VII. Ansible Vault

7.1 Encrypting Sensitive Data

Passwords, keys, and certificates in production cannot be stored in plaintext in Git repositories. Ansible Vault provides transparent encryption:

# Create encrypted variable file
ansible-vault create inventory/production/group_vars/vault.yml

# Encrypt existing file
ansible-vault encrypt inventory/production/group_vars/secrets.yml

# Edit encrypted file (auto decrypt → edit → encrypt)
ansible-vault edit inventory/production/group_vars/vault.yml

# View encrypted file contents
ansible-vault view inventory/production/group_vars/vault.yml

# Change password
ansible-vault rekey inventory/production/group_vars/vault.yml

7.2 Multiple Vault Password Management

Production environments typically have multiple Vault passwords (dev/staging/production) — password files are more secure:

# ansible.cfg
[defaults]
vault_password_file = /etc/ansible/.vault_pass
# Use different password files for different environments
ansible-playbook site.yml -i inventory/production/ \
  --vault-password-file /etc/ansible/.vault_prod

# Multiple password files (for merging encrypted variables from different sources)
ansible-playbook site.yml \
  --vault-password-file /etc/ansible/.vault_prod \
  --vault-password-file /etc/ansible/.vault_common

7.3 Encrypting Individual Variables

You don’t have to encrypt entire files — you can encrypt only sensitive fields:

---
# group_vars/all.yml (mixed file, partially encrypted)
db_host: 10.0.1.30
db_port: 5432
db_name: production_db
# The following is encrypted via ansible-vault encrypt_string
db_password: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          62313365393831383739656138356465396664663339383738383338666637353766623638666139
          ...
api_secret: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          37343734393438363531346337343363636136623633326432353062353366333062393733336532
          ...
# Encrypt a single variable value
ansible-vault encrypt_string 'MySecretPassword123' --name 'db_password'

# Encrypt and append to a specified file
ansible-vault encrypt_string 'sk-xxxxx' --name 'api_key' >> group_vars/all.yml

VIII. Galaxy Ecosystem

8.1 Using Community Roles

---
# requirements.yml - Declare role dependencies
roles:
  - name: geerlingguy.nginx
    version: 3.1.0
  - name: geerlingguy.mysql
    version: 4.3.0
  - name: cloudalchemy.node_exporter
    version: 2.0.0
collections:
  - name: community.general
    version: ">=6.0.0"
  - name: ansible.posix
    version: ">=1.5.0"
  - name: community.docker
    version: ">=3.0.0"
# Install dependencies
ansible-galaxy install -r requirements.yml

# Install to a specified path
ansible-galaxy install -r requirements.yml -p roles/

8.2 Evaluating Community Roles

Don’t blindly adopt community roles — use this evaluation checklist:

Evaluation DimensionCheck PointsMethod
ActivityLast commit time, Issue response speedCheck GitHub repo
StarsCommunity recognitionGalaxy page
Test coverageHas Molecule testsCheck repo molecule/ directory
CompatibilitySupported OS and Ansible versionsCheck meta/main.yml
SecuritySuspicious scripts, hardcoded keysReview tasks/ content
Variable designReasonable defaults, documentationCheck defaults/main.yml

Production recommendation: Fork critical roles to your own repository, lock the version, and perform a security audit before use. Don’t directly depend on the latest upstream tag.

IX. Debugging Techniques

9.1 Common Debugging Methods

# 1. Syntax check
ansible-playbook site.yml --syntax-check

# 2. Dry-run mode (see what would happen without executing)
ansible-playbook site.yml --check --diff

# 3. Step-by-step execution
ansible-playbook site.yml --step

# 4. Start from a specific task
ansible-playbook site.yml --start-at-task="install nginx"

# 5. Verbose output
ansible-playbook site.yml -vvv  # Four levels: -v to -vvvv

# 6. List all hosts and tasks
ansible-playbook site.yml --list-hosts
ansible-playbook site.yml --list-tasks
ansible-playbook site.yml --list-tags

9.2 debug Module

---
# Print variable values
- name: Debug - show host info
  debug:
    var: ansible_default_ipv4

# Print custom messages
- name: Debug - show execution progress
  debug:
    msg: |
      Current host: {{ inventory_hostname }}
      OS: {{ ansible_distribution }} {{ ansible_distribution_version }}
      Memory: {{ ansible_memtotal_mb }}MB
      Disks: {{ ansible_mounts | map(attribute='mount') | list }}      

# Conditional debugging
- name: Debug - output only in verbose mode
  debug:
    var: result
  when: ansible_verbosity >= 2

# Use verbosity to control output level
- name: Debug - detailed output
  debug:
    var: complex_data_structure
  verbosity: 2  # Requires -vv to display

9.3 Capturing Execution Results

---
- name: Execute script and capture full output
  shell: /opt/app/deploy.sh 2>&1
  register: deploy_output
  changed_when: deploy_output.rc == 0

- name: Output execution result
  debug:
    msg: "{{ deploy_output.stdout_lines }}"

- name: Check execution result
  assert:
    that:
      - deploy_output.rc == 0
      - "'SUCCESS' in deploy_output.stdout"
    fail_msg: "Deployment failed, output:\n{{ deploy_output.stderr }}"

X. Performance Optimization

10.1 SSH Connection Optimization

SSH connections are Ansible’s biggest performance bottleneck — optimizing them yields immediate results:

# ansible.cfg
[defaults]
# SSH pipelining mode, avoids repeated connection setup
pipelining = true
# Concurrency
forks = 50
# SSH timeout
timeout = 30
# Persistent connections
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o ServerAliveInterval=30
# Fact caching (avoid collecting facts every time)
gathering = smart
fact_caching = redis
fact_caching_timeout = 86400
fact_caching_connection = localhost:6379:0

10.2 Fact Caching

Before each Playbook execution, Ansible by default runs the setup module on all target hosts to collect facts — this is very time-consuming with hundreds of hosts. With fact caching enabled, facts are stored in Redis/JSON files and read directly on subsequent runs:

# Option 1: Redis cache (recommended, shared across machines)
[defaults]
fact_caching = redis
fact_caching_timeout = 86400
fact_caching_connection = localhost:6379:0

# Option 2: JSON file cache (single machine)
[defaults]
fact_caching = jsonfile
fact_caching_timeout = 86400
fact_caching_connection = /tmp/ansible_facts

10.3 On-Demand Fact Collection

---
# Skip fact collection entirely
- name: Quick task (no facts needed)
  hosts: all
  gather_facts: false
  tasks:
    - name: Restart service
      systemd:
        name: nginx
        state: restarted

# Collect only needed facts
- name: Precise fact collection
  hosts: all
  gather_facts: true
  module_defaults:
    setup:
      gather_subset:
        - "!all"
        - "network"
        - "hardware"

10.4 Performance Comparison

Here’s a real-world comparison of a simple task across 200 hosts:

OptimizationDurationImprovement
Default config320sBaseline
pipelining=true180s44%
forks=50120s63%
fact_caching=redis65s80%
gather_facts=false45s86%
All optimizations combined28s91%

10.5 Async Tasks

Use async for long-running tasks to avoid blocking:

---
- name: Large file download (async)
  get_url:
    url: "https://releases.example.com/data-{{ version }}.tar.gz"
    dest: /tmp/data.tar.gz
  async: 3600          # Timeout 1 hour
  poll: 0              # Don't wait, return immediately
  register: download_task

- name: Execute other tasks
  debug:
    msg: "Download running in background, continuing other work..."

- name: Wait for download to complete
  async_status:
    jid: "{{ download_task.ansible_job_id }}"
  register: job_result
  until: job_result.finished
  retries: 120
  delay: 30

XI. Production-Grade Configuration Template

11.1 Complete ansible.cfg

# ansible.cfg - Production-grade configuration
[defaults]
# Basic configuration
inventory = inventory/production
roles_path = roles
collections_path = collections
host_key_checking = False
timeout = 30
forks = 50

# Output
stdout_callback = yaml
callbacks_enabled = timer, profile_tasks, profile_roles
callbacks_whitelist = timer, profile_tasks

# SSH optimization
pipelining = true
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o ServerAliveInterval=30

# Fact caching
gathering = smart
fact_caching = redis
fact_caching_timeout = 86400
fact_caching_connection = localhost:6379:0

# Vault
vault_password_file = /etc/ansible/.vault_pass

# Error handling
any_errors_fatal = true
max_fail_percentage = 10

# Roles
private_role_vars = true
allow_world_readable_tmpfiles = false

[privilege_escalation]
become = true
become_method = sudo

[ssh_connection]
transfer_method = smart
retries = 3

11.2 Molecule Test Framework

Write automated tests for roles to ensure changes don’t introduce regressions:

---
# roles/nginx/molecule/default/converge.yml
- name: Test Nginx role
  hosts: all
  become: true
  roles:
    - role: nginx

# roles/nginx/molecule/default/verify.yml
- name: Verify Nginx installation
  hosts: all
  tasks:
    - name: Check nginx service status
      service:
        name: nginx
        state: started
        enabled: true
      check_mode: true
      register: svc

    - name: Assert service is started
      assert:
        that:
          - not svc.changed

    - name: Check port listening
      wait_for:
        port: 80
        timeout: 5

    - name: HTTP health check
      uri:
        url: http://localhost
        status_code: 200
# Run tests
cd roles/nginx
molecule test

# Use Docker driver
molecule test --driver-name docker

Summary

The core of production-grade Ansible Playbooks lies in engineering thinking: it’s not just about piling commands into YAML — you need to systematically design across five dimensions: structure organization, variable management, error handling, performance optimization, and test coverage. Key takeaways from this article:

  1. Structure first: Standard directory layout + role-based decomposition + clear entry points — the foundation of maintainability
  2. Layered variables: defaults for safe defaults, group_vars for environment differences, extra vars for temporary overrides — know the priority levels inside out
  3. Idempotency above all: For every task, ask “what happens if I run it again?” — make good use of lineinfile, blockinfile, assert
  4. Recoverable errors: block/rescue is Ansible’s exception handling weapon — critical operations must have a rollback plan
  5. Security loop: Vault encrypts sensitive data, dynamic Inventory adapts to cloud environments, Galaxy dependencies must be version-locked
  6. Controllable performance: pipelining + forks + fact_caching — the three essentials that take 200 hosts from 5 minutes to 30 seconds
  7. Test-driven: Molecule makes role changes verifiable, CI integration makes Playbook quality measurable

Ansible’s value isn’t in replacing manual operations — it’s in making infrastructure configuration versionable, auditable, and reproducible. When you can turn hundreds of servers’ configuration into a PR that can be reviewed with a single git diff, operations has truly entered the engineering phase.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

  1. Ansible Official Best Practices — Ansible Community, referenced for Ansible Official Best Practices