Overview
Ansible has a low barrier to entry — a YAML file and a few lines of yum install can get things running. But when you face hundreds of servers, multiple environments, complex dependencies, and strict change audit requirements, the gap between “it runs” and “it’s production-ready” is an entire engineering discipline. This article distills the core practices of production-grade Playbooks into an actionable guide, covering the full chain from structure organization to performance optimization.
Reference: Ansible Official Best Practices
I. Playbook Structure Optimization
1.1 Directory Layout
A production-grade Ansible project should follow a standard directory structure. This isn’t an Ansible requirement — it’s a consensus formed after many teams hit the same pitfalls:
production-project/
├── ansible.cfg # Project-level configuration
├── inventory/
│ ├── production/
│ │ ├── hosts.ini # Production host inventory
│ │ └── group_vars/
│ │ ├── all.yml # Variables shared across all environments
│ │ ├── web.yml # Web group-specific variables
│ │ └── db.yml # DB group-specific variables
│ └── staging/
│ ├── hosts.ini
│ └── group_vars/
│ └── all.yml
├── roles/
│ ├── nginx/
│ │ ├── defaults/
│ │ │ └── main.yml # Default variables (low priority)
│ │ ├── vars/
│ │ │ └── main.yml # Role-internal variables (high priority)
│ │ ├── tasks/
│ │ │ └── main.yml
│ │ ├── handlers/
│ │ │ └── main.yml
│ │ ├── templates/
│ │ │ └── nginx.conf.j2
│ │ ├── files/
│ │ └── meta/
│ │ └── main.yml
│ └── postgresql/
├── playbooks/
│ ├── site.yml # Main entry point
│ ├── web.yml
│ └── db.yml
└── requirements.yml # Galaxy dependencies
1.2 Entry Point Design
site.yml is the main entry point for the entire Playbook and should provide an at-a-glance view of the architecture:
---
# site.yml - Main deployment entry point
- name: Deploy web tier
import_playbook: playbooks/web.yml
- name: Deploy database tier
import_playbook: playbooks/db.yml
- name: Deploy cache tier
import_playbook: playbooks/cache.yml
- name: Deploy monitoring tier
import_playbook: playbooks/monitoring.yml
Each sub-Playbook is responsible only for its own tier:
---
# playbooks/web.yml
- name: Configure web servers
hosts: web
become: true
roles:
- role: common
tags: [common, base]
- role: nginx
tags: [nginx, web]
- role: logrotate
tags: [logrotate]
Key principle: Keep entry points flat, use
import_playbookto organize the flow; each role does one thing only; use tags to control execution scope.
1.3 import vs include
Ansible provides two ways to include tasks, with significantly different behavior:
| Feature | import_* (static) | include_* (dynamic) |
|---|---|---|
| Parse time | At Playbook parse time | When execution reaches that point |
| Variable availability | Only variables known at parse time | Can use runtime dynamic variables |
| Loops | Not supported | Supported |
| Conditionals | Applied to all sub-tasks | Applied to the include statement itself |
| Tags | Inherited by all sub-tasks | Only applied to the include statement |
# Static import: determined at compile time, suitable for fixed structures
- import_tasks: install.yml
# Dynamic include: determined at runtime, suitable for conditional branching
- include_tasks: "{{ ansible_os_family | lower }}_setup.yml"
II. Variable Management
2.1 Variable Priority
Ansible has up to 22 levels of variable priority from low to high. Understanding the core levels covers 90% of scenarios:
| Priority | Variable Source | Typical Use |
|---|---|---|
| 1 (low) | role defaults | Safe default values for roles |
| 2 | inventory group_vars | Environment-level shared configuration |
| 3 | inventory host_vars | Single-host special configuration |
| 4 | play vars | Playbook-level variables |
| 5 | role vars | Role-internal enforced values |
| 6 | extra vars (-e) | Command-line overrides, highest priority |
Best practice: Put “values that may be overridden” in
defaults/, and “values required for the role to function properly” invars/. Default values should be safe enough that “the role runs without any variables passed.”
2.2 Variable Layered Organization
# inventory/production/group_vars/all.yml
# === Global shared variables ===
---
ntp_servers:
- ntp1.aliyun.com
- ntp2.aliyun.com
dns_servers:
- 223.5.5.5
- 8.8.8.8
sysctl_config:
net.core.somaxconn: 65535
vm.swappiness: 10
net.ipv4.tcp_max_syn_backlog: 65535
# inventory/production/group_vars/web.yml
# === Web tier-specific variables ===
---
nginx_worker_processes: auto
nginx_worker_connections: 10240
nginx_keepalive_timeout: 65
upstream_backends:
- { name: app1, host: 10.0.1.11, port: 8080 }
- { name: app2, host: 10.0.1.12, port: 8080 }
# inventory/production/host_vars/web-01.yml
# === Single-host variables ===
---
ansible_host: 10.0.1.21
nginx_worker_processes: 4 # Override group-level config
2.3 Variable Validation
Use the assert module to validate variables before execution, preventing errors from propagating downstream:
---
# roles/nginx/tasks/main.yml
- name: Validate required variables
assert:
that:
- nginx_worker_processes is defined
- nginx_worker_processes | int >= 1
- nginx_port in [80, 443, 8080, 8443]
- upstream_backends | length > 0
fail_msg: "Nginx role required variables are missing or invalid, check group_vars configuration"
success_msg: "Variable validation passed"
tags: [validate]
III. Role Design
3.1 Single Responsibility per Role
Good roles are like Unix tools — they do one thing and do it well. Compare these two designs:
# ❌ Bad design: one role does everything
roles/
└── lamp/
└── tasks/main.yml # Install Linux+Apache+MySQL+PHP all crammed here
# ✅ Good design: split into composable atomic roles
roles/
├── common/ # Base initialization
├── nginx/ # Web server
├── php-fpm/ # PHP runtime
├── mysql/ # Database
└── composer/ # Dependency management
3.2 Role Meta Dependency Declaration
Declare role dependencies through meta/main.yml — Ansible will execute them in order automatically:
---
# roles/php-fpm/meta/main.yml
dependencies:
- role: common
tags: [common]
- role: repo-remi
when: ansible_os_family == "RedHat"
Note: Role dependencies execute before the current role. Avoid circular dependencies and complex conditional logic in meta.
3.3 Idempotency Design
Idempotency is Ansible’s core advantage, but it’s not automatic — you must deliberately ensure it when writing tasks:
---
# ❌ Not idempotent: appends every time
- name: Configure hosts
shell: echo "10.0.1.10 app-server" >> /etc/hosts
# ✅ Idempotent: using the lineinfile module
- name: Ensure hosts entry exists
lineinfile:
path: /etc/hosts
line: "10.0.1.10 app-server"
state: present
# ✅ Idempotent: using blockinfile for multi-line config blocks
- name: Ensure hosts config block exists
blockinfile:
path: /etc/hosts
marker: "# {mark} ANSIBLE MANAGED BLOCK"
block: |
10.0.1.10 app-server
10.0.1.11 db-server
10.0.1.12 cache-server
state: present
IV. Conditionals and Loops
4.1 Conditional Logic
---
# Based on OS family
- name: Install Nginx (RedHat family)
yum:
name: nginx
state: present
when: ansible_os_family == "RedHat"
- name: Install Nginx (Debian family)
apt:
name: nginx
state: present
update_cache: true
when: ansible_os_family == "Debian"
# Based on variable flags
- name: Configure high-availability parameters
sysctl:
name: "{{ item.key }}"
value: "{{ item.value }}"
sysctl_set: true
loop: "{{ sysctl_config | dict2items }}"
when: enable_ha_tuning | default(false) | bool
# Based on command result
- name: Check if config rebuild is needed
command: nginx -t
register: nginx_test
changed_when: false
failed_when: false
- name: Reload Nginx configuration
systemd:
name: nginx
state: reloaded
when: nginx_test.rc == 0
4.2 Loop Patterns
---
# Basic loop
- name: Create multiple users
user:
name: "{{ item.name }}"
groups: "{{ item.groups }}"
shell: "{{ item.shell | default('/bin/bash') }}"
loop:
- { name: deploy, groups: www-data }
- { name: monitor, groups: www-data, shell: /sbin/nologin }
- { name: backup, groups: backup }
# Dictionary loop
- name: Create data directories
file:
path: "/data/{{ item.key }}"
state: directory
owner: "{{ item.value.owner }}"
mode: "{{ item.value.mode }}"
loop: "{{ data_dirs | dict2items }}"
# Filtered loop
- name: Alert on servers exceeding disk threshold
debug:
msg: "Disk alert: {{ item.mount }} usage {{ item.use_percent }}"
loop: "{{ ansible_mounts }}"
when: item.use_percent | regex_replace('%','') | int > 80
# Throttled loop (Ansible 2.12+)
- name: Batch download files
get_url:
url: "{{ item }}"
dest: "/tmp/{{ item | basename }}"
loop: "{{ file_list }}"
throttle: 5 # Limit concurrency to 5
V. Error Handling
5.1 block/rescue/always
This is Ansible’s try-catch-finally — essential for production:
---
- name: Database migration (with rollback)
block:
- name: Backup current database
command: "pg_dump {{ db_name }} > /backup/{{ db_name }}_{{ ansible_date_time.epoch }}.sql"
register: backup_result
- name: Execute database migration
command: "psql -d {{ db_name }} -f /tmp/migration.sql"
register: migrate_result
rescue:
- name: Migration failed, restore database
command: "psql -d {{ db_name }} < /backup/{{ db_name }}_{{ ansible_date_time.epoch }}.sql"
when: backup_result is succeeded
- name: Send failure alert
mail:
to: ops@example.com
subject: "[CRITICAL] Database migration failed - {{ inventory_hostname }}"
body: "Migration script execution failed, auto-rollback performed. Check /tmp/migration.sql"
always:
- name: Clean up temporary files
file:
path: /tmp/migration.sql
state: absent
5.2 failed_when Custom Failure Conditions
---
- name: Run health check script
command: /opt/app/health_check.sh
register: health_result
changed_when: false
failed_when:
- health_result.rc != 0
- "'CRITICAL' in health_result.stdout"
- name: Check application startup log
shell: "tail -100 {{ app_log_dir }}/startup.log"
register: startup_log
changed_when: false
failed_when: startup_log.rc != 0 or "'FATAL' in startup_log.stdout"
- name: Build project
command: make build
register: build_result
failed_when: build_result.rc != 0 or "'error' in build_result.stderr | lower"
5.3 changed_when Controlling Change Status
Non-Ansible module commands are marked as changed by default — you need to control this manually:
---
- name: Check if config needs updating
shell: diff /tmp/nginx.conf.new /etc/nginx/nginx.conf
register: config_diff
changed_when: false
failed_when: false
- name: Update Nginx config
copy:
src: /tmp/nginx.conf.new
dest: /etc/nginx/nginx.conf
remote_src: true
when: config_diff.rc != 0
notify: reload nginx
VI. Dynamic Inventory
6.1 Why Dynamic Inventory
Static hosts files work when physical servers are fixed. In cloud or K8s environments, nodes are added and removed constantly — static files can’t keep up. Dynamic Inventory queries cloud APIs in real time to return host lists.
6.2 AWS Dynamic Inventory
# ansible.cfg
[defaults]
inventory = inventory/aws_ec2.yml
host_key_checking = False
---
# inventory/aws_ec2.yml
plugin: aws_ec2
regions:
- cn-north-1
- cn-northwest-1
keyed_groups:
- key: tags.Environment
prefix: env
- key: tags.Role
prefix: role
- key: tags.Stack
prefix: stack
filters:
instance-state-name: running
tag:Project: production-platform
compose:
ansible_host: private_ip_address
ansible_user: "'ec2-user'"
host_vars:
ansible_python_interpreter: /usr/bin/python3
Verify:
# List all hosts
ansible-inventory -i inventory/aws_ec2.yml --list
# View by group
ansible-inventory -i inventory/aws_ec2.yml --graph
# View host details
ansible-inventory -i inventory/aws_ec2.yml --host 10.0.1.21
6.3 Custom Dynamic Inventory Script
When existing plugins don’t meet your needs, you can write a custom script in any language — it just needs to output JSON:
#!/usr/bin/env python3
"""Custom dynamic inventory: fetch host list from CMDB API"""
import json
import requests
import sys
def get_inventory():
resp = requests.get(
"https://cmdb.internal/api/v1/hosts",
headers={"Authorization": "Bearer ${CMDB_TOKEN}"},
params={"env": "production", "status": "active"},
timeout=10
)
hosts = resp.json()
inventory = {"_meta": {"hostvars": {}}}
for host in hosts:
group = host.get("role", "ungrouped")
if group not in inventory:
inventory[group] = {"hosts": [], "vars": {}}
inventory[group]["hosts"].append(host["hostname"])
inventory["_meta"]["hostvars"][host["hostname"]] = {
"ansible_host": host["ip"],
"ansible_port": host.get("ssh_port", 22),
"datacenter": host["datacenter"],
}
return inventory
if __name__ == "__main__":
# Support --list and --host parameters
if len(sys.argv) == 2 and sys.argv[1] == "--list":
print(json.dumps(get_inventory(), indent=2))
elif len(sys.argv) == 3 and sys.argv[1] == "--host":
print(json.dumps({}))
VII. Ansible Vault
7.1 Encrypting Sensitive Data
Passwords, keys, and certificates in production cannot be stored in plaintext in Git repositories. Ansible Vault provides transparent encryption:
# Create encrypted variable file
ansible-vault create inventory/production/group_vars/vault.yml
# Encrypt existing file
ansible-vault encrypt inventory/production/group_vars/secrets.yml
# Edit encrypted file (auto decrypt → edit → encrypt)
ansible-vault edit inventory/production/group_vars/vault.yml
# View encrypted file contents
ansible-vault view inventory/production/group_vars/vault.yml
# Change password
ansible-vault rekey inventory/production/group_vars/vault.yml
7.2 Multiple Vault Password Management
Production environments typically have multiple Vault passwords (dev/staging/production) — password files are more secure:
# ansible.cfg
[defaults]
vault_password_file = /etc/ansible/.vault_pass
# Use different password files for different environments
ansible-playbook site.yml -i inventory/production/ \
--vault-password-file /etc/ansible/.vault_prod
# Multiple password files (for merging encrypted variables from different sources)
ansible-playbook site.yml \
--vault-password-file /etc/ansible/.vault_prod \
--vault-password-file /etc/ansible/.vault_common
7.3 Encrypting Individual Variables
You don’t have to encrypt entire files — you can encrypt only sensitive fields:
---
# group_vars/all.yml (mixed file, partially encrypted)
db_host: 10.0.1.30
db_port: 5432
db_name: production_db
# The following is encrypted via ansible-vault encrypt_string
db_password: !vault |
$ANSIBLE_VAULT;1.1;AES256
62313365393831383739656138356465396664663339383738383338666637353766623638666139
...
api_secret: !vault |
$ANSIBLE_VAULT;1.1;AES256
37343734393438363531346337343363636136623633326432353062353366333062393733336532
...
# Encrypt a single variable value
ansible-vault encrypt_string 'MySecretPassword123' --name 'db_password'
# Encrypt and append to a specified file
ansible-vault encrypt_string 'sk-xxxxx' --name 'api_key' >> group_vars/all.yml
VIII. Galaxy Ecosystem
8.1 Using Community Roles
---
# requirements.yml - Declare role dependencies
roles:
- name: geerlingguy.nginx
version: 3.1.0
- name: geerlingguy.mysql
version: 4.3.0
- name: cloudalchemy.node_exporter
version: 2.0.0
collections:
- name: community.general
version: ">=6.0.0"
- name: ansible.posix
version: ">=1.5.0"
- name: community.docker
version: ">=3.0.0"
# Install dependencies
ansible-galaxy install -r requirements.yml
# Install to a specified path
ansible-galaxy install -r requirements.yml -p roles/
8.2 Evaluating Community Roles
Don’t blindly adopt community roles — use this evaluation checklist:
| Evaluation Dimension | Check Points | Method |
|---|---|---|
| Activity | Last commit time, Issue response speed | Check GitHub repo |
| Stars | Community recognition | Galaxy page |
| Test coverage | Has Molecule tests | Check repo molecule/ directory |
| Compatibility | Supported OS and Ansible versions | Check meta/main.yml |
| Security | Suspicious scripts, hardcoded keys | Review tasks/ content |
| Variable design | Reasonable defaults, documentation | Check defaults/main.yml |
Production recommendation: Fork critical roles to your own repository, lock the version, and perform a security audit before use. Don’t directly depend on the latest upstream tag.
IX. Debugging Techniques
9.1 Common Debugging Methods
# 1. Syntax check
ansible-playbook site.yml --syntax-check
# 2. Dry-run mode (see what would happen without executing)
ansible-playbook site.yml --check --diff
# 3. Step-by-step execution
ansible-playbook site.yml --step
# 4. Start from a specific task
ansible-playbook site.yml --start-at-task="install nginx"
# 5. Verbose output
ansible-playbook site.yml -vvv # Four levels: -v to -vvvv
# 6. List all hosts and tasks
ansible-playbook site.yml --list-hosts
ansible-playbook site.yml --list-tasks
ansible-playbook site.yml --list-tags
9.2 debug Module
---
# Print variable values
- name: Debug - show host info
debug:
var: ansible_default_ipv4
# Print custom messages
- name: Debug - show execution progress
debug:
msg: |
Current host: {{ inventory_hostname }}
OS: {{ ansible_distribution }} {{ ansible_distribution_version }}
Memory: {{ ansible_memtotal_mb }}MB
Disks: {{ ansible_mounts | map(attribute='mount') | list }}
# Conditional debugging
- name: Debug - output only in verbose mode
debug:
var: result
when: ansible_verbosity >= 2
# Use verbosity to control output level
- name: Debug - detailed output
debug:
var: complex_data_structure
verbosity: 2 # Requires -vv to display
9.3 Capturing Execution Results
---
- name: Execute script and capture full output
shell: /opt/app/deploy.sh 2>&1
register: deploy_output
changed_when: deploy_output.rc == 0
- name: Output execution result
debug:
msg: "{{ deploy_output.stdout_lines }}"
- name: Check execution result
assert:
that:
- deploy_output.rc == 0
- "'SUCCESS' in deploy_output.stdout"
fail_msg: "Deployment failed, output:\n{{ deploy_output.stderr }}"
X. Performance Optimization
10.1 SSH Connection Optimization
SSH connections are Ansible’s biggest performance bottleneck — optimizing them yields immediate results:
# ansible.cfg
[defaults]
# SSH pipelining mode, avoids repeated connection setup
pipelining = true
# Concurrency
forks = 50
# SSH timeout
timeout = 30
# Persistent connections
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o ServerAliveInterval=30
# Fact caching (avoid collecting facts every time)
gathering = smart
fact_caching = redis
fact_caching_timeout = 86400
fact_caching_connection = localhost:6379:0
10.2 Fact Caching
Before each Playbook execution, Ansible by default runs the setup module on all target hosts to collect facts — this is very time-consuming with hundreds of hosts. With fact caching enabled, facts are stored in Redis/JSON files and read directly on subsequent runs:
# Option 1: Redis cache (recommended, shared across machines)
[defaults]
fact_caching = redis
fact_caching_timeout = 86400
fact_caching_connection = localhost:6379:0
# Option 2: JSON file cache (single machine)
[defaults]
fact_caching = jsonfile
fact_caching_timeout = 86400
fact_caching_connection = /tmp/ansible_facts
10.3 On-Demand Fact Collection
---
# Skip fact collection entirely
- name: Quick task (no facts needed)
hosts: all
gather_facts: false
tasks:
- name: Restart service
systemd:
name: nginx
state: restarted
# Collect only needed facts
- name: Precise fact collection
hosts: all
gather_facts: true
module_defaults:
setup:
gather_subset:
- "!all"
- "network"
- "hardware"
10.4 Performance Comparison
Here’s a real-world comparison of a simple task across 200 hosts:
| Optimization | Duration | Improvement |
|---|---|---|
| Default config | 320s | Baseline |
| pipelining=true | 180s | 44% |
| forks=50 | 120s | 63% |
| fact_caching=redis | 65s | 80% |
| gather_facts=false | 45s | 86% |
| All optimizations combined | 28s | 91% |
10.5 Async Tasks
Use async for long-running tasks to avoid blocking:
---
- name: Large file download (async)
get_url:
url: "https://releases.example.com/data-{{ version }}.tar.gz"
dest: /tmp/data.tar.gz
async: 3600 # Timeout 1 hour
poll: 0 # Don't wait, return immediately
register: download_task
- name: Execute other tasks
debug:
msg: "Download running in background, continuing other work..."
- name: Wait for download to complete
async_status:
jid: "{{ download_task.ansible_job_id }}"
register: job_result
until: job_result.finished
retries: 120
delay: 30
XI. Production-Grade Configuration Template
11.1 Complete ansible.cfg
# ansible.cfg - Production-grade configuration
[defaults]
# Basic configuration
inventory = inventory/production
roles_path = roles
collections_path = collections
host_key_checking = False
timeout = 30
forks = 50
# Output
stdout_callback = yaml
callbacks_enabled = timer, profile_tasks, profile_roles
callbacks_whitelist = timer, profile_tasks
# SSH optimization
pipelining = true
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o ServerAliveInterval=30
# Fact caching
gathering = smart
fact_caching = redis
fact_caching_timeout = 86400
fact_caching_connection = localhost:6379:0
# Vault
vault_password_file = /etc/ansible/.vault_pass
# Error handling
any_errors_fatal = true
max_fail_percentage = 10
# Roles
private_role_vars = true
allow_world_readable_tmpfiles = false
[privilege_escalation]
become = true
become_method = sudo
[ssh_connection]
transfer_method = smart
retries = 3
11.2 Molecule Test Framework
Write automated tests for roles to ensure changes don’t introduce regressions:
---
# roles/nginx/molecule/default/converge.yml
- name: Test Nginx role
hosts: all
become: true
roles:
- role: nginx
# roles/nginx/molecule/default/verify.yml
- name: Verify Nginx installation
hosts: all
tasks:
- name: Check nginx service status
service:
name: nginx
state: started
enabled: true
check_mode: true
register: svc
- name: Assert service is started
assert:
that:
- not svc.changed
- name: Check port listening
wait_for:
port: 80
timeout: 5
- name: HTTP health check
uri:
url: http://localhost
status_code: 200
# Run tests
cd roles/nginx
molecule test
# Use Docker driver
molecule test --driver-name docker
Summary
The core of production-grade Ansible Playbooks lies in engineering thinking: it’s not just about piling commands into YAML — you need to systematically design across five dimensions: structure organization, variable management, error handling, performance optimization, and test coverage. Key takeaways from this article:
- Structure first: Standard directory layout + role-based decomposition + clear entry points — the foundation of maintainability
- Layered variables: defaults for safe defaults, group_vars for environment differences, extra vars for temporary overrides — know the priority levels inside out
- Idempotency above all: For every task, ask “what happens if I run it again?” — make good use of lineinfile, blockinfile, assert
- Recoverable errors: block/rescue is Ansible’s exception handling weapon — critical operations must have a rollback plan
- Security loop: Vault encrypts sensitive data, dynamic Inventory adapts to cloud environments, Galaxy dependencies must be version-locked
- Controllable performance: pipelining + forks + fact_caching — the three essentials that take 200 hosts from 5 minutes to 30 seconds
- Test-driven: Molecule makes role changes verifiable, CI integration makes Playbook quality measurable
Ansible’s value isn’t in replacing manual operations — it’s in making infrastructure configuration versionable, auditable, and reproducible. When you can turn hundreds of servers’ configuration into a PR that can be reviewed with a single git diff, operations has truly entered the engineering phase.
References & Acknowledgments
This article referenced the following materials during writing. We thank the original authors for their contributions:
- Ansible Official Best Practices — Ansible Community, referenced for Ansible Official Best Practices