paramiko: SSH Batch Management Fundamentals
paramiko is a Python implementation of the SSHv2 protocol and the foundational building block for ops automation. When you need fine-grained control over SSH connections or need to handle non-standard scenarios, paramiko offers maximum flexibility.
Basic Connection and Command Execution
import paramiko
import time
def ssh_exec(host, port, username, password, command):
"""Basic SSH command execution"""
client = paramiko.SSHClient()
# Auto-add host keys (use known_hosts in production)
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
client.connect(host, port=port, username=username, password=password, timeout=10)
stdin, stdout, stderr = client.exec_command(command)
# Get exit code
exit_code = stdout.channel.recv_exit_status()
output = stdout.read().decode().strip()
error = stderr.read().decode().strip()
return {
'host': host,
'exit_code': exit_code,
'output': output,
'error': error
}
finally:
client.close()
# Usage example
result = ssh_exec('10.0.1.10', 22, 'root', 'password', 'uname -r')
print(f"{result['host']}: exit={result['exit_code']}, output={result['output']}")
Connection Pool and Concurrent Execution
When managing hundreds of servers in production, you need connection pooling and concurrency control:
import paramiko
from concurrent.futures import ThreadPoolExecutor, as_completed
from threading import Lock
class SSHConnectionPool:
"""SSH connection pool — reuses connections to reduce handshake overhead"""
def __init__(self, max_connections=20):
self.pool = {} # {host: SSHClient}
self.lock = Lock()
self.max_connections = max_connections
def get_connection(self, host, port, username, password):
with self.lock:
if host in self.pool and self.pool[host].get_transport().is_active():
return self.pool[host]
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(host, port=port, username=username, password=password, timeout=10)
self.pool[host] = client
return client
def close_all(self):
with self.lock:
for client in self.pool.values():
client.close()
self.pool.clear()
def batch_execute(hosts, command, max_workers=10):
"""Concurrent batch command execution"""
pool = SSHConnectionPool()
results = []
def exec_on_host(host_info):
host = host_info['host']
try:
client = pool.get_connection(
host, host_info['port'],
host_info['username'], host_info['password']
)
stdin, stdout, stderr = client.exec_command(command, timeout=30)
return {
'host': host,
'exit_code': stdout.channel.recv_exit_status(),
'output': stdout.read().decode().strip()
}
except Exception as e:
return {'host': host, 'exit_code': -1, 'output': str(e)}
with ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = {executor.submit(exec_on_host, h): h for h in hosts}
for future in as_completed(futures):
results.append(future.result())
pool.close_all()
return results
# Batch execution example
hosts = [
{'host': '10.0.1.10', 'port': 22, 'username': 'root', 'password': 'pass'},
{'host': '10.0.1.11', 'port': 22, 'username': 'root', 'password': 'pass'},
{'host': '10.0.1.12', 'port': 22, 'username': 'root', 'password': 'pass'},
]
results = batch_execute(hosts, 'df -h / && free -m')
for r in results:
print(f"[{r['host']}] exit={r['exit_code']}")
print(r['output'])
print('-' * 60)
SFTP File Transfer
def sftp_upload(host, port, username, password, local_path, remote_path):
"""SFTP file upload"""
transport = paramiko.Transport((host, port))
try:
transport.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(transport)
sftp.put(local_path, remote_path)
# Verify file size
local_size = os.path.getsize(local_path)
remote_size = sftp.stat(remote_path).st_size
assert local_size == remote_size, f"Size mismatch: {local_size} vs {remote_size}"
return True
finally:
transport.close()
Fabric: Simplifying Remote Operations
Fabric wraps a higher-level API on top of paramiko, reducing common SSH operations to simple function calls:
from fabric import Connection, Config
# Batch deployment config
env_config = Config({
'run': {'warn': True, 'echo': True, 'timeout': 30},
'sudo': {'password': 'sudo-password'},
})
def deploy_nginx(c):
"""Deploy Nginx via Fabric"""
c.run('yum install -y nginx')
c.put('nginx.conf', '/etc/nginx/nginx.conf', use_sudo=True)
c.sudo('systemctl enable nginx')
c.sudo('systemctl restart nginx')
c.run('nginx -t')
# Execute deployment
for host in ['10.0.1.10', '10.0.1.11']:
conn = Connection(host=host, user='root', config=env_config)
deploy_nginx(conn)
conn.close()
Fabric’s strength lies in its declarative API — you describe “what to do,” and the framework handles connection management, error handling, and other details. It’s well-suited for medium-scale ops tasks, but once you exceed a hundred machines, maintaining the scripts themselves becomes complex.
Ansible: Declarative Automation
When your infrastructure scales to hundreds or thousands of servers, Ansible becomes the better choice. It requires no agents, uses YAML for declarative descriptions, and ships with a rich set of built-in modules.
Core Concepts
| Concept | Description | Analogy |
|---|---|---|
| Inventory | Host list | Server list |
| Module | Execution unit | Function |
| Task | An action that invokes a module | Function call |
| Playbook | Collection of tasks | Script |
| Role | Reusable task package | Function library |
| Variables | Parameterized configuration | Variables |
Inventory Host List
# /etc/ansible/hosts
[webservers]
web-[01:03].sre.wang # Wildcard match: web-01, web-02, web-03
[databases]
db-01.sre.wang ansible_host=10.0.2.10
db-02.sre.wang ansible_host=10.0.2.11
[webservers:vars]
ansible_user=deploy
ansible_ssh_private_key_file=~/.ssh/id_ed25519
nginx_version=1.25.3
[databases:vars]
ansible_user=deploy
mysql_version=8.0.35
[production:children]
webservers
databases
Ansible in Action: Batch Nginx Deployment
---
# playbook: deploy-nginx.yml
- name: Batch deploy and configure Nginx
hosts: webservers
become: yes
vars:
nginx_version: "1.25.3"
nginx_worker_processes: "auto"
nginx_worker_connections: "10240"
tasks:
- name: Install EPEL repository
yum:
name: epel-release
state: present
- name: Install Nginx
yum:
name: "nginx-{{ nginx_version }}"
state: present
notify: restart nginx
- name: Create Nginx config directory
file:
path: /etc/nginx/conf.d
state: directory
mode: '0755'
- name: Deploy main config file
template:
src: nginx.conf.j2
dest: /etc/nginx/nginx.conf
backup: yes
validate: nginx -t -c %s
notify: reload nginx
- name: Deploy site configurations
template:
src: site.conf.j2
dest: "/etc/nginx/conf.d/{{ item.name }}.conf"
loop:
- { name: 'app', port: 8080, server_name: 'app.sre.wang' }
- { name: 'api', port: 8081, server_name: 'api.sre.wang' }
notify: reload nginx
- name: Ensure Nginx is started and enabled on boot
systemd:
name: nginx
state: started
enabled: yes
- name: Open firewall ports
firewalld:
port: "{{ item }}/tcp"
permanent: yes
immediate: yes
state: enabled
loop: [80, 443]
handlers:
- name: restart nginx
systemd:
name: nginx
state: restarted
- name: reload nginx
systemd:
name: nginx
state: reloaded
Jinja2 template file nginx.conf.j2:
# {{ ansible_managed }}
worker_processes {{ nginx_worker_processes }};
worker_rlimit_nofile 65535;
events {
worker_connections {{ nginx_worker_connections }};
use epoll;
multi_accept on;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
server_tokens off;
gzip on;
gzip_min_length 1k;
gzip_types text/plain application/json application/javascript text/css;
include /etc/nginx/conf.d/*.conf;
}
Role Directory Structure
When a Playbook grows to hundreds of lines, you should split it into Roles:
roles/nginx/
├── defaults/
│ └── main.yml # Default variables
├── vars/
│ └── main.yml # Higher-priority variables
├── tasks/
│ └── main.yml # Main tasks
├── handlers/
│ └── main.yml # Triggers
├── templates/
│ ├── nginx.conf.j2
│ └── site.conf.j2
└── meta/
└── main.yml # Dependency declarations
Architectural Considerations: Migrating from paramiko to Ansible
Evolution Path
Manual SSH on single host → paramiko scripts → Fabric wrapper → Ansible declarative
│ │ │ │
│ │ │ │
1-5 hosts 5-50 hosts 50-100 hosts 100+ hosts
Comparison
| Dimension | paramiko | Fabric | Ansible |
|---|---|---|---|
| Abstraction level | Protocol-level | Function-level | Declarative |
| Agent | None | None | None |
| Idempotency | Manual implementation | Manual implementation | Built into modules |
| Template system | None | None | Jinja2 |
| Role reuse | None | Weak | Strong |
| Learning curve | Low | Low | Medium |
| Scale | Small | Medium | Large |
| Maintainability | Poor | Medium | High |
When paramiko Is Still the Better Choice
Ansible isn’t a silver bullet. In these scenarios, paramiko remains the better option:
- Non-standard SSH scenarios: Interactive sessions, port forwarding, SSH tunnels
- Complex conditional logic: Dynamic decision-making based on real-time output from the previous command
- Embedded environments: Lightweight devices where Ansible cannot be installed
- Performance-sensitive use cases: Fine-grained control over timeouts, reconnection, and connection reuse
Hybrid Architecture in Practice
A hybrid approach is recommended for production environments:
# Use paramiko for health checks and pre-flight validation
def pre_check(host):
result = ssh_exec(host, 22, 'deploy', 'pass', 'uname -r')
kernel = result['output']
# Select different Ansible Playbooks based on kernel version
if '4.18' in kernel:
return 'playbook-rhel8.yml'
else:
return 'playbook-rhel7.yml'
# Use Ansible for actual deployment
import subprocess
for host in hosts:
playbook = pre_check(host['host'])
subprocess.run([
'ansible-playbook', playbook,
'-l', host['host'],
'-e', f'target_host={host["host"]}'
])
Summary
Ops automation is not a tool selection problem — it’s an architectural evolution problem. paramiko provides protocol-level SSH control, Fabric simplifies common operations, and Ansible solves maintainability at scale with its declarative model.
The key principle: use paramiko/Fabric for flexibility at small scale, use Ansible for idempotency and maintainability at large scale, and combine both to leverage their respective strengths. Regardless of which tool you choose, the ultimate goal remains the same: reduce manual operations, improve consistency, and make infrastructure auditable and reproducible.
References & Acknowledgments
This article referenced the following materials during writing. We thank the original authors for their contributions: