Introduction

Shell scripts are the most common automation tool for operations engineers. This post records several practical patterns used in the field.

Log Rotation Cleanup

#!/bin/bash
# clean_logs.sh - Log cleanup script
LOG_DIR="/var/log/app"
KEEP_DAYS=30

find "$LOG_DIR" -name "*.log" -type f -mtime +${KEEP_DAYS} -exec gzip {} \;
find "$LOG_DIR" -name "*.gz" -type f -mtime +${KEEP_DAYS} -delete

echo "$(date): Log cleanup complete, kept ${KEEP_DAYS} days"

Run daily via crontab:

0 2 * * * /opt/scripts/clean_logs.sh >> /var/log/clean_logs.log 2>&1

Batch Health Check

Check multiple servers via SSH:

#!/bin/bash
# health_check.sh - Batch health check
HOSTS=("web-01" "web-02" "db-01" "cache-01")

for host in "${HOSTS[@]}"; do
    echo "--- ${host} ---"
    ssh "$host" "
        echo \"CPU: \$(top -bn1 | grep 'Cpu' | awk '{print \$2}')%\"
        echo \"MEM: \$(free | awk '/Mem/{printf \"%.0f\", \$3/\$2*100}')%\"
        echo \"DISK: \$(df -h / | awk 'NR==2{print \$5}')\"
    " 2>/dev/null || echo "Connection failed"
done

Auto-Restart on Failure

Detect service failure and auto-recover:

#!/bin/bash
# auto_restart.sh - Service auto-restart
SERVICE="nginx"
MAX_RETRY=3

for i in $(seq 1 $MAX_RETRY); do
    if systemctl is-active --quiet "$SERVICE"; then
        exit 0
    fi
    echo "$(date): ${SERVICE} not running, attempting restart (attempt ${i})"
    systemctl restart "$SERVICE"
    sleep 5
done

# Retry failed, send alert
echo "$(date): ${SERVICE} restart failed, manual intervention required" | \
    mail -s "Alert: ${SERVICE} abnormal" admin@example.com

Config File Backup

Regularly backup critical configs with timestamps:

#!/bin/bash
# backup_conf.sh - Config backup
BACKUP_DIR="/data/backup/config"
DATE=$(date +%Y%m%d_%H%M%S)
CONFIGS=(
    "/etc/nginx/nginx.conf"
    "/etc/ssh/sshd_config"
    "/etc/sysctl.conf"
)

mkdir -p "$BACKUP_DIR"
for conf in "${CONFIGS[@]}"; do
    if [ -f "$conf" ]; then
        cp "$conf" "${BACKUP_DIR}/$(basename $conf).${DATE}"
    fi
done

# Keep only last 30 days
find "$BACKUP_DIR" -type f -mtime +30 -delete

Script Writing Best Practices

Good habits prevent pitfalls:

  1. Start with set -euo pipefail: exit on error, undefined variable errors
  2. Quote variables: "$VAR" prevents space and glob issues
  3. Log critical operations: echo "$(date): xxx" for debugging
  4. Lock mechanism: flock or PID file to prevent duplicate execution
  5. Standard exit codes: 0 for success, non-zero for failure
#!/bin/bash
set -euo pipefail

LOCK_FILE="/tmp/$(basename $0).lock"
exec 200>"$LOCK_FILE"
flock -n 200 || { echo "Script already running"; exit 1; }

# Business logic...

Summary

Shell scripts are simple, but good writing habits can significantly reduce operations incidents. Core principle: automate what can be automated, add checks for what can’t.