Automation

Overview 2 AM, alert fires. Investigation reveals a server running an outdated OpenSSL version with known vulnerabilities. Security team asks: how many more machines are in the same state? You don’t have an answer. That’s the cost of configuring servers manually. Each machine gets its own apt install, producing environments that drift apart. Someone forgot a patch, someone installed extra packages, someone changed a kernel parameter without documenting it. When things break, you realize—you have no idea how many “configuration drift” variants are running in production....

Overview 1:47 AM. Phone buzzed. Alert channel exploded. A newly launched recommendation service at a ride-hailing project went OOMKilled on all 3 Pods less than 2 hours after launch. Memory spiked from 800MB to the 2GB limit, the kernel killed the process, Kubernetes restarted the Pod, and it got killed again. Loop. Users were routed to the fallback path. P99 latency shot from 80ms to 2000ms. The postmortem found the root cause....

Overview 1:47 AM. Phone buzzes. The alert group explodes with 200+ messages. Order success rate drops from 99.8% to 92%. The on-call engineer restarts the service, rolls back the version, and fights for 40 minutes to restore service. Next day’s postmortem: root cause was a one-line database connection pool config change pushed during the afternoon release. This change went through the full approval process — developer submission, test verification, manager sign-off, SRE review....

Overview Code Review Automation: Lint, CI, and Intelligent Checking Toolchains is an essential skill in SRE operations. In production environments, mastering these techniques can significantly improve system stability and operational efficiency. Why Code Review Automation Matters As systems grow in scale and complexity, traditional operations approaches struggle to meet the demands of modern distributed systems. Code Review Automation helps operations teams: Rapid Problem Resolution: Systematic tools and methods reduce troubleshooting time Improved System Visibility: Establish comprehensive monitoring and observability Proactive Fault Prevention: Identify and fix potential risks before they cause outages Resource Optimization: Allocate and schedule resources efficiently Core Concepts and Principles Basic Concepts The core of Code Review Automation lies in establishing standardized processes and automated toolchains....

paramiko: SSH Batch Management Fundamentals paramiko is a Python implementation of the SSHv2 protocol and the foundational building block for ops automation. When you need fine-grained control over SSH connections or need to handle non-standard scenarios, paramiko offers maximum flexibility. Basic Connection and Command Execution import paramiko import time def ssh_exec(host, port, username, password, command): """Basic SSH command execution""" client = paramiko.SSHClient() # Auto-add host keys (use known_hosts in production) client....

Overview In automated operations, Ansible Playbooks frequently handle sensitive information such as database passwords, API keys, and SSH private keys. If this data is stored in plaintext in code repositories, a single repository leak exposes all credentials. Ansible Vault, Ansible’s built-in encryption tool, protects sensitive data using the AES-256 symmetric encryption algorithm, ensuring that only authorized users can access it. This article will systematically cover Ansible Vault usage, password management strategies, and CI/CD pipeline integration—from basic concepts to production practices....

Overview The Google SRE Book contains a frequently cited principle: SRE teams should spend no more than 50% of their total work time on toil. This principle seems simple, but in practice, many SRE teams’ toil ratio far exceeds 50% — some even reach 80% or more. Why should SRE take “toil” so seriously? Because toil is the invisible killer of reliability: Toil consumes enormous amounts of time, leaving engineers no energy for work that genuinely improves reliability Toil typically involves manual operations that are error-prone, actually introducing new incidents Toil leads to burnout and attrition of talented engineers Toil doesn’t scale — when the system grows 10x, toil grows 10x too This article systematically covers how to manage operational work across toil definition and identification, source analysis, automation elimination paths, the 50% cap principle, measurement and tracking methods, and team practices....

Overview 3 AM. Your phone buzzes. You get out of bed, open your laptop, SSH in, and find a service with CPU spiking. Kill the process, restart the service, 12 minutes done — but you’re wide awake now. 4:30 AM, another alert: disk usage exceeds 85%. Get up again, du -sh to locate, delete old logs, 15 minutes. This is the daily reality for countless operations engineers. Monitoring is set up, alerts are configured, scripts are written — but the final step still relies on humans....

Introduction Shell scripts are the most common automation tool for operations engineers. This post records several practical patterns used in the field. Log Rotation Cleanup #!/bin/bash # clean_logs.sh - Log cleanup script LOG_DIR="/var/log/app" KEEP_DAYS=30 find "$LOG_DIR" -name "*.log" -type f -mtime +${KEEP_DAYS} -exec gzip {} \; find "$LOG_DIR" -name "*.gz" -type f -mtime +${KEEP_DAYS} -delete echo "$(date): Log cleanup complete, kept ${KEEP_DAYS} days" Run daily via crontab: 0 2 * * * /opt/scripts/clean_logs....