Posts

Overview In the era of cloud-native and microservices, a single request may traverse dozens of service nodes. Traditional monitoring scatters Metrics, Logs, and Traces across different systems — Prometheus for metrics, ELK for log search, Jaeger for tracing — with no unified way to correlate them. When an online incident occurs, you need to switch between three systems, manually piecing together correlated information, which is highly inefficient. OpenTelemetry (OTel) is the CNCF-led unified observability standard, aiming to use a single SDK/API to collect all three signals (Metrics, Logs, Traces), process them through a unified Collector, and send them to any backend....

Why Go Is Ideal for Ops Tools Ops CLI tools demand high deployment convenience and execution efficiency — areas where Go has natural advantages: Advantage Description Compared to Python Single binary Compiles to a standalone executable with no runtime dependencies Requires Python environment + dependencies Cross-platform GOOS/GOARCH cross-compilation — write once, run anywhere Requires virtualenv management Startup speed Millisecond-level cold start Interpreter overhead Concurrency model Lightweight goroutines Requires threading/asyncio Memory footprint Low memory usage as a static binary Interpreter overhead Mature ecosystem Standard library covers networking/files/crypto Relies on third-party libraries Kubernetes, Docker, Terraform, Prometheus — the core cloud-native tools are all written in Go....

paramiko: SSH Batch Management Fundamentals paramiko is a Python implementation of the SSHv2 protocol and the foundational building block for ops automation. When you need fine-grained control over SSH connections or need to handle non-standard scenarios, paramiko offers maximum flexibility. Basic Connection and Command Execution import paramiko import time def ssh_exec(host, port, username, password, command): """Basic SSH command execution""" client = paramiko.SSHClient() # Auto-add host keys (use known_hosts in production) client....

Overview The core philosophy of SRE is: manage reliability through engineering methods. The most important tools are SLI, SLO, and Error Budgets. SLI: Service Level Indicator SLI is a quantitative metric for system reliability. Common SLIs include: Availability: successful requests / total requests Latency: P99 response time < 200ms Throughput: QPS > 10000 Correctness: data consistency check pass rate Key principle for choosing SLIs: start from the user’s perspective. Users don’t care about your CPU usage — they care whether requests succeed and are fast enough....

Overview “Hope for the best, prepare for the worst.” In Kubernetes production environments, disaster recovery is the last line of defense. Whether it’s etcd corruption, accidental deletion, node failure, or even entire cluster loss, having a solid backup and recovery strategy is critical. This article systematically covers K8s disaster recovery—from etcd backup and restore, Velero full-cluster backup, PV data backup, to cross-cluster recovery and disaster recovery architecture design. Based on Kubernetes v1....

Overview Many teams treat SRE as just “operations with a new name” — hire a few people who can write scripts, change their titles, and call it a transformation. This mindset ignores a fundamental truth: SRE is an engineering methodology, not a toolchain. When Google created the SRE function in 2003, the core idea was “treat operations problems with software engineering methods,” which fundamentally changed the positioning, workflow, and culture of operations....

Overview Traditional monitoring is “passive” — it waits for users to access the system, triggering system behavior, then collects metrics and logs. This approach has a fundamental flaw: when monitoring detects a problem, users have already been impacted. If your homepage takes 10 seconds to load, your monitoring alert may not trigger for 5 minutes — by which time thousands of users have experienced poor performance. Synthetic Monitoring is a “proactive” monitoring approach — it simulates real user behavior, regularly accessing critical paths to discover and fix issues before users perceive them....

Overview As the de facto standard for cloud-native monitoring, Prometheus has become the default choice for microservice and Kubernetes monitoring. However, as business scale grows, Prometheus’s local storage architecture increasingly exposes significant bottlenecks: limited single-machine storage capacity (default 15-day retention), no native horizontal scaling, memory spikes under high-cardinality scenarios, and difficulty querying historical data. Many teams find themselves facing the “triple dilemma” of disk IO pressure, storage cost inflation, and query latency growth once time series exceed the million-level mark....

Overview In automated operations, Ansible Playbooks frequently handle sensitive information such as database passwords, API keys, and SSH private keys. If this data is stored in plaintext in code repositories, a single repository leak exposes all credentials. Ansible Vault, Ansible’s built-in encryption tool, protects sensitive data using the AES-256 symmetric encryption algorithm, ensuring that only authorized users can access it. This article will systematically cover Ansible Vault usage, password management strategies, and CI/CD pipeline integration—from basic concepts to production practices....

Overview Performance issues are among the most common scenarios every SRE encounters: users report “it’s slow,” alerts say “P99 latency exceeds threshold,” monitoring shows “CPU is almost maxed.” But many teams handle performance issues with a “tune wherever it’s high” approach — add machines when CPU is high, add indexes when SQL is slow, add cache when latency is high. This symptomatic treatment may work short-term, but over time it makes the system increasingly complex, costs keep rising, and problems become harder to troubleshoot....