Archives

All posts archive

2026  73

October  8

Database Connection Pool Blew Up Mid-Traffic-Shift: 6 Production-Grade Decisions for Blue-Green and Canary Deployment with Traffic Governance

October 17, 2026 · 23 mins · 4898 words · Xu Baojin

One Namespace Ate 80% of Cluster Memory: 4 Production-Grade Decisions for K8s ResourceQuota Multi-Tenant Governance

October 15, 2026 · 16 mins · 3339 words · Xu Baojin

5000 Alerts Nobody Reads: 6 Architecture Decisions from Fragmented to Unified Observability

October 13, 2026 · 21 mins · 4396 words · Xu Baojin

From 60% Toil to 20%: A 5-Level SRE Team Maturity Model and 6 Engineering Decisions for Quantitative Self-Assessment

October 10, 2026 · 23 mins · 4780 words · Xu Baojin

Pods OOM Killed at 3 AM: From working_set Blind Spots to a Memory Warning System in 5 Production Decisions

October 9, 2026 · 21 mins · 4427 words · Xu Baojin

When Your Self-Hosted Terraform Backend Cracks at 30 Engineers: 6 Engineering Decisions and a TCO Ledger from Moving to HCP Terraform

October 7, 2026 · 40 mins · 8382 words · ABao

Rollback Made It Worse: 5-Layer CI/CD Auto-Rollback Architecture and 4 Production Pitfalls

October 3, 2026 · 22 mins · 4596 words · Xu Baojin

Zero-Failure K8s Migration of 120 Microservices: 6 Pitfalls Nobody Talks About from a 3-Month Real Migration

October 1, 2026 · 27 mins · 5746 words · Xu Baojin

September  15

Stop Configuring Servers by Hand: A Complete Guide to Packer Golden Image Pipelines from Build to Security Scanning

September 29, 2026 · 25 mins · 5210 words · Xu Baojin

New Service OOMed 2 Hours After Launch: Replacing Manual Approval with a 7-Dimension Production Readiness Gate

September 27, 2026 · 22 mins · 4618 words · ABao

Cache Hit Rate from 58% to 94%: Building a CDN Monitoring System and 5 Production-Grade Troubleshooting Decisions

September 24, 2026 · 20 mins · 4104 words · Xu Baojin

VPC Peering Became Your Bottleneck: Cloud-Native Network Architecture Selection and 5 Production-Grade Decisions

September 22, 2026 · 25 mins · 5314 words · Xu Baojin

Pushing the Same Image 3 Times to 3 Environments? CI/CD Artifact Promotion Pipeline Layered Design and 4 Production Decisions

September 19, 2026 · 24 mins · 5110 words · Xu Baojin

3,000 Lines of HCL Nobody Dares to Touch? 5 Production Decisions and Migration Pitfalls of Redoing IaC in a Real Programming Language with Pulumi

September 17, 2026 · 16 mins · 3398 words · Xu Baojin

Promised 99.9%, Got 0.7 Yuan Back: SLA Clause Breakdown and 6 Engineering Decisions for Internal Design

September 15, 2026 · 33 mins · 6998 words · Xu Baojin

14-Day Change Freeze Still Blew Up P0: 5 Engineering Decisions from Manual Freeze to Dynamic Change Windows

September 12, 2026 · 30 mins · 6243 words · ABao

2GB of Zombie Shared Memory Nobody Reclaimed: Linux IPC Mechanism Selection and Production Troubleshooting Journal

September 10, 2026 · 22 mins · 4513 words · Xu Baojin

Error Budget Depleted, PM Still Pushing Releases: Three-Tier Gates Turning Reliability into Hard Product Decision Constraints

September 10, 2026 · 22 mins · 4527 words · Xu Baojin

Pods Stuck Pending After IP Pool Exhaustion: 5 Production Decisions for TKE Network Selection and Node Pool Governance

September 8, 2026 · 17 mins · 3587 words · ABao

Don’t Manage Your Infrastructure Twice: The Terraform + Ansible Collaboration Boundary and 5 Production-Grade Decisions

September 5, 2026 · 22 mins · 4651 words · ABao

Pods Stuck for 6 Minutes After Network Partition Injection: Blast Radius Control and 5 Production-Grade Decisions in Fault Injection Testing

September 3, 2026 · 24 mins · 4936 words · ABao

Pushed an Image and Forgot About It? Harbor Artifact Repository: Permission Layering, Vulnerability Scanning, and 200GB Storage Governance

September 3, 2026 · 15 mins · 3153 words · ABao

API Gateway Monitoring in Practice: Metrics Collection and Alert Design from Kong to APISIX

September 1, 2026 · 18 mins · 3623 words · ABao

August  13

Don’t Wait for the Crash to Check Monitoring: A Practical Methodology for Linux Performance Baselines and Anomaly Detection

August 30, 2026 · 23 mins · 4689 words · ABao

From “@ Ops in Chat” to Self-Service: 6-Layer Architecture and 4 Landing Phases for Platformizing 20+ Operations

August 27, 2026 · 27 mins · 5653 words · ABao

The 3 AM Phone Rings Again: 7 Engineering Decisions for On-Call Rotation Design and Fatigue Management

August 25, 2026 · 24 mins · 4917 words · ABao

Don’t Fall for the No-Ops Myth: 5 Production-Level Governance Decisions for Serverless Cold Starts, Cost Explosions, and Observability Blind Spots

August 22, 2026 · 22 mins · 4662 words · ABao

Cutting 80% of Annual Downtime: 6 Engineering Decisions and Hard-Learned Lessons Going from 99.5% to 99.9% Availability

August 20, 2026 · 22 mins · 4628 words · ABao

False Positive Rate from 70% to 5%: A Four-Layer Security Gate Design and False Positive Management in CI/CD

August 18, 2026 · 25 mins · 5153 words · ABao

Don’t Turn Terraform Modules Into Black Boxes: IaC Layered Design Decisions and 6 Production Anti-Patterns Broken Down

August 15, 2026 · 17 mins · 3529 words · ABao

Starting from Docker Machine Deprecation: 7 Production Decisions for GitLab CI Runner Elastic Architecture and Cache Governance

August 13, 2026 · 17 mins · 3581 words · ABao

Disaster Recovery Is Not Backup: Cross-Datacenter RPO<5min, RTO<30min Architecture Decisions and Lessons Learned

August 11, 2026 · 26 mins · 5335 words · ABao

Managed Clusters Aren’t Set-and-Forget: 5 Production Pitfalls After Migrating from Self-Hosted K8s to ACK

August 8, 2026 · 23 mins · 4859 words · ABao

3 Days of Approval, Still Crashed: Replacing Manual CAB with Error Budget Gates Cut Change-Related Incidents by 60%

August 6, 2026 · 23 mins · 4818 words · ABao

Don’t Let Jenkins Dictate Your Release Cadence: Architecture Decisions and Lessons from Building a Go DAG Scheduler Engine

August 4, 2026 · 26 mins · 5482 words · ABao

Deploying IaC Without Tests? A Four-Layer Validation Pipeline That Cuts 3AM Alerts by 80%

August 1, 2026 · 20 mins · 4051 words · ABao

July  30

Reliability Measurement Is Not About Piling Metrics: Turning ‘Is the System Stable?’ into Decision-Driven Numbers with a Four-Layer Model

July 30, 2026 · 24 mins · 5062 words · ABao

MTTR from 40 Minutes to 8: Cutting Fault Localization Time by 5x with a Four-Layer Defense

July 28, 2026 · 22 mins · 4552 words · ABao

Don’t Get Fooled by Multi-Cloud: Architecture Decisions and Lessons from Vendor Lock-in to Cross-Cloud Disaster Recovery

July 25, 2026 · 7 mins · 1475 words · ABaoPlus

Lost Your State File? Terraform State Disaster Recovery and Production-Grade Backend Architecture

July 25, 2026 · 16 mins · 3384 words · ABao

CI/CD Deployment Speed Optimization in Practice: From 90 Minutes to 5 Minutes

July 25, 2026 · 21 mins · 4408 words · ABao

Microservice Rate Limiting, Circuit Breaking, and Degradation: A Practical Guide from Avalanche Prevention to Fine-Grained Traffic Governance

July 24, 2026 · 26 mins · 5536 words · ABao

Disk Space and inode Management: A Complete Troubleshooting Guide from df to lsof

July 23, 2026 · 19 mins · 3914 words · ABao

System Resilience Engineering: From Reactive Firefighting to Proactive Defense

July 23, 2026 · 20 mins · 4219 words · ABao

K8s Security Scanning and CIS Benchmark Compliance: A Complete Hardening Guide from kube-bench to Production

July 22, 2026 · 20 mins · 4201 words · ABao

Postmortem Action Items Tracking: Engineering a Closed-Loop Process from Checklist to Completion

July 21, 2026 · 24 mins · 5022 words · ABao

Container Image Signing and Verification: Using cosign to Guard the Last Gate of Supply Chain Security

July 20, 2026 · 15 mins · 3045 words · ABao

Frontend Performance Monitoring in Practice: A Full-Stack Guide from Core Web Vitals to RUM

July 20, 2026 · 17 mins · 3584 words · ABao

Linux Time Synchronization and NTP: chrony Configuration, Troubleshooting, and Production Practices

July 19, 2026 · 17 mins · 3555 words · ABao

Message Queue Monitoring in Practice: Kafka Lag, RabbitMQ Backlog, and Consumer Latency Alerting

July 19, 2026 · 16 mins · 3363 words · ABao

K8s Log Collection Strategy: Choosing Between DaemonSet, Sidecar, and Agentless Patterns

July 18, 2026 · 19 mins · 4001 words · ABao

DNS Resolution and Troubleshooting: A Practical Guide from resolv.conf to systemd-resolved

July 18, 2026 · 17 mins · 3531 words · ABao

Linux Kernel Crashes and kdump: Installing a Flight Recorder for Your Servers

July 17, 2026 · 21 mins · 4328 words · ABao

Building a Database Monitoring System: From Exporters to Alert Rules in Practice

July 17, 2026 · 21 mins · 4265 words · ABao

SRE Incident Preparedness and Drills: From Paper Plans to Muscle Memory

July 16, 2026 · 20 mins · 4112 words · ABao

K3s Edge Computing in Practice: Lightweight Kubernetes Deployment and Operations for Resource-Constrained Environments

July 16, 2026 · 20 mins · 4085 words · ABao

APM Tool Selection: A Practical Guide from Open-Source to Commercial Solutions

July 15, 2026 · 21 mins · 4277 words · ABao

System Security Auditing: auditd Rule Configuration and Log Analysis in Practice

July 15, 2026 · 23 mins · 4737 words · ABao

Building an Automated Inspection Platform from Scratch: Plugin Architecture Design and Go Implementation Guide

July 14, 2026 · 24 mins · 4992 words · ABao

Jenkins Freestyle vs Pipeline: A Practical Comparison and Configuration Guide

July 14, 2026 · 19 mins · 3929 words · ABao

Linux Network Packet Capture and Protocol Analysis: A Practical Troubleshooting Guide from tcpdump to Wireshark

July 14, 2026 · 20 mins · 4177 words · ABao

Building an SRE Team: From Hiring to Organizational Capability Model

July 13, 2026 · 14 mins · 2883 words · ABao

Linux CPU Isolation and NUMA Tuning: A Practical Guide to Exclusive Compute for Critical Workloads

July 13, 2026 · 18 mins · 3724 words · ABao

Monitoring Data Governance: From Metrics Explosion to Precision Observability

July 12, 2026 · 14 mins · 2892 words · ABao

Istio Service Mesh Getting Started: A Practical Guide from Sidecar to Ambient

July 12, 2026 · 12 mins · 2523 words · ABao

Kubernetes Security Hardening: RBAC, NetworkPolicy, and Pod Security Policies

July 4, 2026 · 3 mins · 571 words · ABao

June  3

Linux Namespaces and cgroups: The Foundation of Container Technology

June 28, 2026 · 3 mins · 576 words · ABao

LLM-Assisted Troubleshooting: From Log Analysis to Root Cause Identification

June 13, 2026 · 3 mins · 566 words · ABao

AIOps Anomaly Detection: From Static Thresholds to Intelligent Alerting

June 5, 2026 · 3 mins · 571 words · ABao

May  1

Vulnerability Scanning in CI/CD: From Dependency Checks to Image Security

May 15, 2026 · 3 mins · 577 words · ABao

April  1

Code Review Automation: Lint, CI, and Intelligent Checking Toolchains

April 3, 2026 · 3 mins · 571 words · ABao

January  2

Git Workflow and Team Collaboration: Branching Strategies and Code Review

January 17, 2026 · 3 mins · 582 words · ABao

Data-Driven SRE Decisions: From Metrics to Actions

January 2, 2026 · 3 mins · 569 words · ABao

2025  21

December  3

SRE Perspective on FinOps: Cloud Cost Visibility and Optimization Strategies

December 27, 2025 · 3 mins · 577 words · ABao

Platform Engineering Introduction: Designing and Implementing Internal Developer Platforms

December 22, 2025 · 3 mins · 571 words · ABao

Monitoring as Code: Managing Alert Rules with Terraform and YAML

December 19, 2025 · 3 mins · 572 words · ABao

November  1

SRE Documentation Practice: Runbooks, Architecture Diagrams, and Knowledge Base

November 9, 2025 · 3 mins · 571 words · ABao

October  4

Cloud Native Security: Container Security, Image Scanning, and Runtime Protection

October 26, 2025 · 3 mins · 572 words · ABao

AWS Operations Basics: EC2, S3, and IAM Permission Management

October 21, 2025 · 3 mins · 571 words · ABao

PostgreSQL High Availability: Patroni and etcd Cluster Deployment

October 16, 2025 · 3 mins · 570 words · ABao

MySQL Performance Optimization: Slow Query Analysis and Index Tuning

October 13, 2025 · 3 mins · 571 words · ABao

September  4

Redis Operations: Persistence, High Availability, and Performance Monitoring

September 15, 2025 · 3 mins · 565 words · ABao

TLS/SSL Certificate Management: From Application to Auto-Renewal

September 14, 2025 · 3 mins · 569 words · ABao

Nginx Performance Tuning: From Configuration to Kernel Parameters

September 12, 2025 · 3 mins · 570 words · ABao

Linux System Call Tracing: strace and ltrace Debugging

September 11, 2025 · 3 mins · 575 words · ABao

August  1

Reliability Design for Multi-Region Active-Active Architecture

August 29, 2025 · 17 mins · 3513 words · ABao

June  1

SRE and Development Team Collaboration Models: Engineering Practices for Breaking Down Walls

June 24, 2025 · 20 mins · 4224 words · ABao

April  1

eBPF Programming Guide: From Principles to Production-Grade Tooling

April 1, 2025 · 18 mins · 3632 words · ABao

February  2

Change Management: Canary Release and Rollback Strategies

February 11, 2025 · 9 mins · 1902 words · ABao

Blackbox Exporter: External Probing and Uptime Monitoring

February 4, 2025 · 12 mins · 2429 words · ABao

January  4

Helm Chart Writing and Private Repository Management

January 16, 2025 · 13 mins · 2652 words · ABao

Linux Firewall: iptables/nftables from Beginner to Expert

January 10, 2025 · 16 mins · 3292 words · ABao

SRE Reliability Engineering: From Theory to Practice

January 9, 2025 · 12 mins · 2395 words · ABao

Kubernetes Cluster Upgrade Strategy: From Planning to Zero-Downtime Execution

January 3, 2025 · 19 mins · 3962 words · ABao

2024  72

December  3

Prometheus Automated Inspection Script Collection

December 19, 2024 · 22 mins · 4590 words · ABao

Chaos Engineering: Proactively Discovering System Weaknesses

December 17, 2024 · 16 mins · 3405 words · ABao

Service Dependency Mapping and Failure Domain Analysis: From Topology Discovery to Blast Radius Control

December 16, 2024 · 25 mins · 5210 words · ABao

November  7

Postmortem Culture: Engineering Practices for Learning from Failures

November 26, 2024 · 20 mins · 4055 words · ABao

Kubernetes Cost Optimization in Practice: From Resource Governance to FinOps

November 22, 2024 · 23 mins · 4737 words · ABao

Kubernetes Multi-Cluster Management in Practice

November 20, 2024 · 10 mins · 2124 words · ABao

Thanos Deployment and Practices: Prometheus Long-Term Storage and Global Query

November 19, 2024 · 15 mins · 3055 words · ABao

Linux Performance Analysis Powerhouse: BPF and the bcc Toolset

November 13, 2024 · 9 mins · 1860 words · ABao

OpenTelemetry: Unified Observability Standard

November 11, 2024 · 13 mins · 2633 words · ABao

Building Ops CLI Tools with Go

November 6, 2024 · 10 mins · 1964 words · ABao

October  5

Python Operations Automation: From paramiko to Ansible

October 31, 2024 · 6 mins · 1247 words · ABao

SRE Core Concepts: SLI, SLO and Error Budgets

October 28, 2024 · 2 mins · 260 words · ABao

Kubernetes Disaster Recovery: From etcd Backup to Velero

October 21, 2024 · 14 mins · 2846 words · ABao

The Fundamental Differences Between SRE and Traditional Operations

October 11, 2024 · 10 mins · 2082 words · ABao

Synthetic Monitoring: Proactively Safeguarding User Experience

October 9, 2024 · 16 mins · 3326 words · ABao

September  2

VictoriaMetrics Deployment and Practices: A High-Performance Alternative for Prometheus Long-Term Storage

September 13, 2024 · 19 mins · 3895 words · ABao

Ansible Vault Password Management: A Practical Guide to Encrypting Sensitive Data

September 5, 2024 · 12 mins · 2521 words · ABao

August  4

Performance Engineering: A System Optimization Methodology from the SRE Perspective

August 29, 2024 · 18 mins · 3738 words · ABao

Incident Management and On-Call Mechanism Design

August 23, 2024 · 13 mins · 2559 words · ABao

Kubernetes Autoscaling: HPA/VPA/CA Deep Dive

August 19, 2024 · 13 mins · 2692 words · ABao

Alerting Strategy Design: From Noise to Signal

August 19, 2024 · 14 mins · 2928 words · ABao

July  6

Jenkins Pipeline as Code in Practice

July 29, 2024 · 17 mins · 3551 words · ABao

Eliminating Toil: SRE’s Approach to Managing Operational Work

July 24, 2024 · 17 mins · 3598 words · ABao

cgroup v2 Complete Guide: From Architecture Principles to Production Practices

July 23, 2024 · 23 mins · 4791 words · ABao

Kubernetes Operator Development Guide

July 23, 2024 · 17 mins · 3507 words · ABao

Prometheus High Availability and Federation

July 23, 2024 · 13 mins · 2659 words · ABao

GitOps Workflow: ArgoCD in Practice

July 22, 2024 · 8 mins · 1690 words · ABao

June  5

Capacity Planning and Elastic Scaling in Practice

June 27, 2024 · 10 mins · 1970 words · ABao

Container Runtime: containerd and CRI Deep Dive

June 14, 2024 · 12 mins · 2385 words · ABao

Runbook Writing Guide: Making Operational Knowledge Reproducible

June 11, 2024 · 18 mins · 3664 words · ABao

Prometheus Service Discovery Mechanisms Explained

June 6, 2024 · 13 mins · 2600 words · ABao

Python Log Analysis Scripts in Practice

June 5, 2024 · 18 mins · 3813 words · ABao

May  11

Linux Process Scheduler: CFS Principles and Tuning

May 29, 2024 · 10 mins · 2121 words · ABao

Alert Automation and Remediation: From Alert Storms to Self-Healing Systems

May 28, 2024 · 31 mins · 6470 words · ABao

Incident Response Framework: SEV Classification and Escalation Flow

May 23, 2024 · 20 mins · 4176 words · ABao

Distributed Tracing: Jaeger Implementation in Practice

May 16, 2024 · 9 mins · 1742 words · ABao

Linux Memory Management Mechanisms and Tuning in Practice

May 9, 2024 · 13 mins · 2564 words · ABao

Elasticsearch + Kibana Log Analysis Platform

May 9, 2024 · 14 mins · 2896 words · ABao

Kubernetes Pod Troubleshooting Cheatsheet

May 8, 2024 · 2 mins · 280 words · ABao

Linux Network Stack Tuning: From Kernel to Application

May 7, 2024 · 12 mins · 2493 words · ABao

Linux Boot Process Explained: From Firmware to Userspace

May 3, 2024 · 18 mins · 3668 words · ABao

Git Hooks Automation: From Code Quality to Deployment

May 3, 2024 · 18 mins · 3628 words · ABao

Kubernetes RBAC and Security Contexts

May 2, 2024 · 12 mins · 2449 words · ABao

April  10

Kubernetes Scheduler Principles and Tuning

April 29, 2024 · 12 mins · 2478 words · ABao

Error Budget Consumption Strategies and Action Guidelines

April 26, 2024 · 17 mins · 3556 words · ABao

Linux Package Management: apt/yum/dnf and Package Building

April 26, 2024 · 19 mins · 3900 words · ABao

Getting Started with Terraform Infrastructure as Code

April 24, 2024 · 12 mins · 2429 words · ABao

SLO Design in Practice: From Business Goals to Technical Metrics

April 24, 2024 · 18 mins · 3637 words · ABao

Commercial vs Self-Hosted Monitoring: Datadog vs Open-Source Solutions

April 17, 2024 · 14 mins · 2978 words · ABao

Grafana Dashboard Best Practices

April 17, 2024 · 8 mins · 1646 words · ABao

K8s Networking Model: CNI and Service Networking

April 12, 2024 · 7 mins · 1389 words · ABao

Ansible Playbook Best Practices: From Basics to Production

April 5, 2024 · 15 mins · 3107 words · ABao

Prometheus PromQL: From Beginner to Expert

April 4, 2024 · 7 mins · 1294 words · ABao

March  7

Disk I/O Performance Diagnosis and Optimization

March 27, 2024 · 8 mins · 1627 words · ABao

Kubernetes Ingress Controller Selection and Configuration

March 27, 2024 · 11 mins · 2161 words · ABao

Makefile Build Automation: More Than Just Compilation

March 26, 2024 · 14 mins · 2964 words · ABao

Scheduled Task Management: cron vs systemd timer

March 18, 2024 · 21 mins · 4276 words · ABao

Docker Image Optimization: From 1GB to 50MB

March 14, 2024 · 8 mins · 1618 words · ABao

kubectl Productivity Guide: Plugins and Aliases

March 12, 2024 · 17 mins · 3535 words · ABao

CI/CD Pipeline Design: GitHub Actions in Practice

March 7, 2024 · 12 mins · 2388 words · ABao

February  9

Log Monitoring System: Loki + Promtail Deployment

February 29, 2024 · 10 mins · 2124 words · ABao

Alertmanager Alert Routing and Silencing Strategies

February 27, 2024 · 9 mins · 1867 words · ABao

Quick Setup: Prometheus Monitoring Stack

February 23, 2024 · 1 min · 197 words · ABao

Linux Log Management: journald and Log Rotation

February 14, 2024 · 14 mins · 2925 words · ABao

systemd Service Management Deep Guide

February 13, 2024 · 9 mins · 1787 words · ABao

Docker Compose Production Environment Guide

February 7, 2024 · 13 mins · 2730 words · ABao

Container Persistent Storage Solution Selection

February 6, 2024 · 7 mins · 1315 words · ABao

Zabbix vs Prometheus: Monitoring System Selection Guide

February 5, 2024 · 15 mins · 3041 words · ABao

Kubernetes Resource Management: Requests and Limits

February 2, 2024 · 9 mins · 1862 words · ABao

January  3

Linux Performance Profiling Toolkit: From top to perf

January 19, 2024 · 18 mins · 3658 words · ABao

Practical Shell Script Automation Tips

January 11, 2024 · 2 mins · 369 words · ABao

Linux Filesystem Selection and Performance Optimization

January 1, 2024 · 12 mins · 2507 words · ABao

2023  4

December  4

Production-Grade Bash Scripting Guide

December 27, 2023 · 22 mins · 4513 words · ABao

Linux Kernel Parameter Tuning Practical Handbook

December 21, 2023 · 13 mins · 2727 words · ABao

Linux Security Hardening Checklist: From Minimal Installation to Compliance

December 18, 2023 · 12 mins · 2482 words · ABao

SSH Security Best Practices: From Authentication to Tunneling

December 12, 2023 · 15 mins · 3005 words · ABao