Kubernetes

Overview 1 AM. You get an alert: GitLab CI pipeline queue has 47 pending jobs. The dev chat explodes—“code pushed 40 minutes ago, still not running"“is the Runner down?““just add more machines!” You check the console: all 3 Runners are at capacity, each running 10 concurrent jobs, 30 slots fully occupied. Add machines? Docker Machine executor needs 3 minutes to spin up a new EC2, then another 2 minutes to become Ready....

Overview “Fully managed control plane” — this phrase makes many teams think migrating to an ACK managed cluster means they can kick back and relax. In reality, managed only takes care of etcd, kube-apiserver, kube-controller-manager, and kube-scheduler. Data plane problems? Still all yours. I led a migration of 120+ microservices from a self-hosted K8s cluster to ACK Managed Cluster Pro. The migration took 3 months with zero-downtime cutover — but the first month after switching, 3 AM alerts never stopped....

Overview You manage a K8s cluster where the API Server has --anonymous-auth=true, etcd certificates are world-readable at 644, and the kubelet --read-only-port=10255 is still open. Each of these looks like a “minor issue” on its own, but an attacker who gets one foothold can pivot through the entire cluster. This isn’t hypothetical — the CNCF 2025 annual survey showed that over 90% of production K8s clusters have at least one CIS Benchmark-level configuration deficiency....

Overview Production blows up, you open Kibana to search logs, and discover the critical Pod was OOM-killed 3 minutes ago—the old logs vanished with the container. This is extremely common in K8s environments: Pods are ephemeral, and logs can’t disappear with them. K8s log collection is fundamentally different from traditional VM environments. In a traditional setup, logs sit quietly in /var/log/ and you just SSH in to read them. In K8s, Pods can be scheduled to any node at any time, destroyed and recreated at will, with logs scattered across the entire cluster....

Overview You’re an ops engineer at a smart manufacturing company. The factory floor has 200 edge gateways, each running data collection and real-time quality inspection services. Previously deployed with bare Docker, every update meant writing scripts to SSH into each machine, pull images, and restart containers. Running through 200 machines took half an hour, with a few always failing due to network jitter. You think: isn’t this exactly what Kubernetes solves?...

Overview Let’s answer the most fundamental question first: what does a Service Mesh do? In one sentence: it handles the messy communication stuff between microservices for you. In a microservices architecture, when service A calls service B, what looks like a simple HTTP request actually involves a bunch of concerns: what if it times out? How many retries? Should we circuit break? How do we do canary traffic? Certificate management? Distributed tracing?...

Overview Kubernetes Security Hardening: RBAC, NetworkPolicy, and Pod Security Policies is an essential skill in SRE operations. In production environments, mastering these techniques can significantly improve system stability and operational efficiency. Why Kubernetes Security Hardening Matters As systems grow in scale and complexity, traditional operations approaches struggle to meet the demands of modern distributed systems. Kubernetes Security Hardening helps operations teams: Rapid Problem Resolution: Systematic tools and methods reduce troubleshooting time Improved System Visibility: Establish comprehensive monitoring and observability Proactive Fault Prevention: Identify and fix potential risks before they cause outages Resource Optimization: Allocate and schedule resources efficiently Core Concepts and Principles Basic Concepts The core of Kubernetes Security Hardening lies in establishing standardized processes and automated toolchains....

The Role of Change Management in SRE Google SRE identified an iron rule: approximately 70% of production incidents are directly caused by changes. Whether it’s code deployment, configuration modification, infrastructure adjustment, or dependency upgrades, every change injects uncertainty into the system. Change management is therefore not bureaucratic red tape — it’s the first line of defense in SRE reliability engineering. The core objectives of change management can be summarized in three points:...

Why Helm? Managing K8s applications with bare kubectl apply -f works at small scale, but as environments multiply (dev/staging/prod) and services grow, problems quickly surface: Hardcoded configuration: Each environment has its own YAML file with image tags, replica counts, and resource limits hardcoded. Changing a single value means editing ten files. No version management: Upgrades and rollbacks rely on manual records—no way to know what version was last deployed. No reusability: Deploying Redis and MySQL requires two completely different YAML sets with no way to templatize....

Reliability Engineering: More Than Just “Not Breaking” The goal of reliability engineering is not to pursue zero failures — that’s neither realistic nor economical. The real goal is: given that failures are inevitable, make the system capable of fast detection, automatic recovery, and continuous learning. Google SRE proposes a core formula: MTTR << MTBF / (MTBF + MTTR) × (1 - SLO) This formula reveals a key insight: when the time between failures (MTBF) is much greater than the repair time (MTTR), system availability naturally approaches the SLO target....