Kubernetes Pod Troubleshooting Cheatsheet

Troubleshooting Path kubectl get pods → check status kubectl describe pod → check Events kubectl logs → check logs Common Pod States State Meaning Common Cause Pending Not scheduled Insufficient resources, scheduling constraints CrashLoopBackOff Crash loop App error, config issue ImagePullBackOff Image pull failed Image not found, auth failure OOMKilled Out of memory Memory limit too low CrashLoopBackOff Most common issue. Troubleshooting steps: # Check previous crash logs kubectl logs <pod> --previous # Check exit code kubectl get pod <pod> -o jsonpath='{....

May 8, 2024 · 2 mins · 280 words · XuBaojin

Linux Network Stack Tuning: From Kernel to Application

Why Tune the Linux Network Stack The Linux kernel’s default network parameters are designed for general-purpose scenarios — conservative and safe. However, under high-concurrency web services, large-scale load balancers, CDN nodes, and similar workloads, these defaults become bottlenecks. Typical symptoms include: dmesg showing nf_conntrack: table full, dropping packet ss -s showing a large number of TIME-WAIT connections During load testing, CPU is not saturated but throughput plateaus NIC PPS (packets per second) is far below hardware capability Most of these issues stem from untuned kernel network parameters....

May 7, 2024 · 12 mins · 2493 words · XuBaojin

Linux Boot Process Explained: From Firmware to Userspace

Overview The Linux boot process is a precisely orchestrated multi-stage sequence — from firmware power-on self-test to kernel loading, all the way to userspace service startup. Each stage has its specific responsibilities. Understanding the complete boot process not only helps troubleshoot boot failures but also enables boot performance optimization. This article proceeds layer by layer from the firmware level through BIOS/UEFI, GRUB2, initramfs, kernel initialization, and systemd startup, covering boot optimization and kernel crash recovery....

May 3, 2024 · 18 mins · 3668 words · XuBaojin

Git Hooks Automation: From Code Quality to Deployment

Overview The earlier code quality issues are caught, the cheaper they are to fix. Catching issues at commit time is faster than at CI time, and catching them at CI time is faster than fixing them after they cause production incidents. Git Hooks provide the ability to automatically run checks at critical points — commit and push — including code formatting, linting, commit message validation, and test execution. This article covers everything from Git Hooks fundamentals to toolchain practices, building a code quality automation system from local to CI....

May 3, 2024 · 18 mins · 3628 words · XuBaojin

Kubernetes RBAC and Security Contexts

Overview The default Kubernetes security posture can be summed up in one word: open. The default ServiceAccount has broad API access (depending on version and PSP configuration), Pods run as root, containers can mount the host filesystem, and the network is fully open. This “default open” design lowers the barrier to entry but creates significant security risks in production. This article covers RBAC permission models, ServiceAccount management, Pod Security Standards, SecurityContext, NetworkPolicies, and audit logging—systematically explaining K8s security hardening practices....

May 2, 2024 · 12 mins · 2449 words · XuBaojin

Kubernetes Scheduler Principles and Tuning

Overview The Kubernetes scheduler is one of the most critical control plane components—it decides which node each Pod runs on. Scheduling quality directly impacts cluster resource utilization, application performance, and reliability. Understanding how the scheduler works is fundamental to effective K8s production operations. This article systematically covers the scheduling flow, filter-and-score mechanism, affinity, taints and tolerations, priority and preemption, and custom schedulers. Based on Kubernetes v1.30. Reference: Kubernetes Scheduler Documentation...

April 29, 2024 · 12 mins · 2478 words · XuBaojin

Error Budget Consumption Strategies and Action Guidelines

Overview The Error Budget is the most ingeniously designed mechanism in the SRE framework. It transforms the long-standing “stability vs. iteration speed” debate — previously settled by opinion and politics — into a quantifiable engineering decision framework: your system has an “unavailability allowance,” and when it’s spent, you stop and fix things. In practice, however, many teams define SLOs and error budgets but stop at displaying a percentage number on a dashboard....

April 26, 2024 · 17 mins · 3556 words · XuBaojin

Linux Package Management: apt/yum/dnf and Package Building

Overview Package management is foundational to Linux system administration. Debian-based systems use apt/dpkg, while Red Hat-based systems use yum/dnf/rpm. Understanding both package management ecosystems — their principles and usage — along with repository management, dependency resolution, package building, version pinning, and offline installation, is a prerequisite for efficient operations. This article systematically compares the two ecosystems and dives into practical package building and mirror optimization. apt/dpkg Ecosystem Architecture apt (high-level frontend) │ ├── apt-get → package install/remove/update ├── apt-cache → package query/search └── apt → unified command (user-friendly) │ dpkg (low-level tool) │ ├── dpkg → ....

April 26, 2024 · 19 mins · 3900 words · XuBaojin

Getting Started with Terraform Infrastructure as Code

Manually logging into cloud consoles to create servers, databases, and networks — this approach barely works when resources are few, but once environments grow complex, you end up unable to modify, clean up, or explain your infrastructure. Infrastructure as Code (IaC) uses code to describe infrastructure, making resource creation, modification, and destruction versionable, reviewable, and reusable. Terraform is currently the most popular IaC tool. This article covers everything from concepts to practice....

April 24, 2024 · 12 mins · 2429 words · XuBaojin

SLO Design in Practice: From Business Goals to Technical Metrics

Overview The first dilemma many teams face when practicing SRE is: they know what an SLO is, but they don’t know how to set one. They either copy Google’s 99.99% or pick an arbitrary 99.9% — only to find that the number neither reflects user experience nor drives engineering decisions. A good SLO isn’t plucked from thin air. It’s derived from business goals through a series of engineering methods: user journey analysis, metric selection, value calibration, multi-tier design, and regular review....

April 24, 2024 · 18 mins · 3637 words · XuBaojin