Posts

K8s Storage System: Three-Layer Abstraction The Kubernetes storage system decouples storage consumers from providers through three layers of abstraction. This is the core concept for understanding container persistent storage. According to the K8s storage documentation, the three-layer structure is: ┌──────────────────────────────────────────────────────┐ │ Pod (Consumer) │ │ volumeMounts → volumes │ ├──────────────────────────────────────────────────────┤ │ PVC (Claim) — User requests storage │ │ "I need 10Gi RWO storage" │ ├──────────────────────────────────────────────────────┤ │ StorageClass (Dynamic Provisioning) │ │ "Use ceph-rbd driver, reclaim: Retain" │ ├──────────────────────────────────────────────────────┤ │ PV (Physical Resource) — Actual storage │ │ "10....

Overview In the monitoring system space, Zabbix and Prometheus are the two giants. Zabbix comes from the traditional operations era, dominating physical machine/VM environments for nearly 25 years; Prometheus rose in the cloud-native era, becoming the de facto standard for the Kubernetes ecosystem. Many teams face a question when choosing a monitoring system: Zabbix or Prometheus? The answer isn’t either/or. Many mature teams run both systems simultaneously in production — Zabbix handles the infrastructure layer (network, hardware, OS), while Prometheus handles the application and cloud-native layers....

Semantics of Requests and Limits In Kubernetes, each container can be configured with CPU and Memory requests and limits. Many developers fail to distinguish between the two, leading to frequent Pod evictions or OOMKilled events. apiVersion: v1 kind: Pod metadata: name: api-server spec: containers: - name: app image: myapp:latest resources: requests: cpu: "250m" # 0.25 core memory: "256Mi" limits: cpu: "500m" # 0.5 core memory: "512Mi" Core Differences Dimension Requests Limits Phase At scheduling time At runtime Meaning Minimum guaranteed resources for the Pod Maximum resource cap for the Pod Scheduler behavior Scheduler uses requests to determine if a node has sufficient resources Scheduler ignores limits Runtime behavior Guaranteed share in cgroups CPU is throttled; Memory triggers OOMKilled Overcommit allowed Yes (sum of all Pod limits on a node can exceed node capacity) Memory overcommit is not recommended In simple terms:...

Overview Performance profiling is a core SRE skill. Linux provides a rich set of performance analysis tools, from the simple top to the powerful perf/eBPF — each with its applicable scenario. Knowing when and how to use these tools is key to quickly identifying performance bottlenecks. This article systematically covers the Linux performance profiling toolkit across five dimensions — CPU, memory, IO, network, and comprehensive tools — and provides use-case quick reference tables and real-world examples....

Introduction Shell scripts are the most common automation tool for operations engineers. This post records several practical patterns used in the field. Log Rotation Cleanup #!/bin/bash # clean_logs.sh - Log cleanup script LOG_DIR="/var/log/app" KEEP_DAYS=30 find "$LOG_DIR" -name "*.log" -type f -mtime +${KEEP_DAYS} -exec gzip {} \; find "$LOG_DIR" -name "*.gz" -type f -mtime +${KEEP_DAYS} -delete echo "$(date): Log cleanup complete, kept ${KEEP_DAYS} days" Run daily via crontab: 0 2 * * * /opt/scripts/clean_logs....

Overview The filesystem is the bridge between the operating system and storage devices, directly impacting data reliability, I/O performance, and operational complexity. Linux supports multiple filesystems, each with its own design trade-offs. This article compares the four mainstream filesystems — ext4, xfs, btrfs, and zfs — and dives into mount parameter optimization, I/O scheduler selection, journal modes, fsync performance, and other core topics, with production-grade selection recommendations and tuning strategies....

Overview Bash is the most widely used scripting language in the ops world — nearly every Linux server ships with the interpreter, requiring no runtime installation. But Bash is also the language most prone to producing scripts that “work but can’t be trusted”: no type checking, errors are silent by default, variables get word-split without quotes, and pipeline failures get swallowed. The root cause of many production incidents is a Bash script that didn’t handle errors properly....

Overview The Linux kernel exposes thousands of tunable parameters through /proc/sys/ and the sysctl interface, covering networking, memory, filesystem, security, and more. Properly adjusting these parameters can significantly improve system performance and stability, but blind tuning can be counterproductive. This article systematically covers the sysctl system across four dimensions — networking, file descriptors, memory, and security — and provides production tuning templates and parameter validation methods. sysctl System Viewing and Modifying Parameters # View all parameters $ sysctl -a # View specific parameter $ sysctl net....

Introduction A default Linux server installation exposes a far larger attack surface than most realize: unnecessary RPM packages, open network services, permissive SSH configurations, and inactive audit systems. Security hardening is not a one-time operation but a full lifecycle checklist from installation to runtime. This article follows the CIS Benchmark framework to organize hardening essentials, with directly executable configurations for each step. System Minimal Installation Principles Package Management Trimming The core principle of minimal installation: install only what you need, not what you might need....

Overview SSH (Secure Shell) is the core channel for Linux operations and a primary target for attackers. A misconfigured SSH service can lead to a full server compromise. This article covers SSH security best practices comprehensively — from authentication mechanisms, server-side hardening, bastion host architecture, and tunneling techniques to audit logging and brute-force protection. Key-Based Authentication Generating Key Pairs # Generate Ed25519 key (recommended) $ ssh-keygen -t ed25519 -C "admin@sre.wang" -f ~/....