Troubleshooting

Overview 3 AM, you get paged — the production database can’t write, Nginx won’t start, SSH is sluggish. You log in and df -h shows the root partition at 100%. Every ops engineer has been there. Disk full is one of the most frequent production incidents — based on real-world operations statistics, disk-related alerts account for roughly 15%-20% of all alerts. But the 100% you see in df -h may not be the full picture:...

Overview It’s 2 AM and alerts are firing. Your service can’t connect to the database, curl times out, but ping to the IP works fine. Is it a network issue or an application issue? Nine times out of ten, it’s DNS resolution gone wrong. DNS resolution is the most fundamental yet most overlooked piece of Linux infrastructure. Nobody thinks about it until it breaks—and when it does, the entire system acts like it suddenly has amnesia: every domain becomes unrecognizable....

Overview 2 AM. Your phone rings. The order system is timing out across the board — CPU is fine, memory is fine, disk I/O is fine. Restarting services does nothing. Rolling back does nothing. You stare at the dashboard; every metric is green. Only the users are screaming. Nine times out of ten, it is a network-layer problem. And what you need is a pair of eyes that can actually see the packets....

Overview LLM-Assisted Troubleshooting: From Log Analysis to Root Cause Identification is an essential skill in SRE operations. In production environments, mastering these techniques can significantly improve system stability and operational efficiency. Why LLM-Assisted Troubleshooting Matters As systems grow in scale and complexity, traditional operations approaches struggle to meet the demands of modern distributed systems. LLM-Assisted Troubleshooting helps operations teams: Rapid Problem Resolution: Systematic tools and methods reduce troubleshooting time Improved System Visibility: Establish comprehensive monitoring and observability Proactive Fault Prevention: Identify and fix potential risks before they cause outages Resource Optimization: Allocate and schedule resources efficiently Core Concepts and Principles Basic Concepts The core of LLM-Assisted Troubleshooting lies in establishing standardized processes and automated toolchains....

Troubleshooting Path kubectl get pods → check status kubectl describe pod → check Events kubectl logs → check logs Common Pod States State Meaning Common Cause Pending Not scheduled Insufficient resources, scheduling constraints CrashLoopBackOff Crash loop App error, config issue ImagePullBackOff Image pull failed Image not found, auth failure OOMKilled Out of memory Memory limit too low CrashLoopBackOff Most common issue. Troubleshooting steps: # Check previous crash logs kubectl logs <pod> --previous # Check exit code kubectl get pod <pod> -o jsonpath='{....