Observability

Overview At 2 AM, I got a phone call. Users of a ride-hailing project reported that opening the App to view trip history during late-night hours caused a 3-5 second blank screen. After investigation, I found that this API group used Serverless function compute. During the day, traffic was normal, but after traffic dropped off at night, function instances were reclaimed. The first request triggered a cold start, pushing P99 latency to 4....

Overview 3 AM, phone vibrating. The alert channel exploded with 200 messages, and user complaints are already flooding customer service. You scramble to your laptop, log into the bastion host, and discover a core API endpoint’s P99 latency spiked to 8 seconds. For the next 30 minutes, you dig through logs, check Grafana dashboards, and ping upstream and downstream teams—only to find that a config center pushed an incorrect parameter....

Overview Production blows up, you open Kibana to search logs, and discover the critical Pod was OOM-killed 3 minutes ago—the old logs vanished with the container. This is extremely common in K8s environments: Pods are ephemeral, and logs can’t disappear with them. K8s log collection is fundamentally different from traditional VM environments. In a traditional setup, logs sit quietly in /var/log/ and you just SSH in to read them. In K8s, Pods can be scheduled to any node at any time, destroyed and recreated at will, with logs scattered across the entire cluster....

Overview Have you ever encountered this situation: users report “the system is slow,” you open Grafana and check a bunch of dashboards—CPU is fine, memory is fine, network is fine—but users insist it’s slow. What you need at this point is not more metric dashboards, but a complete request trace—from the moment the user clicks a button to when the database returns results, showing exactly how long each hop took and where it got stuck....

Overview Picture this: it’s 3 AM, you get woken up by an alert, you drag yourself to open Grafana, and you flip through dozens of dashboards—none of them tell you anything useful. You do have millions of metrics, sure. But they’re all garbage. This is not an isolated case. I’ve seen too many teams deploy Prometheus and then forget about it. Metrics pile up, alerting rules multiply, and eventually the monitoring system itself crashes first: Prometheus OOMs on memory, queries time out after 30 seconds, alert evaluation lags by 5+ minutes....

Overview In modern microservice architectures, a seemingly simple user request may traverse dozens of service nodes. When an incident occurs, the first question an SRE engineer faces is often not “how to fix it” but “what is the scope of impact.” Without a fast answer to this question, incident recovery gets bogged down in endless investigation. Service Dependency Maps and Failure Domain Analysis are the engineering methodologies that address this problem....

Overview In the era of cloud-native and microservices, a single request may traverse dozens of service nodes. Traditional monitoring scatters Metrics, Logs, and Traces across different systems — Prometheus for metrics, ELK for log search, Jaeger for tracing — with no unified way to correlate them. When an online incident occurs, you need to switch between three systems, manually piecing together correlated information, which is highly inefficient. OpenTelemetry (OTel) is the CNCF-led unified observability standard, aiming to use a single SDK/API to collect all three signals (Metrics, Logs, Traces), process them through a unified Collector, and send them to any backend....

Why Distributed Tracing In a microservices architecture, a single user request often traverses multiple services. When an endpoint’s latency spikes from 200ms to 2s, logs are scattered across N machines, making it difficult to pinpoint the bottleneck — is it slow gateway forwarding, a slow downstream DB query, or queuing in an inter-service call? Distributed tracing solves exactly this problem: it generates a globally unique Trace ID for each request, propagates it across services, and ultimately renders a complete call chain tree in the UI, making the duration of each segment visible at a glance....

Why Choose Loki The traditional ELK (Elasticsearch + Logstash + Kibana) stack is powerful, but has two core pain points: High storage costs: Elasticsearch fully indexes log content, with index bloat reaching 3-5x the raw data Operational complexity: ES cluster scaling, shard rebalancing, and index lifecycle management are complex, making production clusters costly to maintain Loki, open-sourced by Grafana Labs, is designed around the philosophy of “doing for logs what Prometheus did for metrics....