Monitoring & Alerting

Overview At 2 AM, the payment system alerts exploded. After digging through logs for ages, we discovered the downstream services weren’t down — the API gateway’s connection pool was exhausted. Requests piled up at the gateway layer and never made it through. But our monitoring dashboard only showed CPU and memory curves, completely blind to gateway-level latency, connection counts, and error rates. This isn’t an isolated case. I’ve seen too many teams treat API gateways as “fancy Nginx” and only monitor nginx_active_connections, only digging into logs when things break....

Overview No matter how fancy your backend monitoring is, if the user opens the page and sees a blank screen for 5 seconds, your all-green SLOs mean nothing. This isn’t me being contrarian. Many teams have dashboards full of CPU, memory, QPS, and P99 latency metrics, looking perfectly healthy. But real users are already flooding customer support with complaints — “page loads slow,” “can’t click anything,” “images keep jumping around.” Where’s the problem?...

Overview Prometheus’s conventional monitoring is “inside-out” — Prometheus scrapes Exporter metrics to understand internal system state. But when users access your service, they follow an “outside-in” path: DNS resolution → network routing → load balancing → backend service. A link that looks perfectly healthy from the inside may be completely inaccessible to users due to DNS misconfiguration, CDN cache issues, or expired SSL certificates. Blackbox Exporter solves the “external probing” problem....

Overview In the era of cloud-native and microservices, a single request may traverse dozens of service nodes. Traditional monitoring scatters Metrics, Logs, and Traces across different systems — Prometheus for metrics, ELK for log search, Jaeger for tracing — with no unified way to correlate them. When an online incident occurs, you need to switch between three systems, manually piecing together correlated information, which is highly inefficient. OpenTelemetry (OTel) is the CNCF-led unified observability standard, aiming to use a single SDK/API to collect all three signals (Metrics, Logs, Traces), process them through a unified Collector, and send them to any backend....

Overview Traditional monitoring is “passive” — it waits for users to access the system, triggering system behavior, then collects metrics and logs. This approach has a fundamental flaw: when monitoring detects a problem, users have already been impacted. If your homepage takes 10 seconds to load, your monitoring alert may not trigger for 5 minutes — by which time thousands of users have experienced poor performance. Synthetic Monitoring is a “proactive” monitoring approach — it simulates real user behavior, regularly accessing critical paths to discover and fix issues before users perceive them....

Overview Alerting is the “last mile” of a monitoring system — and the hardest to get right. A common predicament: servers run dozens of alerting rules generating hundreds of alert notifications daily. On-call engineers, bombarded by WeChat/DingTalk/email, gradually become desensitized — truly urgent alerts are drowned in noise until customer complaints reveal the system has been broken for hours. The golden rule of SRE: every alert must have a clear action....

Overview Prometheus defaults to a single-node architecture, which is a dangerous liability in production environments. A single Prometheus instance going down means your entire monitoring system goes blind — you can’t see any metrics during the failure, and alerts stop working because rules are no longer evaluated. As data volume grows to the limits of a single machine’s storage and processing capacity, you’ll face write timeouts, slow queries, and disk exhaustion....

Overview In the Prometheus monitoring system, Service Discovery (SD) is the bridge connecting “monitoring targets” to the “scrape engine.” When your infrastructure scales from a few VMs to hundreds of Kubernetes Pods, cross-AZ cloud instances, and Consul-registered nodes, manually maintaining static_configs becomes a nightmare — every scale-up, scale-down, or migration requires config changes and Prometheus restarts, and alerts may misfire due to unreachable targets. Prometheus natively supports over a dozen service discovery mechanisms that can automatically detect target changes without restarts....

Overview Among the three pillars of observability (Metrics, Logs, Traces), logs are the data closest to the application layer. When an online service misbehaves, the first instinct is usually “check the logs.” The ELK Stack (Elasticsearch + Logstash + Kibana) is the de facto standard in log analysis, offering powerful capabilities in full-text search, log parsing, and visual analytics. With the rise of “lightweight” log solutions like Loki, ELK faces criticism for “high storage costs and operational complexity....

Overview When selecting a monitoring system, one of the most debated questions is “commercial platform or self-hosted open-source.” Datadog is the benchmark for commercial observability platforms — out-of-the-box, feature-complete, with rich integrations, but at a premium price. Prometheus + Grafana represents the open-source self-hosted approach — flexible, controllable, with no license fees, but requiring investment in operations personnel. This isn’t a simple “save money vs save effort” choice. For fast-growing startups, Datadog’s out-of-the-box experience may be more valuable than the license fees saved....