Monitoring & Alerting

Overview Prometheus’s conventional monitoring is “inside-out” — Prometheus scrapes Exporter metrics to understand internal system state. But when users access your service, they follow an “outside-in” path: DNS resolution → network routing → load balancing → backend service. A link that looks perfectly healthy from the inside may be completely inaccessible to users due to DNS misconfiguration, CDN cache issues, or expired SSL certificates. Blackbox Exporter solves the “external probing” problem....

Overview In the era of cloud-native and microservices, a single request may traverse dozens of service nodes. Traditional monitoring scatters Metrics, Logs, and Traces across different systems — Prometheus for metrics, ELK for log search, Jaeger for tracing — with no unified way to correlate them. When an online incident occurs, you need to switch between three systems, manually piecing together correlated information, which is highly inefficient. OpenTelemetry (OTel) is the CNCF-led unified observability standard, aiming to use a single SDK/API to collect all three signals (Metrics, Logs, Traces), process them through a unified Collector, and send them to any backend....

Overview Traditional monitoring is “passive” — it waits for users to access the system, triggering system behavior, then collects metrics and logs. This approach has a fundamental flaw: when monitoring detects a problem, users have already been impacted. If your homepage takes 10 seconds to load, your monitoring alert may not trigger for 5 minutes — by which time thousands of users have experienced poor performance. Synthetic Monitoring is a “proactive” monitoring approach — it simulates real user behavior, regularly accessing critical paths to discover and fix issues before users perceive them....

Overview Alerting is the “last mile” of a monitoring system — and the hardest to get right. A common predicament: servers run dozens of alerting rules generating hundreds of alert notifications daily. On-call engineers, bombarded by WeChat/DingTalk/email, gradually become desensitized — truly urgent alerts are drowned in noise until customer complaints reveal the system has been broken for hours. The golden rule of SRE: every alert must have a clear action....

Overview Prometheus defaults to a single-node architecture, which is a dangerous liability in production environments. A single Prometheus instance going down means your entire monitoring system goes blind — you can’t see any metrics during the failure, and alerts stop working because rules are no longer evaluated. As data volume grows to the limits of a single machine’s storage and processing capacity, you’ll face write timeouts, slow queries, and disk exhaustion....

Overview In the Prometheus monitoring system, Service Discovery (SD) is the bridge connecting “monitoring targets” to the “scrape engine.” When your infrastructure scales from a few VMs to hundreds of Kubernetes Pods, cross-AZ cloud instances, and Consul-registered nodes, manually maintaining static_configs becomes a nightmare — every scale-up, scale-down, or migration requires config changes and Prometheus restarts, and alerts may misfire due to unreachable targets. Prometheus natively supports over a dozen service discovery mechanisms that can automatically detect target changes without restarts....

Overview Among the three pillars of observability (Metrics, Logs, Traces), logs are the data closest to the application layer. When an online service misbehaves, the first instinct is usually “check the logs.” The ELK Stack (Elasticsearch + Logstash + Kibana) is the de facto standard in log analysis, offering powerful capabilities in full-text search, log parsing, and visual analytics. With the rise of “lightweight” log solutions like Loki, ELK faces criticism for “high storage costs and operational complexity....

Overview When selecting a monitoring system, one of the most debated questions is “commercial platform or self-hosted open-source.” Datadog is the benchmark for commercial observability platforms — out-of-the-box, feature-complete, with rich integrations, but at a premium price. Prometheus + Grafana represents the open-source self-hosted approach — flexible, controllable, with no license fees, but requiring investment in operations personnel. This isn’t a simple “save money vs save effort” choice. For fast-growing startups, Datadog’s out-of-the-box experience may be more valuable than the license fees saved....

Grafana is the most popular visualization platform in the cloud-native era, but there’s a world of difference between “functional” and “effective.” A cluttered dashboard leaves on-call engineers lost in a sea of panels, while a well-designed one conveys system health in 5 seconds. This article starts from design principles, covers the variable system, panel selection, and alerting integration, and ties everything together with a complete SLO dashboard. Reference: Grafana Official Documentation...

PromQL (Prometheus Query Language) is the query language of the Prometheus monitoring system and the core of the cloud-native monitoring stack. Whether you’re building Grafana dashboards, writing alerting rules, or running ad-hoc queries during incident troubleshooting, PromQL is indispensable. This article starts from the data model and progressively covers aggregation operations, common functions, practical queries, and advanced techniques like subqueries. Reference: Prometheus Official Documentation — Querying basics I. PromQL Data Model PromQL has four fundamental data types....