Monitoring & Alerting

Overview In the Prometheus monitoring system, Service Discovery (SD) is the bridge connecting “monitoring targets” to the “scrape engine.” When your infrastructure scales from a few VMs to hundreds of Kubernetes Pods, cross-AZ cloud instances, and Consul-registered nodes, manually maintaining static_configs becomes a nightmare — every scale-up, scale-down, or migration requires config changes and Prometheus restarts, and alerts may misfire due to unreachable targets. Prometheus natively supports over a dozen service discovery mechanisms that can automatically detect target changes without restarts....

Overview 3 AM. Your phone buzzes. You get out of bed, open your laptop, SSH in, and find a service with CPU spiking. Kill the process, restart the service, 12 minutes done — but you’re wide awake now. 4:30 AM, another alert: disk usage exceeds 85%. Get up again, du -sh to locate, delete old logs, 15 minutes. This is the daily reality for countless operations engineers. Monitoring is set up, alerts are configured, scripts are written — but the final step still relies on humans....

Why Distributed Tracing In a microservices architecture, a single user request often traverses multiple services. When an endpoint’s latency spikes from 200ms to 2s, logs are scattered across N machines, making it difficult to pinpoint the bottleneck — is it slow gateway forwarding, a slow downstream DB query, or queuing in an inter-service call? Distributed tracing solves exactly this problem: it generates a globally unique Trace ID for each request, propagates it across services, and ultimately renders a complete call chain tree in the UI, making the duration of each segment visible at a glance....

Overview Among the three pillars of observability (Metrics, Logs, Traces), logs are the data closest to the application layer. When an online service misbehaves, the first instinct is usually “check the logs.” The ELK Stack (Elasticsearch + Logstash + Kibana) is the de facto standard in log analysis, offering powerful capabilities in full-text search, log parsing, and visual analytics. With the rise of “lightweight” log solutions like Loki, ELK faces criticism for “high storage costs and operational complexity....

Overview When selecting a monitoring system, one of the most debated questions is “commercial platform or self-hosted open-source.” Datadog is the benchmark for commercial observability platforms — out-of-the-box, feature-complete, with rich integrations, but at a premium price. Prometheus + Grafana represents the open-source self-hosted approach — flexible, controllable, with no license fees, but requiring investment in operations personnel. This isn’t a simple “save money vs save effort” choice. For fast-growing startups, Datadog’s out-of-the-box experience may be more valuable than the license fees saved....

Grafana is the most popular visualization platform in the cloud-native era, but there’s a world of difference between “functional” and “effective.” A cluttered dashboard leaves on-call engineers lost in a sea of panels, while a well-designed one conveys system health in 5 seconds. This article starts from design principles, covers the variable system, panel selection, and alerting integration, and ties everything together with a complete SLO dashboard. Reference: Grafana Official Documentation...

PromQL (Prometheus Query Language) is the query language of the Prometheus monitoring system and the core of the cloud-native monitoring stack. Whether you’re building Grafana dashboards, writing alerting rules, or running ad-hoc queries during incident troubleshooting, PromQL is indispensable. This article starts from the data model and progressively covers aggregation operations, common functions, practical queries, and advanced techniques like subqueries. Reference: Prometheus Official Documentation — Querying basics I. PromQL Data Model PromQL has four fundamental data types....

Why Choose Loki The traditional ELK (Elasticsearch + Logstash + Kibana) stack is powerful, but has two core pain points: High storage costs: Elasticsearch fully indexes log content, with index bloat reaching 3-5x the raw data Operational complexity: ES cluster scaling, shard rebalancing, and index lifecycle management are complex, making production clusters costly to maintain Loki, open-sourced by Grafana Labs, is designed around the philosophy of “doing for logs what Prometheus did for metrics....

In the Prometheus ecosystem, Prometheus generates alerts based on alerting rules, while Alertmanager manages the entire alert lifecycle: grouping, routing, inhibition, deduplication, and notification delivery. A poorly configured Alertmanager can drown on-call engineers in a flood of duplicate alerts at 3 AM, whereas a well-designed routing and inhibition strategy ensures that “the right person receives the right alert at the right time.” Reference: Prometheus Official Documentation — Alertmanager I. Alertmanager Architecture Alertmanager’s processing pipeline consists of five stages:...

Overview In the monitoring system space, Zabbix and Prometheus are the two giants. Zabbix comes from the traditional operations era, dominating physical machine/VM environments for nearly 25 years; Prometheus rose in the cloud-native era, becoming the de facto standard for the Kubernetes ecosystem. Many teams face a question when choosing a monitoring system: Zabbix or Prometheus? The answer isn’t either/or. Many mature teams run both systems simultaneously in production — Zabbix handles the infrastructure layer (network, hardware, OS), while Prometheus handles the application and cloud-native layers....