Monitoring & Alerting

Overview When the database slows down, the business slows down. Every ops engineer has heard this, but honestly, not many teams do database monitoring right. I’ve seen too many teams whose database monitoring looks like this: Zabbix templates running decade-old metrics, only two alert rules — “CPU over 90%” and “disk full” — while connection count, lock waits, cache hit rate, replication lag, and other metrics that actually provide early warning are not being collected at all....

Overview Have you ever encountered this situation: users report “the system is slow,” you open Grafana and check a bunch of dashboards—CPU is fine, memory is fine, network is fine—but users insist it’s slow. What you need at this point is not more metric dashboards, but a complete request trace—from the moment the user clicks a button to when the database returns results, showing exactly how long each hop took and where it got stuck....

Overview Picture this: it’s 3 AM, you get woken up by an alert, you drag yourself to open Grafana, and you flip through dozens of dashboards—none of them tell you anything useful. You do have millions of metrics, sure. But they’re all garbage. This is not an isolated case. I’ve seen too many teams deploy Prometheus and then forget about it. Metrics pile up, alerting rules multiply, and eventually the monitoring system itself crashes first: Prometheus OOMs on memory, queries time out after 30 seconds, alert evaluation lags by 5+ minutes....

Overview Prometheus’s conventional monitoring is “inside-out” — Prometheus scrapes Exporter metrics to understand internal system state. But when users access your service, they follow an “outside-in” path: DNS resolution → network routing → load balancing → backend service. A link that looks perfectly healthy from the inside may be completely inaccessible to users due to DNS misconfiguration, CDN cache issues, or expired SSL certificates. Blackbox Exporter solves the “external probing” problem....

Overview Prometheus is the de facto standard in the cloud-native monitoring space, but it has notable shortcomings in long-term data storage and global querying: local storage retains only 15 days of data by default, single instances cannot aggregate queries across clusters, and high-availability solutions are relatively complex. Thanos, a CNCF incubating project, achieves unlimited-capacity long-term storage by uploading Prometheus data to object storage (such as S3, GCS, MinIO) and provides a cross-cluster global view through distributed query components....

Overview In the era of cloud-native and microservices, a single request may traverse dozens of service nodes. Traditional monitoring scatters Metrics, Logs, and Traces across different systems — Prometheus for metrics, ELK for log search, Jaeger for tracing — with no unified way to correlate them. When an online incident occurs, you need to switch between three systems, manually piecing together correlated information, which is highly inefficient. OpenTelemetry (OTel) is the CNCF-led unified observability standard, aiming to use a single SDK/API to collect all three signals (Metrics, Logs, Traces), process them through a unified Collector, and send them to any backend....

Overview Traditional monitoring is “passive” — it waits for users to access the system, triggering system behavior, then collects metrics and logs. This approach has a fundamental flaw: when monitoring detects a problem, users have already been impacted. If your homepage takes 10 seconds to load, your monitoring alert may not trigger for 5 minutes — by which time thousands of users have experienced poor performance. Synthetic Monitoring is a “proactive” monitoring approach — it simulates real user behavior, regularly accessing critical paths to discover and fix issues before users perceive them....

Overview As the de facto standard for cloud-native monitoring, Prometheus has become the default choice for microservice and Kubernetes monitoring. However, as business scale grows, Prometheus’s local storage architecture increasingly exposes significant bottlenecks: limited single-machine storage capacity (default 15-day retention), no native horizontal scaling, memory spikes under high-cardinality scenarios, and difficulty querying historical data. Many teams find themselves facing the “triple dilemma” of disk IO pressure, storage cost inflation, and query latency growth once time series exceed the million-level mark....

Overview Alerting is the “last mile” of a monitoring system — and the hardest to get right. A common predicament: servers run dozens of alerting rules generating hundreds of alert notifications daily. On-call engineers, bombarded by WeChat/DingTalk/email, gradually become desensitized — truly urgent alerts are drowned in noise until customer complaints reveal the system has been broken for hours. The golden rule of SRE: every alert must have a clear action....

Overview Prometheus defaults to a single-node architecture, which is a dangerous liability in production environments. A single Prometheus instance going down means your entire monitoring system goes blind — you can’t see any metrics during the failure, and alerts stop working because rules are no longer evaluated. As data volume grows to the limits of a single machine’s storage and processing capacity, you’ll face write timeouts, slow queries, and disk exhaustion....