Prometheus

Overview 3 AM. The alert group exploded. Five pods were OOMKilled and restarted within 10 minutes. The on-call engineer opened Grafana, and every pod’s working_set was still 200MB below its limit—80% utilization. Monitoring looked fine. The pods died anyway. This is not an isolated case. When I was doing a Kubernetes migration for a new-energy logistics platform, I dealt with this exact pattern—“monitoring is green but pods die”—more than 10 times....

Overview After deploying a CDN, many teams stop at glancing at the hit rate number on the vendor’s console. Hit rate at 90%? Looks fine, moving on. Then one day the origin bandwidth gets maxed out, pages take 8 seconds to load, user complaints flood in — and you discover that 90% was the request hit rate, not the byte hit rate. Large files were all going back to origin, while small requests inflated the number....

Overview At 2 AM, the payment system alerts exploded. After digging through logs for ages, we discovered the downstream services weren’t down — the API gateway’s connection pool was exhausted. Requests piled up at the gateway layer and never made it through. But our monitoring dashboard only showed CPU and memory curves, completely blind to gateway-level latency, connection counts, and error rates. This isn’t an isolated case. I’ve seen too many teams treat API gateways as “fancy Nginx” and only monitor nginx_active_connections, only digging into logs when things break....

Overview 3 AM, your phone is buzzing with alerts. The Kafka Consumer Lag for some consumer group jumped from 200 to 500,000. Downstream real-time dashboards are frozen. The data team is pinging you non-stop. You scramble onto the jump host, run kafka-consumer-groups.sh --describe, and see a column of alarming Lag numbers. Over the next half hour, you need to answer three questions: which partition is backed up? Is the consumer dead or just slow?...

Overview When the database slows down, the business slows down. Every ops engineer has heard this, but honestly, not many teams do database monitoring right. I’ve seen too many teams whose database monitoring looks like this: Zabbix templates running decade-old metrics, only two alert rules — “CPU over 90%” and “disk full” — while connection count, lock waits, cache hit rate, replication lag, and other metrics that actually provide early warning are not being collected at all....

Overview Picture this: it’s 3 AM, you get woken up by an alert, you drag yourself to open Grafana, and you flip through dozens of dashboards—none of them tell you anything useful. You do have millions of metrics, sure. But they’re all garbage. This is not an isolated case. I’ve seen too many teams deploy Prometheus and then forget about it. Metrics pile up, alerting rules multiply, and eventually the monitoring system itself crashes first: Prometheus OOMs on memory, queries time out after 30 seconds, alert evaluation lags by 5+ minutes....

Overview Prometheus’s conventional monitoring is “inside-out” — Prometheus scrapes Exporter metrics to understand internal system state. But when users access your service, they follow an “outside-in” path: DNS resolution → network routing → load balancing → backend service. A link that looks perfectly healthy from the inside may be completely inaccessible to users due to DNS misconfiguration, CDN cache issues, or expired SSL certificates. Blackbox Exporter solves the “external probing” problem....

Overview Monitoring systems themselves need to be monitored. Prometheus collects metrics from your entire infrastructure, but if Prometheus’s own configuration has issues, a target goes down, an SSL certificate is about to expire, or an alerting rule has a syntax error — who discovers these problems? The answer: an automated inspection script set. This article builds a complete inspection toolkit around the Prometheus ecosystem, covering “probing → certificate checking → config auditing → rule validation → alert simulation → report generation....

Overview Prometheus is the de facto standard in the cloud-native monitoring space, but it has notable shortcomings in long-term data storage and global querying: local storage retains only 15 days of data by default, single instances cannot aggregate queries across clusters, and high-availability solutions are relatively complex. Thanos, a CNCF incubating project, achieves unlimited-capacity long-term storage by uploading Prometheus data to object storage (such as S3, GCS, MinIO) and provides a cross-cluster global view through distributed query components....

Overview As the de facto standard for cloud-native monitoring, Prometheus has become the default choice for microservice and Kubernetes monitoring. However, as business scale grows, Prometheus’s local storage architecture increasingly exposes significant bottlenecks: limited single-machine storage capacity (default 15-day retention), no native horizontal scaling, memory spikes under high-cardinality scenarios, and difficulty querying historical data. Many teams find themselves facing the “triple dilemma” of disk IO pressure, storage cost inflation, and query latency growth once time series exceed the million-level mark....