https://www.sre.wang/en/tags/alert-noise/Recent content in Alert Noise on SRE Engineering Practicehttps://www.sre.wang/share-card-square.jpghttps://www.sre.wang/share-card-square.jpgHugo -- gohugo.ioenFri, 30 Oct 2026 00:40:16 +0800https://www.sre.wang/en/posts/loki-log-based-alerting-rules/Fri, 30 Oct 2026 00:40:16 +0800https://www.sre.wang/en/posts/loki-log-based-alerting-rules/Log alerting spams because it treats 'this appeared in the logs' as 'this is broken right now.' Using a real incident that fired 400+ notifications in ten minutes, this article walks through a two-tier fix: recording rules first compress the event stream into state metrics, then alert rules fire only when that state persists. Includes a complete Loki ruler configuration, copy-ready rule files, label convergence thresholds, and the four production traps — silent WAL replay, random re-sharding, no globs on local storage, and inconsistent local rule evaluation.