One Error Log Pinged Me 400 Times: Two-Tier Log Alerting That Tells 'It Happened' Apart From 'It Is Happening'

Overview

At 1:12 a.m. on a night in August, my phone started buzzing and would not stop.

Checkout was broken on an e-commerce platform. A downstream connection pool behind the payment gateway had been saturated, the checkout service retried in a tight loop, and the logs filled up with PAYMENT_GATEWAY_TIMEOUT. The on-call engineer found the root cause within seven minutes, reset the pool ceiling and the retry count, and traffic was back to normal by 1:59 a.m.

The incident itself was nothing special — 47 minutes, start to finish. What wrecked my night was the pager: the alert channel received more than 400 notifications in those ten minutes, every one of them from the same rule, every one of them describing the same problem.

The rule looked like this:

- alert: CheckoutPaymentTimeout
  expr: count_over_time({app="checkout"} |= "PAYMENT_GATEWAY_TIMEOUT" [1m]) > 0
  labels:
    severity: critical

Nothing in it is literally wrong. The string PAYMENT_GATEWAY_TIMEOUT really was in the logs. But the expression means “appeared at least once in the past minute,” and everyone reading the page assumed it meant “payments are failing right now.” So every minute the log got “seen” once, and every minute a brand-new notification went out. Multiply that by the number of matching pods and 47 minutes turns into several hundred pages.

While writing the postmortem, I pinned one sentence in the on-call channel: when log alerts flood the channel, nine times out of ten it is not a misconfigured threshold — it is a rule that confuses “it happened” with “it is happening.”

This article is about fixing that. The core idea fits in one sentence: log alerting needs two tiers — a recording rule turns the event stream into a state you can actually judge, and an alert rule fires only when that state persists. A log alert should not be “the keyword showed up, so page someone.” It should be a state machine.

The article has seven parts:

• 1. Where log alerting belongs: black-box components, event-style alerts, and high-cardinality sources — not resource saturation or availability
• 2. The root cause: logs are an event stream, not state; count_over_time > 0 is guaranteed to spam, and why you need two tiers
• 3. Tier one, recording rules: turning “it appeared” into “how fast, how dense, for how long” — and how far label convergence should go
• 4. Tier two, alert rules: using for properly, why > 0 is the wrong threshold, how a bad limit silently swallows alerts, and multi-tenant routing limits
• 5. Four production traps in the ruler: no evaluation during WAL replay, no re-sharding handover, no globs on local storage, and local evaluation disagreeing with Grafana
• 6. Rollout order: five checkpoints from rewriting rules to rewriting the on-call contract
• 7. Seven anti-patterns I have watched run in production

If you are still alerting on “ERROR appeared in the logs,” this should cut your interrupted nights by an order of magnitude. How you build the collection layer is the other half of the problem — see Related: Log Monitoring System: Loki + Promtail Deployment.

1. Where Log Alerting Belongs

Draw the boundary first. Otherwise the rule count only ever grows.

1.1 Three Use Cases That Actually Fit

The first is black-box components. You do not control every component in your stack. Load balancers, third-party gateways, and legacy Java middleware often expose no metrics at all — but they do produce logs. Deriving an availability signal from those logs and alerting on the signal is the cheapest way to drag inherited architecture into your observability stack. Grafana’s Loki documentation calls this black box monitoring.

The second is the event-style alert. Some problems only matter once: you care whether something happened, not how long it lasted. Leaked credentials are the classic case — a URL in the form http://user:pass@ appearing anywhere in the logs deserves a human look. The example rule in the Loki docs is exactly this shape:

- name: credentials_leak
  rules:
    - alert: http-credentials-leaked
      annotations:
        message: "{{ $labels.job }} is leaking http basic auth credentials."
      expr: 'sum by (cluster, job, pod) (count_over_time({namespace="prod"} |~ "http(s?)://(\\w+):(\\w+)@" [5m]) > 0)'
      for: 10m
      labels:
        severity: critical

Notice that the two-tier idea is already hiding in there: > 0 is an instantaneous condition, but for: 10m turns it into “leaked credentials have been visible continuously for ten minutes” (the pending/firing semantics of for are specified in the Prometheus alerting rules documentation). Event-style alerts may use > 0, but they must be paired with for and real deduplication — otherwise you get the rule that paged 400 times.

The third is high-cardinality sources. This is where log alerting is irreplaceable. Suppose you want to alert per tenant. Add a tenant label to any Prometheus metric and its cardinality multiplies by the number of tenants; the TSDB falls over. In LogQL, aggregation happens at query time — the raw data stays in log storage and only the aggregated result reaches the rule engine. The Loki docs use Loki monitoring itself as the example:

sum by (org_id) (rate({job="loki-prod/query-frontend"} |= "metrics.go" | logfmt | duration > 10s [1m]))

That answers “which tenant’s queries are slower than ten seconds.” As a metric label, the same dimension would cost something completely different.

1.2 What Log Alerting Should Not Do

Without a boundary, rules grow into a thicket. This table is the split I use in real projects:

1.3 Three Questions Before You Write a Rule

I now ask three questions before writing any rule:

1. Can this signal be collected as a metric at all? If yes, take the metric path. Leave the log path for what metrics cannot reach.
2. Does the alert need to judge persistence? If yes, write the recording rule first — or just use metrics.
3. Can the person who receives this page act on it immediately? If the only possible action is “open Grafana and look,” then it is not an alert. It is a dashboard panel.

The third question gets ignored most often, and it determines whether your alert channel stays usable. The Google SRE book (Monitoring Distributed Systems) keeps coming back to one point: every page should be actionable, and should target symptoms rather than causes. Turning something you merely “glance at” into a critical alert is borrowing against your future on-call shifts.

2. The Root Cause: Confusing “It Happened” With “It Is Happening”

2.1 How 400 Alerts Were Generated

Back to that night. During the postmortem we did the arithmetic:

alerts ≈ ceil(incident duration / rule evaluation interval) × matching series
       = ceil(47 min / 1 min) × (3 replicas × 2 env labels)
       ≈ 282 ~ 400+

Not a single parameter in that formula is wrong. A one-minute window is a normal choice. Three replicas is a normal replica count. The problem is the semantics of the whole expression: count_over_time(... [1m]) > 0 answers “did this appear in the last minute?” while the on-call engineer needs “is it broken right now, and how badly?”

The expression’s output is binary, and as long as the problem persists, it keeps answering “yes.” So it fires on every evaluation, sends a fresh notification on every evaluation, and keeps doing that until the problem goes away.

That is the spam mechanism: a persistent problem gets announced repeatedly by an instantaneous test, and every announcement is a new notification.

2.2 Logs Are an Event Stream, Not a State

One comparison with metrics makes this obvious.

Prometheus’s up == 0 can express “it is down right now” because the metric model is periodic sampling: every 15 seconds (or whatever the scrape interval is) the target reports “I am still here.” It is a curve extending through time, and a single missing point means something is wrong.

Logs do not work that way. Logs are discrete events. Matching |= "PAYMENT_GATEWAY_TIMEOUT" finds a line of text. Its presence on the timeline means only that the event happened once — not that it is still happening, not how many times it happened, and not how bad it is.

An everyday analogy: a metric is a thermometer you read on a schedule, so you can say “38.5 °C now, 37 °C yesterday.” A log line is a note in a medical chart that says “fever, some day.” Staring at the word “fever” in a chart tells you a fever occurred; it tells you nothing about the current temperature. Alerting on “the word fever exists in the chart” means paging someone every time you turn to that page.

2.3 Two-Tier Design: Recording Rules as a State Machine

The fix is to separate the two concerns:

• Tier one (recording rules): compress the event stream into numeric time series. The question changes from “did it happen” to “how fast, how dense, what share, for how long.” The ruler computes the answer on a fixed interval and stores it as a new time series.
• Tier two (alert rules): evaluate that state metric using ratios or rates, and require persistence via for.

log stream (discrete events)
   │
   ├─ tier one: recording rules, fixed evaluation interval
   │     ex. sum by (service) (rate({app=~"checkout|payment"} |= "PAYMENT_GATEWAY_TIMEOUT" [5m]))
   │     → produces  loki:payment_timeout:rate5m{service="checkout"}
   │
   └─ tier two: alert rules, judging "has the state persisted"
         ex. loki:payment_timeout:rate5m > 0.5  plus  for: 10m
         → one incident, one alert

Why not cram both halves into a single LogQL expression and skip recording rules? Three reasons, in order of importance:

1. An alert rule’s expression can only describe the current instant, not the shape of a time window. “Every minute for the last ten minutes, the rate exceeded the threshold” is awkward to write as one expression, and it re-runs the expensive query on every evaluation.
2. The recording rule’s output can be reused by other consumers. A dashboard can read that metric directly instead of scanning logs on every refresh, and a cross-cluster view can be unified by shipping it via remote_write to Prometheus or Mimir.
3. Evaluation cost becomes controllable. Tier one produces a low-cardinality series; tier two merely compares a few numbers. Un-tiered rules re-run the log query on every evaluation, so ruler load grows linearly with rule count.

Worth adding: this is also why Loki ships recording rules and alert rules in the same rule file, evaluated by the same ruler component. They are two halves of one pipeline, not an optional advanced feature. For the Prometheus semantics behind alert and recording rules, see Related: Alerting Strategy Design: From Noise to Signal.

3. Tier One: Recording Rules Turn Events Into a Judgeable State

3.1 What Recording Rules Solve Here

A recording rule does something simple to describe: it runs a query on a fixed interval and stores the result as a new time series. Prometheus has worked this way for years (Recording rules | Prometheus); Loki just moved the same mechanism to the log side. The value is not “fewer queries” — it is that a semantically fuzzy thing (an event stream) becomes a semantically precise thing (a numeric curve).

Three numeric questions are enough to judge “is payments broken”:

• Timeout event rate: how many per second (an absolute value, good for spotting volume changes)
• Timeout share of all requests: a percentage (removes the “we are just busy” confounder)
• Is it still growing: deriv(), or a direct comparison of consecutive windows

Logs cannot answer any of these. Aggregates can.

3.2 A Recording Rule Set You Can Copy

Below is the set of recording rules we run in production, with business details stripped out. Evaluation interval 30 seconds, window 5 minutes:

groups:
  # tier one: turn the log event stream into low-cardinality state metrics
  - name: log_derived_state
    interval: 30s
    limit: 2000
    rules:
      # 1) timeout event rate: payment gateway timeouts per second
      - record: loki:payment_timeout:rate5m
        expr: |
          sum by (service) (
            rate({app=~"checkout|payment"} |= "PAYMENT_GATEWAY_TIMEOUT" [5m])
          )          

      # 2) total checkout request rate: the denominator
      - record: loki:checkout_requests:rate5m
        expr: |
          sum by (service) (
            rate({app="checkout"} |= "order_submit" [5m])
          )          

      # 3) timeout ratio: the number the alert should actually use
      - record: loki:payment_timeout:ratio5m
        expr: |
          loki:payment_timeout:rate5m
            /
          loki:checkout_requests:rate5m          

      # 4) error log ratio: a generic service health metric, also used by dashboards
      - record: loki:service_error:ratio5m
        expr: |
          sum by (service) (rate({app=~"checkout|payment"} |~ "(?i)\\b(error|exception|fatal)\\b" [5m]))
            /
          sum by (service) (rate({app=~"checkout|payment"} [5m]))          

Only rules 1 and 2 touch raw logs; rules 3 and 4 are pure arithmetic. That is the most direct cost-benefit of tiering: the expensive log query runs once per evaluation, and every downstream consumer reuses its result.

3.3 Label Convergence: Keep Only What You Would Slice By On-Call

The output of a recording rule is a time series, so cardinality is unavoidable. I have exactly one principle:

Labels are the dimensions you would actually slice and localize by at 3 a.m. If you cannot slice with it, or it cannot help you localize, it does not belong in a label.

A few common choices, judged against that principle:

One mistake shows up constantly: keeping pod in an alert rule’s labels. In Kubernetes, pod names carry random suffixes, so every deployment produces a brand-new batch of time series. The result is that the same incident looks like two different problems before and after a release, and alert history no longer lines up. Unless the alert genuinely needs instance precision (“this specific machine’s disk failed”), stop at service.

3.4 Before and After

Back to that night. The same failure mode — the payment gateway’s downstream pool saturated — observed with the old and new designs:

Two numbers in that table deserve their own paragraphs.

First, “time to first notification went from 5 seconds to 10 minutes” is the price we deliberately paid. Someone always asks: so the business stays broken for those ten minutes? No — those ten minutes are the alert engine’s confirmation window, not the detection window. Detection lives in dashboards and SLO burn-rate curves (Alerting on SLOs). An alert like “payment timeout ratio above 2% for ten minutes” is meant to catch failures that require a human decision, not every change a human might want to know about.

Second, “acknowledgement rate from 1 in 4 to nearly 100%” is the real prize. The cost of alert fatigue is not the number of interruptions; it is the conditioned reflex it trains — “these pages are probably not worth reading.” Industry surveys keep finding the same numbers: operations teams spend close to 40% of their working time analyzing alerts (When the Alert Storm Hits: How IT Operations Copes With Information Overload), and security operations centers receive thousands of alerts a day, of which analysts only manage to work through roughly a third, while more than 60% of teams admit to having ignored an alert that later turned out to be real (Why Nobody Reads Alerts After EDR Goes Live). Operations alerting runs on the same mechanism under a different name.

3.5 Recording Rules Can Blow Up Your Time Series Too

Tiering cures spam, but recording rules are in the business of “turning logs into time series,” so they move the cardinality problem from the log side to the metric side. Getting this wrong costs far more than spam.

In the first half of 2025, we hit it while reworking log alerting for a mobility platform. The first version of the recording rule looked like this:

- record: loki:order_timeout:rate5m
  expr: |
    sum by (service, pod, endpoint, tenant_id) (
      rate({app=~"order|dispatch"} |= "TIMEOUT" [5m])
    )    

The reasoning felt sound: if the point is to localize problems, then more dimensions must be better — service, instance, endpoint, tenant, all of them. Four hours after rollout:

• single rule evaluation went from roughly 400 ms to over 4 seconds, with occasional evaluation timeouts;
• each ruler replica’s WAL directory grew from a few hundred MB to over ten GB;
• Loki ingesters started showing visible memory churn, and write-path P99 latency went from 30 ms to 200 ms.

The root cause is label cardinality. A recording rule’s output is a time series, so new series ≈ number of rules × the product of each label’s distinct value count. In that environment:

service(12) × pod(4) × endpoint(37) × tenant_id(600)
  = 1,065,600 series (upper bound)

We actually landed 620,000 series in four hours — below the ceiling only because many dimension combinations had no data at all. For the ruler and Loki, that was already a disaster. After cutting the labels down to two dimensions, the same rule behaved like this:

My rule of thumb: if a single recording rule could theoretically produce more than 10,000 series, do not ship it. In a 20-service environment with 4 replicas each, by (service) gives you 20 series and by (service, pod) gives you 80 — both safe. The moment you add endpoint or tenant_id, you jump into five digits. Run the arithmetic against one day of historical data before rollout; it is far cheaper than firefighting afterwards.

People worry that converging labels means they cannot localize to a specific instance. That worry answers the wrong question: an alert label’s job is to point at the unit that needs action, not to carry every piece of context. The pod, endpoint, and tenant_id fields are still in the raw logs. Ship service and env in the alert, and the on-call engineer can open a dashboard or run one LogQL query for full context. Stuffing troubleshooting dimensions into alert labels makes the ruler hold an index on Grafana’s behalf — high cost, low benefit.

3.6 How Much Evaluation Cost Does Tiering Actually Save

“Tiering is cheaper” only counts if it comes with numbers. Here is a measured comparison from an environment ingesting about 120 GB/day with 2 ruler replicas (before: 22 single-tier alert rules; after: 8 recording rules + 14 alert rules):

Two numbers need explaining.

First, do not over-read the “only 27% fewer log queries per hour.” The real gain is not in that 27%. Before, those 1320 queries each scanned the full log stream; after, all 960 sit in the recording rules, and tier two only compares a few dozen floats. In other words, the expensive log aggregation went from scaling with rule count to scaling with recording-rule count — when we later grew alert rules from 14 to 40, log query volume did not move at all.

Second, panel latency and missed evaluations are what on-call engineers actually feel. In an on-call room with 30 concurrent users, every dashboard refresh used to trigger another round of log scanning; now those queries all hit precomputed metrics. There is a bonus effect people overlook: recording-rule output can be shipped to Prometheus or Mimir via remote_write, so building a cross-cluster view moves aggregates instead of log data.

4. Tier Two: Writing Alert Rules That Do Not Spam

4.1 for Is Your Only Noise Gate — Use All of It

for means the same thing in Prometheus and Loki: once the expression becomes true, the alert enters pending, and only after continuously satisfying it for the specified duration does it move to firing and actually send. A single failing evaluation resets it to inactive.

- alert: CheckoutPaymentTimeoutRatio
  expr: loki:payment_timeout:ratio5m{service="checkout"} > 0.02
  for: 10m          # only fire after the timeout ratio stays above 2% for 10 straight minutes
  labels:
    severity: critical
  annotations:
    summary: "Checkout payment timeout ratio is too high"
    description: "{{ $labels.service }} payment gateway timeout ratio is {{ $value | humanizePercentage }}, sustained for 10 minutes"

Here is the method I use to pick a value: look back at historical data for this failure mode, find the shortest window between “the business started taking damage” and “a human intervention would still help,” and take 1.5× that.

• Transient jitter (network blips, GC pauses, a single failed retry): start at for: 5m; these recover on their own within minutes.
• Capacity problems (connection pools, thread pools, disk, queue buildup): for: 10m. They do not heal themselves, so waiting ten minutes costs you no mitigation window while filtering out momentary traffic spikes.
• Events (credential leaks, config changes, certificate anomalies): for can be short or even 0m, but the rule’s natural firing frequency must be genuinely low.

4.2 Do Not Use > 0 as a Threshold

> 0 is close to an anti-pattern in an alert rule, because it mistakes existence for severity. Three replacements:

• Ratio: errors / total > 2%. Adapts to traffic automatically, so a big promotion will not trigger it just because volume rose.
• Rate: errors per second > 5. Good when errors below a certain level are just background noise, and far more precise than > 0.
• Rhythm-aware rate: rate(...[5m]) > 2 * rate(...[1h] offset ...). Fits “any sudden change is abnormal,” but it is hard to read and easy to confuse yourself with; I would not put it on the pager.

The only legitimate use of > 0 is an event-style alert, and even then two conditions must hold: the rule’s candidate firing frequency is naturally low (leaked credentials, say), and it is paired with enough for to deduplicate. Otherwise, convert it to a ratio.

4.3 limit: It Saves You, and It Silently Eats Your Alerts

Loki lets you set limit on a rule group to cap how many alerts or samples that group can produce. It deserves its own section because the failure mode is counterintuitive.

The documentation is explicit: once the limit is exceeded, all samples produced by that group’s recording rules are dropped; for alert rules, every alert in the rule — active, pending, and inactive — is cleared, the event is recorded as an error, and the rule’s health is set to err. The default is 0, meaning unlimited.

In plain terms, limit is a circuit breaker. It protects the ruler from a runaway rule (say, one whose label cardinality exploded), but when it trips, the outcome is not “a few fewer alerts” — it is “every alert in this group disappears.” If all you monitor is whether the alert channel is alive, what you observe is “production was on fire and no alert arrived” — considerably more dangerous than spam.

How I use it:

• Give each high-cardinality-risk rule its own group with a limit slightly above the normal peak (peak of 5 alerts, limit: 50).
• Bring rule health into meta-monitoring: any rule group with health=err should raise an alert. That alert should reach the platform owner, not the on-call rota.
• Never set one limit across a group of 20 rules. That puts a shared circuit breaker on the whole group.

4.4 Multi-Tenancy and Routing: alertmanager_url Is Global

This is the most frequently misread part of a Loki deployment: the ruler’s alertmanager_url is a single global setting — one Loki cluster points at exactly one Alertmanager. It cannot route per tenant, and you cannot point one rule at a different destination.

The only workable pattern in a multi-tenant setup is: send every tenant’s alerts to the same Alertmanager and do the routing and isolation there with tenant labels. In other words, move the “who gets notified” decision out of Loki and into Alertmanager:

# Alertmanager side: split traffic using the tenant / labels that Loki attaches
route:
  receiver: default-oncall
  group_by: ["alertname", "service", "tenant"]
  routes:
    - matchers: ['tenant="tenant-a"']
      receiver: tenant-a-oncall
    - matchers: ['severity="critical"']
      receiver: pagerduty

One more warning: Loki’s enable_alertmanager_v2 has been true by default since 3.2.0. If your Alertmanager is still on the v1 API, you will trip over this the next time you upgrade Loki. For the full routing and inhibition syntax on the Alertmanager side, see Related: Alertmanager Routing and Silencing Strategies.

4.5 A Complete, Copy-Ready Two-Tier Rule File

Putting it all together — this is a rule file you can drop straight into storage (tier one and tier two live in the same file because one ruler evaluates both):

groups:
  # ============ tier one: recording rules, producing state metrics ============
  - name: log_derived_state
    interval: 30s
    rules:
      - record: loki:payment_timeout:rate5m
        expr: |
          sum by (service) (
            rate({app=~"checkout|payment"} |= "PAYMENT_GATEWAY_TIMEOUT" [5m])
          )          
      - record: loki:checkout_requests:rate5m
        expr: |
          sum by (service) (rate({app="checkout"} |= "order_submit" [5m]))          
      - record: loki:payment_timeout:ratio5m
        expr: |
          loki:payment_timeout:rate5m / loki:checkout_requests:rate5m          

  # ============ tier two: alert rules, firing only when state persists ============
  - name: checkout_alerts
    interval: 30s
    limit: 50
    rules:
      # state-style: use a ratio, pair it with for
      - alert: CheckoutPaymentTimeoutRatio
        expr: loki:payment_timeout:ratio5m{service="checkout"} > 0.02
        for: 10m
        labels:
          severity: critical
          team: checkout
        annotations:
          summary: "Checkout payment timeout ratio is too high"
          description: "{{ $labels.service }} timeout ratio {{ $value | humanizePercentage }}, sustained for 10 minutes"

      # event-style: > 0 is acceptable, but for is mandatory and the rule must be rare
      - alert: CredentialsLeakedInLog
        expr: |
          sum by (service) (count_over_time({namespace="prod"} |~ "http(s?)://(\\w+):(\\w+)@" [5m]) > 0)          
        for: 10m
        labels:
          severity: warning
          team: platform
        annotations:
          summary: "Plaintext credentials found in production logs"
          description: "{{ $labels.service }} logs contain suspected plaintext credentials; confirm and rotate immediately"

Validate the syntax before it lands. With lokitool available next to Loki, you can check and diff a local rules directory:

# format and align PromQL/LogQL expressions
lokitool rules format ./rules/log-alerting.yaml

# diff local rules against the ruleset currently enforced in Loki (specify a tenant ID in multi-tenant mode)
lokitool rules diff --rule-dirs=./rules --id=fake

# make the remote match local, creating/updating/deleting the differences
lokitool rules sync --rule-dirs=./rules --id=fake

Note the --id=fake: that is the default tenant name in single-tenant mode (auth_enabled: false). In a multi-tenant environment, use a real tenant ID or pass it through the LOKI_TENANT_ID environment variable. Also, before Loki 3.1 this tool was called cortextool; the arguments are largely the same.

5. Four Production Traps in the Ruler

This section is the one I most want people to read. The first four sections cover how to write rules; this one covers why a rule you wrote does not take effect, and when it goes silent without telling anyone. These facts are scattered across different pages of the official documentation, and they are hard to notice before something breaks.

5.1 Trap 1: The Ruler Does Not Evaluate Anything During WAL Replay

Loki’s ruler keeps a per-tenant write-ahead log (WAL) so it can lazily reload past evaluation results when scheduling or sharding changes. On startup the ruler loads and replays those WALs one by one — and the documentation is explicit: during replay, the ruler does not evaluate any rules, alert rules included.

Which means: while the ruler is rolling-restarting or upgrading, every log alert is silent. Nothing errors, nothing alerts; nothing is sent.

Three ways to respond:

1. Alert on ruler liveness through an independent path. Do not use the log alerting chain to monitor itself — when the ruler is down it cannot emit anything. This alert has to come from the metrics path (Prometheus scraping the ruler’s /metrics).
2. Keep rule volume in check to shorten replay. Replay time scales with the number of rules in the WAL and the size of each group; the docs also recommend keeping individual groups small.
3. Schedule ruler rolling upgrades during low-traffic hours, and do not overlap them with ingester upgrades. Stacked restart windows mean you lose log ingestion and alert evaluation in the same slice of time.

5.2 Trap 2: A Ring Change Re-Shards Everything, With No Handover

Once a single ruler cannot keep up, you shard it (enable_sharding: true, which requires a ring). There is a detail here that differs sharply from ingesters:

Ingesters hand off responsibility when they scale up or down; rulers do not. As the documentation puts it, all rules are re-sharded randomly every time a ruler is added to or removed from the ring.

The consequence: scale one ruler replica and the entire rule set gets redistributed. Evaluation cadence jitters during that window — some rules produce results a few cycles late, others may briefly be evaluated by two instances at once. For a 30-second or 1-minute alert rule, that jitter is quite likely to break the continuous counter behind for, which shows up as “the failure happened, but the alert was slow to arrive.”

My approach is unglamorous:

• Scale the ruler in its own window, never alongside an application release. We once bumped rulers from 2 to 3 replicas during a routine release, and that night a low-frequency alert’s for counter got interrupted and it fired more than an hour later than it should have. Small incident, useful lesson.
• Use a stable kvstore backend for the ring (Consul/etcd), not inmemory. inmemory is fine in a single-process demo, but with multiple replicas each ruler keeps its own ring, sharding decisions disagree, and rules get double-evaluated or skipped.
• Do not set a group’s interval too close to for’s granularity. interval: 30s with for: 1m tolerates almost no evaluation jitter; pushing for to five minutes or more is far more stable.

5.3 Trap 3: Local Storage Is Read-Only, Has No Globs, and Ignores Subdirectories

Mounting rules from a ConfigMap is a common choice (ruler.storage.type: local). Three behaviors you must know:

First, the directory layout must be flat. The local backend reads all non-hidden files in the tenant directory but does not support wildcards and ignores subdirectories. This is where it differs most from Prometheus — Prometheus’s rule_files accepts globs like /etc/prometheus/rules/*.yaml, so people organize directories out of habit:

/tmp/loki/rules/fake/
├── alerts/
│   └── checkout.yaml     ← ignored, never loaded
└── recording/
    └── state.yaml        ← ignored

The symptom is “the rule files are definitely mounted, but Loki has zero rules and no obvious error.” The correct layout is flat:

/tmp/loki/rules/fake/
├── checkout-alerts.yaml
├── log-derived-state.yaml

Second, the tenant directory name must match the tenant ID. In single-tenant mode (auth_enabled: false) the default tenant is fake; if you customize it at startup with -auth.no-auth-tenant, the directory name must change with it. If the tenant directory is missing, the ruler does not exit — it logs an error on every rule sync (unable to list rules) and keeps running while evaluating nothing. That is the nastiest failure of all: healthy process, listening port, zero alerts.

Third, do not confuse rule_path with the storage directory. ruler.storage.local.directory is where the ruler reads rule files; rule_path (default /rules) is the scratch directory where it writes temporary files during evaluation. They serve different purposes — on a read-only filesystem you must point rule_path somewhere writable, or rule evaluation fails when it tries to write.

For that third point I added a very cheap safety net: monitor how many rule groups are currently loaded, and monitor rule health. If the group count drops from 12 to 0, or any group reports health=err, alert immediately. That rule has saved us more than once.

5.4 Trap 4: Local Evaluation Can Disagree With What Grafana Shows You

Rules can be evaluated in two modes. In the default local evaluation mode, the ruler carries its own querier and bypasses the query frontend and query scheduler. That means it runs under the same constraints as a normal query — but in large deployments, the documentation states something critical: when rules are evaluated locally, results may differ or be incomplete compared with what you see in Grafana.

The reason is not hard to grasp: in a large cluster the same log data is spread across many ingesters. Local evaluation goes through the ruler’s own query path, while Grafana goes through the full query-frontend chain (query splitting, shard aggregation, result merging). The two paths have different semantic boundaries, so the results naturally diverge.

The fix is to switch to remote rule evaluation, which routes rule evaluation through the query frontend. The cost is that evaluation traffic enters the query path, so you need to reserve capacity for queries.

My judgment is blunt:

• Single cluster, simple rules (a handful of rate/ratio expressions), log volume below roughly a hundred GB per day: local evaluation is fine. Do not adopt remote evaluation yet; skip the extra capacity planning.
• Multi-cluster federation, rules full of by (...) aggregations, or you have ever seen “the same number rendered differently on two dashboards”: switch to remote evaluation. This is not a performance optimization, it is a correctness fix.

5.5 Make Rules Verifiable: Treat Them as Code

Three of the four traps above share a signature — “the config looks like it took effect, but it did not.” That class of problem cannot be caught by human review. Only tooling catches it.

A rule file is code, and I manage it in three layers:

Layer one: syntax validation. promtool check rules from the Prometheus ecosystem validates rule file syntax (Loki rule files are Prometheus-compatible, only the expressions are LogQL); Loki’s own lokitool rules format and diff handle formatting and difference checks.

Layer two: a diff gate in CI. Grafana’s cortex-rules-action supports specifying Loki as the backend, so a pipeline can lint and sync:

- name: Lint Loki rules
  uses: grafana/cortex-rules-action@master
  env:
    ACTION: check
    RULES_DIR: rules/log-alerting
    BACKEND: loki

- name: Deploy rules to Loki
  uses: grafana/cortex-rules-action@master
  env:
    ACTION: sync
    RULES_DIR: rules/log-alerting
    BACKEND: loki
    CORTEX_ADDRESS: http://loki-gateway.monitoring.svc:80
    CORTEX_TENANT_ID: fake

Making ACTION: check a required merge check blocks problems like “the expression is malformed,” “the label is misspelled,” and “the metric referenced does not exist” before deployment. Here is a trap I walked into: RULES_DIR accepts multiple comma-separated directories, but do not nest subdirectories inside them — for the reason in 5.3, the local backend ignores subdirectories, so CI passes and runtime still loads nothing.

Layer three: declarative management with Terraform. A community Terraform provider for Loki models rule groups as resources (loki_rule_group_alerting / loki_rule_group_recording), and terraform plan shows you exactly which rules will be added or removed before you apply. If you want rules to go through a real code review process, this is the smoothest path.

5.6 A Troubleshooting Table for the Four Traps

What the four traps share is that none of them produces an error. This table is the quick-reference version I keep in our on-call handbook, organized as symptom → confirmation → fix:

Two commands belong in an on-call handbook for the “confirmation” column:

# 1) print the ruleset currently enforced in the remote ruler (address flag per lokitool rules print -h)
LOKI_TENANT_ID=fake lokitool rules print

# 2) confirm which metric names the ruler actually exposes before writing meta-alerts
curl -s http://loki-ruler:3100/metrics | grep -i 'ruler' | cut -d'{' -f1 | sort -u | head -30

The second command is a habit I keep pushing: never guess metric names from memory. Ruler metric names have changed across major versions; pull the real ones out of /metrics before writing a meta-alert, and you avoid shipping an alert that can never fire.

And one safety net that costs almost nothing and should always exist (it relies on up, which every Prometheus target has, rather than any Loki-specific metric):

groups:
  - name: loki_ruler_meta
    rules:
      - alert: LokiRulerTargetDown
        expr: up{job="loki-ruler"} == 0
        for: 3m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: "Loki ruler is unavailable; all log alerts are silent"
          description: "The ruler evaluates no rules while it is down. Confirm immediately."

This alert must be emitted by Prometheus (the metrics path) and never by Loki’s own alerting chain — the ruler is the executor of that chain, so if it stops, nobody can report the outage.

6. Rollout Order: 5 Checkpoints

If you are applying this to an environment that already has a pile of rules, sequence matters more than content. This is my order:

Step 1: classify existing log alerts as event-style or state-style. It is paper work, but it immediately shows you the distribution of the problem. In an environment that has been running for two or three years, state-style log alerts are typically more than 70% of the total, and every one of them is a potential spam source.

Step 2: convert every state-style rule to two tiers. Write the recording rule first, then swap the alert rule’s expr for a test against the recording rule’s output. Alert volume usually drops by an order of magnitude at this point. During the transition you can keep the old rules and use different severity labels to compare side by side; delete the old ones only once the new ones prove more accurate.

Step 3: replace > 0 thresholds with ratios, and raise for to five minutes or more. These two changes have the best effort-to-reward ratio of the whole project: the work is editing a few numbers, and the effect is a sharp drop in night interruptions.

Step 4: fill in the meta-monitoring. At minimum three things: ruler process liveness (sourced from the metrics path), the number of loaded rule groups, and rule health. These backstop every trap above — without them, everything you did can silently stop working after some restart.

Step 5: make alert metrics a fixed field in every postmortem. Our template now always captures four numbers: alerts generated by this incident, share that a human marked useful, night interruptions, and time to first notification. Without those four, “is our alerting getting better or worse” stays a matter of opinion.

Two more things to settle before you change anything — capacity and rollback:

• Capacity estimate: recording rules add series. New series ≈ number of recording rules × series produced per rule. In our experience, 10 recording rules with labels converged to service + env add a few hundred to a couple thousand series, which barely touches Loki’s own storage. Let labels get out of hand (bring in pod or tenant) and that multiplies by dozens. Compute the new series count from one day of data before rollout.
• Rollback plan: rule files are declarative, so rolling back means re-applying the previous file (lokitool rules sync or Terraform apply). Which means rule files must live in Git, and you must be able to answer “what did the rule file look like at 3 a.m. last Wednesday?” I keep a dedicated directory of rollback snapshots and export the full live ruleset before every rollout.

6.1 Rewriting That One Bad Rule: Three Iterations

The previous sections were methodology. This one rewrites the spam rule from the opening, so you can see exactly what each step buys.

v1, the original:

- alert: CheckoutPaymentTimeout
  expr: count_over_time({app="checkout"} |= "PAYMENT_GATEWAY_TIMEOUT" [1m]) > 0
  labels:
    severity: critical

v2, replacing “instantaneous existence” with a rate and adding for:

- alert: CheckoutPaymentTimeoutRate
  expr: |
    sum by (service) (
      rate({app="checkout"} |= "PAYMENT_GATEWAY_TIMEOUT" [5m])
    ) > 0.5    
  for: 5m
  labels:
    severity: critical

v3, replacing the rate with a ratio, lengthening the window, and splitting into two tiers:

# tier one: recording rule
- record: loki:payment_timeout:ratio5m
  expr: |
    sum by (service) (rate({app=~"checkout|payment"} |= "PAYMENT_GATEWAY_TIMEOUT" [5m]))
      /
    sum by (service) (rate({app="checkout"} |= "order_submit" [5m]))    

# tier two: alert rule
- alert: CheckoutPaymentTimeoutRatio
  expr: loki:payment_timeout:ratio5m{service="checkout"} > 0.02
  for: 10m

How the three versions behave during the same 47-minute incident:

The v2 → v3 step is the one people skip, and it is the dividing line between “workable” and “trustworthy.” Absolute thresholds fail in both directions at once: triple your traffic and the same threshold is trivially breached; drop traffic to a third and genuinely severe errors no longer reach it. Once the threshold expresses “the failure rate of this path,” it is decoupled from volume and can actually be stable.

How do you verify the result? I replay one day of historical data: plot the new ratio metric in Grafana, overlay the old rule’s firing windows, and look at when the new rule would enter pending and when it would move to firing. “No alert since we shipped it” is not verification — it may simply mean you have not hit that failure mode yet.

6.2 Changing the On-Call Contract

Once the rules change, a notification that still says “checkout payment timeout ratio is too high” leaves the on-call engineer opening Grafana anyway to decide what to do. So the last step is fixing notification content and severity:

annotations:
  summary: "Checkout payment timeout ratio is 3.4%, sustained for 11 minutes"
  description: "Service: {{ $labels.service }} | ratio: {{ $value | humanizePercentage }} | duration: 11m"

Three points:

• The notification must carry the value and the duration. With “3.4%, 11 minutes,” the on-call engineer can decide immediately whether to act; with “ratio is too high,” you have pushed that judgment onto the person you just woke up.
• severity must be bound to an expected response, not filled in freely per rule.

The severity table has to hold for every service. Otherwise the same 3% ratio is a warning for service A and critical for service B, and the on-call engineer cannot tell from the notification alone whether it justifies getting out of bed.

• Hand over pending alerts, not just firing ones. A pending alert means “the condition is met and the for timer is running” — that is where advanced warning actually lives. Our shift-handover template now has a dedicated line for “current pending alerts,” and it is far more useful than handing over alerts that are already firing.

7. Seven Anti-Patterns I Have Watched Run in Production

One last list. I have seen all seven of these in real environments, and they share a signature: they read fine until something breaks.

A sentence on each.

Number 1 was taken apart in section 2. The actionable test: check whether the expression outputs a boolean. Any expression emitting true/false needs for, and you must ask yourself whether the condition can stay true indefinitely.

Number 2 is about regex granularity. (?i)error will match error_code=0, NoError, errorRate:0 — all perfectly normal. Alert regexes need field boundaries: parse with | logfmt first, then filter on the level field equaling error. That is far more reliable than matching a bare word.

Number 3 is the most common “obvious” fix. Moving the window from one minute to one hour does lower the notification rate, but it delays alerts by an hour and touches none of the real problems — cardinality, evaluation cost. The right noise gates are for and ratios, not window length. A window makes rates smoother; it does not make alerts less frequent.

Number 4 came up in 3.3 and 3.5. Here is the executable check: before rollout, extract every by (...) in the rule file and ask, one by one, “do this dimension’s values change? Do they change with each release?”

Number 5 is the most dangerous of the seven because its failure is silent. The intent behind limit is to protect the ruler, but when the breaker trips the result is not “a few fewer alerts” — it is “every alert in this group is gone.” Without independent meta-monitoring, what you observe is “production was on fire and no alert came.”

Number 6 is self-reference. The liveness of the log alerting chain must be monitored by a path that does not depend on it, and that path can only be Prometheus scraping /metrics. Our internal standard: meta-monitoring for any alerting chain may never hang off the chain being monitored.

Number 7 looks like process hygiene but is really a capability gap. If rule files are not in Git, you have no rollback and cannot answer “which version was live at 3 a.m. last Wednesday” — and the first thing you do when investigating a cross-week alert anomaly is compare those two versions. The minimum: rules in Git, ACTION: check in CI, rollout via lokitool rules sync, and the change history preserved in commits.

Summary

Log alert spam looks like a threshold problem. It is a semantics problem.

Logs are an event stream, so they can only answer “did it happen”; on-call engineers need “is it broken right now, and how badly.” Wire a signal that can only express existence directly into a pager and all that is left is spam. The fix is to split the job: recording rules compress the event stream into state metrics, alert rules decide whether that state persists.

Looking back at those 400 alerts, three lessons are worth passing on:

First, the primary question for any log alert is “am I judging existence or state?” For existence, > 0 plus a sensible for is enough. For state, write the recording rule first. Asking that question before writing a rule saves a large amount of rework later.

Second, the goal of noise reduction is not fewer alerts — it is that every alert deserves attention. Cutting night interruptions from 11 to 1 is not the finish line; the finish line is that the surviving alert maps to a failure a human genuinely needed to handle. That is why we track the “marked useful” ratio, and why we only consider the job done when it approaches 100%.

Third, the fragility of the log alerting chain is badly underrated. No evaluation during WAL replay, re-sharding on ring changes, local storage ignoring subdirectories, silent operation when the tenant directory is missing — all four have the same shape: “it looks like it is working, but it is not.” Log alerting is powerful, but it must be backstopped by meta-monitoring from the metrics path. That is not optional.

One closing position: log alerting does not replace metric alerting; it fills the gaps. Wherever you can collect a metric, use the metric. Save logs for what metrics cannot reach — black-box components, event-style anomalies, high-cardinality dimensions. Draw that boundary clearly and the rule count halves while every remaining rule carries twice the weight.

References & Acknowledgments

The following sources were consulted while writing this article. Thanks to their authors.

1. Alerting and recording rules | Grafana Loki documentation — Grafana Labs. Ruler responsibilities, alert and recording rule examples, the three use cases (black box monitoring / event alerting / high-cardinality sources), ruler storage and directory layout, per-group limit behavior, alertmanager_url and enable_alertmanager_v2, ruler sharding, and rules-as-code (lokitool / Terraform / cortex-rules-action)
2. Alerting rules | Prometheus documentation — Prometheus project. Alert rule syntax and the pending/firing semantics of for
3. Monitoring Distributed Systems — Google SRE Book (Betsy Beyer et al.). Alerts should target symptoms rather than causes, and every page should be actionable
4. Recording rules | Prometheus documentation — Prometheus project. The role of recording rules and precomputed values, plus the corresponding per-group limit behavior
5. Alerting on SLOs — Google SRE Workbook. Judging failures that need human intervention by burn rate rather than a single threshold; the basis for this article’s claim that a 10-minute confirmation window is not detection latency
6. When the Alert Storm Hits: How IT Operations Copes With Information Overload — Tencent Cloud Developer Community. The share of operations time spent analyzing alerts and the hidden cost of alert fatigue, supporting the argument that acknowledgement rate matters more than interruption count
7. Why Nobody Reads Alerts After EDR Goes Live: Causes of Security Alert Fatigue and a Four-Level Noise Reduction Practice — Tencent Cloud Developer Community. Daily alert volume and the share analysts actually triage, plus the proportion of teams that ignored real alerts; used to show that alert fatigue is a general mechanism
8. Recording rules | Grafana Loki documentation — Grafana Labs. The ruler’s built-in querier evaluation path, per-tenant WAL loading and replay behavior (no rules are evaluated during replay), and rule group sizing advice
9. Loki scalability | Grafana Loki documentation — Grafana Labs. Role separation in large deployments and remote rule evaluation (local evaluation results may be inconsistent)

Thanks to Grafana Labs and the Prometheus project for documenting the ruler’s behavior in such detail, and to the Google SRE team for the classic argument that alerts must be actionable — it comes closer to the root of the problem than any noise-reduction trick.