Don't Wait for the Crash to Check Monitoring: A Practical Methodology for Linux Performance Baselines and Anomaly Detection

Overview

You’ve probably been here: woken up at 3 AM by a pager alert, scrambled to check Grafana, and found CPU usage spiked to 85%. You panic, investigate for half an hour, and finally realize — this database server runs a scheduled backup job every day from 2:50 to 3:10 AM. CPU is supposed to be at this level.

What went wrong? You didn’t know what “normal” looks like for this machine. Without a baseline, you can’t tell whether 85% is high or low.

A performance baseline solves exactly this problem: when the system is healthy, record key metrics to build a “health profile.” Later, when something looks off, compare current data against the baseline — real failure or normal fluctuation becomes obvious at a glance.

This article covers how to build a production-grade Linux performance baseline system from scratch. Not vague “monitoring is important” platitudes — every step gives you configs and code you can use immediately, from methodology selection to metric collection, baseline modeling, anomaly detection, and automation.

Why Static Thresholds Don’t Work

Most teams configure alerts like this: CPU > 80% alerts, memory > 90% alerts, disk > 85% alerts. These are static thresholds — a number picked by gut feeling, applied to every machine.

Static thresholds have three fatal flaws:

First, “normal” varies by workload and time. A batch processing server hitting 95% CPU during work hours is fine. An API gateway sustaining 70% CPU deserves scrutiny. Apply the same 80% threshold to both, and the former generates constant false alarms while the latter might miss a real problem.

Second, absolute values hide trends. A machine’s CPU creeps from 30% to 60% over two weeks. Static thresholds don’t catch it — it hasn’t hit 80%. But this slow rise often signals a memory leak or connection buildup. By the time it reaches 80%, the system has already crashed.

Third, they can’t detect “abnormally good.” A core service’s QPS suddenly drops from 5,000 to 200, and CPU follows it down to 10%. Static thresholds think everything is fine — CPU is low. But the upstream might be down, and traffic isn’t even reaching this service.

The baseline approach is straightforward: don’t set fixed thresholds. Let the system learn what “normal” means. Only when actual behavior deviates from the learned “normal pattern” do you trigger an alert.

The USE Method: The Backbone of Baseline Collection

Before building a baseline, you need to know what metrics to collect. Grabbing 200 metrics where 80% go unread is waste.

Brendan Gregg’s USE Method (Utilization, Saturation, Errors) is the most practical resource-checking framework. Core idea: for each resource type, check utilization, saturation, and errors.

Resource	Utilization	Saturation	Errors
CPU	CPU usage percentage	Run queue length, load average	Abnormal context switch spikes
Memory	Used memory percentage	Page in/out rate, OOM trigger count	OOM kill events
Disk I/O	%util (device busy percentage)	iostat await, avgqu-sz	Disk I/O error counts
Network	Bandwidth utilization	NIC dropped packets	NIC error packets
Filesystem	Inode usage percentage	Queue wait time	Filesystem errors (ext4 errors)

The USE Method’s strength is simplicity. Brendan himself said: “USE solves about 80% of server issues with 5% of the effort.” You don’t need flame graphs or eBPF right away. Cover the four major resources across U, S, and E, and you’ll handle 90% of daily scenarios.

Baseline Metric Checklist

Based on the USE Method, here are the core metrics you must collect for baselining. Don’t overdo it — start with these:

CPU subsystem:

cpu_usage_user / cpu_usage_system / cpu_usage_idle — User, kernel, and idle time percentages
load_avg_1 / load_avg_5 / load_avg_15 — 1/5/15-minute average load
context_switches — Context switches per second
run_queue_length — Run queue length

Memory subsystem:

mem_available_pct — Available memory percentage (watch available, not free)
swap_used_pct — Swap usage percentage
page_in / page_out — Pages swapped in/out per second
oom_kill_count — Cumulative OOM kill count

Disk I/O subsystem:

disk_util — %util per disk
disk_await — Average I/O request wait time (ms)
disk_iops_read / disk_iops_write — Read/write IOPS
disk_avgqu_sz — Average queue length

Network subsystem:

net_rx_bytes / net_tx_bytes — Bytes received/transmitted per second
net_rx_drop / net_tx_drop — NIC dropped packet counts
net_rx_errs / net_tx_errs — NIC error packet counts
tcp_retransmit_rate — TCP retransmission rate

Collection Tool Selection: From Manual to Automated

The core tension in baseline collection: you need long-term, continuous, low-overhead data collection. Running top manually isn’t a baseline — that’s a glance.

Here are three layers of collection tools.

Layer 1: Lightweight Kernel-Level Collection — sar

sar (System Activity Reporter) comes with the sysstat package and is available on virtually every Linux distribution. It runs a cron job every 10 minutes to collect system metrics into binary files, and you can query historical data at any time.

# Install sysstat (includes sar)
# CentOS/RHEL
yum install -y sysstat
# Ubuntu/Debian  
apt install -y sysstat

# Enable data collection (edit /etc/default/sysstat or /etc/sysconfig/sysstat)
# Ubuntu/Debian:
sed -i 's/ENABLED="false"/ENABLED="true"/' /etc/default/sysstat
systemctl restart sysstat
systemctl enable sysstat

# CentOS/RHEL:
sed -i 's/false/true/' /etc/sysconfig/sysstat
systemctl restart sysstat
systemctl enable sysstat

sar data is stored in /var/log/sa/, rotated daily (sa28 means the 28th’s data). You can look back at any day’s historical performance:

# View today's CPU usage history
sar -u

# View August 28's memory usage history
sar -r -f /var/log/sa/sa28

# View disk I/O history
sar -d -p

# View network traffic history
sar -n DEV

# Collect every 1 second for 60 seconds (real-time mode)
sar -u 1 60

sar is extremely lightweight, has zero dependencies, and supports historical lookback. Its downside: the default 10-minute interval is too coarse, and it only covers basic metrics without process-level data.

Layer 2: Custom Collection Scripts — collectd / In-House Scripts

When you need finer granularity (every minute) or metrics that sar doesn’t cover (like TCP retransmission rate, process-level data), custom collection scripts offer more flexibility.

Here’s a lightweight baseline collection script written in Bash. Run it every minute via cron, output JSON for easy downstream ingestion:

#!/bin/bash
# perf-baseline-collector.sh
# Collects system performance baseline data every minute, outputs JSON

TIMESTAMP=$(date '+%Y-%m-%dT%H:%M:%S%z')
HOSTNAME=$(hostname)

# --- CPU metrics ---
CPU_LINE=$(cat /proc/stat | grep '^cpu ' | awk '{print $2,$3,$4,$5,$6,$7,$8}')
read USER NICE SYSTEM IDLE IOWAIT IRQ SOFTIRQ <<< "$CPU_LINE"
TOTAL=$((USER + NICE + SYSTEM + IDLE + IOWAIT + IRQ + SOFTIRQ))
CPU_USAGE=$(awk "BEGIN {printf \"%.2f\", ($TOTAL - $IDLE) / $TOTAL * 100}")
LOAD_AVG=$(cut -d' ' -f1-3 /proc/loadavg)

# Run queue length
RUNNABLE=$(grep -c 'R' /proc/*/stat 2>/dev/null || echo 0)

# Context switch count
CTXT=$(grep ctxt /proc/stat | awk '{print $2}')

# --- Memory metrics ---
MEMINFO=$(cat /proc/meminfo)
MEM_TOTAL=$(echo "$MEMINFO" | grep MemTotal | awk '{print $2}')
MEM_AVAIL=$(echo "$MEMINFO" | grep MemAvailable | awk '{print $2}')
MEM_AVAIL_PCT=$(awk "BEGIN {printf \"%.2f\", $MEM_AVAIL / $MEM_TOTAL * 100}")
SWAP_TOTAL=$(echo "$MEMINFO" | grep SwapTotal | awk '{print $2}')
SWAP_FREE=$(echo "$MEMINFO" | grep SwapFree | awk '{print $2}')
SWAP_USED_PCT=0
if [ "$SWAP_TOTAL" -gt 0 ]; then
    SWAP_USED_PCT=$(awk "BEGIN {printf \"%.2f\", ($SWAP_TOTAL - $SWAP_FREE) / $SWAP_TOTAL * 100}")
fi

# Page statistics
PAGE_IN=$(grep pgpgin /proc/vmstat | awk '{print $2}')
PAGE_OUT=$(grep pgpgout /proc/vmstat | awk '{print $2}')

# --- Disk I/O metrics ---
# Get first non-loop device
DISK_DEV=$(ls /sys/block/ | grep -v 'loop\|ram\|sr' | head -1)
DISK_STATS=$(cat /sys/block/$DISK_DEV/stat)
read DISK_IOS_READ DISK_SECT_READ DISK_IOS_WRITE DISK_SECT_WRITE DISK_TICKS <<< "$DISK_STATS"

# --- Network metrics ---
NET_DEV=$(ip route get 8.8.8.8 2>/dev/null | grep -oP 'dev \K\S+' || echo "eth0")
NET_STATS=$(cat /proc/net/dev | grep "$NET_DEV:")
read RX_BYTES RX_PACKETS RX_ERRS RX_DROP _ _ _ _ TX_BYTES TX_PACKETS TX_ERRS TX_DROP <<< "$NET_STATS"

# TCP retransmission statistics
TCP_RETRANS=$(grep -c retrans /proc/net/snmp 2>/dev/null || echo 0)

# --- Output JSON ---
cat <<EOF
{
  "timestamp": "$TIMESTAMP",
  "hostname": "$HOSTNAME",
  "cpu": {
    "usage_pct": $CPU_USAGE,
    "load_avg": "$(echo $LOAD_AVG | tr ' ' ',')",
    "runnable_tasks": $RUNNABLE,
    "ctxt_per_sec": $CTXT
  },
  "memory": {
    "available_pct": $MEM_AVAIL_PCT,
    "swap_used_pct": $SWAP_USED_PCT,
    "page_in": $PAGE_IN,
    "page_out": $PAGE_OUT
  },
  "disk": {
    "device": "$DISK_DEV",
    "io_read_ops": $DISK_IOS_READ,
    "io_write_ops": $DISK_IOS_WRITE
  },
  "network": {
    "interface": "$NET_DEV",
    "rx_bytes": $RX_BYTES,
    "tx_bytes": $TX_BYTES,
    "rx_drop": $RX_DROP,
    "tx_drop": $TX_DROP,
    "rx_errs": $RX_ERRS,
    "tx_errs": $TX_ERRS,
    "tcp_retrans": $TCP_RETRANS
  }
}
EOF

This script reads exclusively from /proc and /sys, has zero external dependencies, and adds negligible overhead (single execution < 10ms). Add it to cron every minute:

# crontab: collect every minute
* * * * * /opt/scripts/perf-baseline-collector.sh >> /var/log/perf-baseline/$(date +\%Y\%m\%d).jsonl

Layer 3: Time-Series Database + Visualization

Collected data ultimately needs to land in a time-series database for long-term storage, aggregation analysis, and anomaly detection. Two recommended approaches:

Option A: Prometheus + node_exporter

This is the most mainstream approach. node_exporter already covers CPU, memory, disk, and network basics — no custom scripts needed:

# node_exporter startup flags (recommend enabling textfile collector for custom metrics)
node_exporter --collector.textfile.directory=/var/node_exporter/textfile \
              --collector.cpu \
              --collector.meminfo \
              --collector.diskstats \
              --collector.netdev \
              --collector.filesystem \
              --collector.tcpstat \
              --no-collector.infiniband \
              --no-collector.ipvs

Prometheus scrape config:

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']
    scrape_interval: 60s      # 60-second granularity is sufficient for baselines
    scrape_timeout: 10s

Option B: sar Data + Python Script to Database

If your environment doesn’t allow Prometheus (e.g., strictly isolated compliance environments), you can parse sar’s binary data with Python and push to InfluxDB or save as CSV:

#!/usr/bin/env python3
"""
Export sar historical data to CSV for baseline analysis and anomaly detection modeling.
Dependency: sadf (included with sysstat)
"""
import subprocess
import csv
import sys
from datetime import datetime

def export_sar_to_csv(date_str, output_file):
    """Export sar data for a specific date as CSV"""
    cmd = [
        'sadf', '-d', '--', '-u', '-r', '-d', '-n', 'DEV',
        f'/var/log/sa/sa{date_str}'
    ]
    result = subprocess.run(cmd, capture_output=True, text=True)
    
    if result.returncode != 0:
        print(f"Error reading sar data for {date_str}: {result.stderr}")
        return
    
    with open(output_file, 'w', newline='') as f:
        reader = csv.reader(result.stdout.strip().split('\n'), delimiter=';')
        writer = csv.writer(f)
        for row in reader:
            if row and not row[0].startswith('#'):
                writer.writerow(row)
    
    print(f"Exported {date_str} sar data to {output_file}")

# Batch export last 30 days
if __name__ == '__main__':
    from datetime import datetime, timedelta
    
    end_date = datetime.now()
    start_date = end_date - timedelta(days=30)
    
    current = start_date
    while current <= end_date:
        day_str = current.strftime('%d')
        output = f'/var/log/perf-baseline/sar-{current.strftime("%Y%m%d")}.csv'
        export_sar_to_csv(day_str, output)
        current += timedelta(days=1)

How to Define “Normal”: Baseline Modeling Methods

Data collected. Next question: what counts as normal?

This sounds like a no-brainer — “normal is how it usually is.” But how do you quantify “usually”? Three methods, from simple to complex, pick as needed.

Method 1: Quantile Method (Most Practical)

Simplest and most practical. Take P50 (median) of historical data as the baseline value, P95 and P99 as fluctuation upper bounds.

Example: 30 days of CPU usage data, sorted from low to high:

P50 = 42% — CPU is below this half the time
P95 = 68% — CPU is below this 95% of the time
P99 = 75% — CPU is almost always below this

Baseline: normal CPU hovers around 42%, occasionally hits 68%, extreme cases reach 75%. Only consider alerting above P99.

Pros: simple, no machine learning needed, results are intuitive and explainable. Cons: can’t distinguish time periods — 3 AM and 3 PM CPU baselines are inherently different, but mixing them together flattens the baseline.

Implementing quantile baselines with PromQL:

# 30-day P50 baseline for CPU usage
quantile_over_time(0.50, (100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100))[30d:1m])

# 30-day P95 baseline (upper bound)
quantile_over_time(0.95, (100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100))[30d:1m])

# Current value exceeds P95 baseline
(100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100)) >
  quantile_over_time(0.95, (100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100))[30d:1m])

Method 2: Time-Bucket Method (Recommended)

Slice the day into time buckets and compute baselines per bucket. For example, divide into 24 hourly buckets, each with its own P50 and P95.

Now 3 AM baseline differs from 3 PM baseline. A batch job pushing CPU to 90% during work hours is normal; 90% at 3 AM warrants investigation.

# Hourly bucketed CPU baseline
# Uses avg_over_time with hour() function for time-period baselines
avg_over_time(
  (100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100))[30d:1m]
) * on() group_left() (hour(time()) == 14)  # 14:00 bucket

A more elegant approach uses Recording Rules to pre-compute hourly baselines:

# prometheus rules - perf-baseline.yml
groups:
  - name: performance_baseline
    rules:
      # Hourly CPU baseline (P50)
      - record: cpu_usage_baseline_p50
        expr: |
          quantile_over_time(0.50,
            (100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100))
          [30d:1m])          
      
      # Hourly CPU baseline (P95)
      - record: cpu_usage_baseline_p95
        expr: |
          quantile_over_time(0.95,
            (100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100))
          [30d:1m])          
      
      # Deviation from baseline alert
      - alert: CpuUsageAboveBaseline
        expr: |
          (100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100))
          >
          cpu_usage_baseline_p95 * 1.2          
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.instance }} CPU exceeds 1.2x of baseline P95"

Method 3: Dynamic Baseline (Advanced)

If you have time-series analysis capability (e.g., a machine learning platform), you can use more advanced methods. Core idea: use a sliding window to predict the expected value at the next time point. If the actual value deviates beyond a certain range, flag it as anomalous.

A lightweight implementation uses Python’s statsmodels library for seasonal decomposition:

#!/usr/bin/env python3
"""
Dynamic baseline anomaly detection based on STL seasonal decomposition.
Suitable for metrics with clear daily/weekly periodicity (e.g., QPS, CPU, memory).
Dependency: pip install statsmodels pandas numpy
"""
import numpy as np
import pandas as pd
from statsmodels.tsa.seasonal import seasonal_decompose
from datetime import datetime, timedelta

def build_dynamic_baseline(metric_series, period=1440, window_days=30):
    """
    Build dynamic baseline.
    
    Parameters:
        metric_series: pd.Series, time-indexed metric series
        period: seasonal period (1440 = minutes in a day)
        window_days: how many days of history to use for modeling
    
    Returns:
        baseline: baseline value (expected value)
        upper_bound: upper limit (baseline + 3 * residual std)
        lower_bound: lower limit
    """
    # Take last N days of data
    cutoff = metric_series.index[-1] - timedelta(days=window_days)
    data = metric_series[metric_series.index >= cutoff]
    
    # Drop missing values
    data = data.dropna()
    
    if len(data) < period * 3:
        # Less than 3 periods of data, fall back to simple statistics
        baseline = data.rolling(window=period, min_periods=1).median()
        std = data.rolling(window=period, min_periods=1).std()
        return baseline, baseline + 3 * std, baseline - 3 * std
    
    # STL decomposition: trend + seasonality + residual
    decomposition = seasonal_decompose(data, model='additive', period=period)
    
    # Baseline = trend + seasonality
    baseline = decomposition.trend + decomposition.seasonal
    
    # Residual standard deviation as fluctuation range
    resid_std = decomposition.resid.std()
    upper_bound = baseline + 3 * resid_std
    lower_bound = baseline - 3 * resid_std
    
    return baseline, upper_bound, lower_bound

def detect_anomalies(metric_series, baseline, upper_bound, lower_bound):
    """Detect anomaly points"""
    df = pd.DataFrame({
        'actual': metric_series,
        'baseline': baseline,
        'upper': upper_bound,
        'lower': lower_bound
    })
    
    df['is_anomaly'] = (df['actual'] > df['upper']) | (df['actual'] < df['lower'])
    df['deviation'] = abs(df['actual'] - df['baseline'])
    
    anomalies = df[df['is_anomaly']].copy()
    return anomalies

# Usage example
if __name__ == '__main__':
    # Simulated data: 30 days of CPU usage with daily periodicity
    np.random.seed(42)
    dates = pd.date_range(start='2026-08-01', end='2026-08-30', freq='1min')
    
    # Daily cycle + weekly cycle + noise
    daily_pattern = 30 * np.sin(2 * np.pi * np.arange(len(dates)) / 1440)
    weekly_pattern = 5 * np.sin(2 * np.pi * np.arange(len(dates)) / (1440 * 7))
    noise = np.random.normal(0, 5, len(dates))
    cpu_values = 50 + daily_pattern + weekly_pattern + noise
    
    series = pd.Series(cpu_values, index=dates)
    
    # Build baseline
    baseline, upper, lower = build_dynamic_baseline(series)
    
    # Detect anomalies
    anomalies = detect_anomalies(series, baseline, upper, lower)
    
    print(f"Baseline modeling complete, detected {len(anomalies)} anomaly points")
    if len(anomalies) > 0:
        print("\nAnomaly examples:")
        print(anomalies[['actual', 'baseline', 'upper']].head(10))

Honestly, Method 3 is overkill for most scenarios. Unless you have hundreds of thousands of machines, massive alert volumes, and simple quantile methods can’t keep up — only then is dynamic baselining worth the investment. Methods 1 and 2 solve 80% of problems. Nail those two first.

From Baseline to Alerts: Engineering Anomaly Detection

Baseline built. How do you use it? The key is converting “deviation from baseline” into actionable alert rules.

Alert Rule Design Principles

Baseline-based alerts differ fundamentally from static threshold alerts. The core distinction: the condition isn’t “absolute value exceeds threshold” but “relative deviation exceeds threshold.”

Three design principles:

Duration filtering: Baseline deviation must persist for a period before alerting. Transient jitter (GC pauses, momentary network blips) shouldn’t trigger alerts. Start with for: 5m.
Deviation multiplier: Don’t alert on “exceeds baseline.” Alert on “exceeds baseline by Nx” or “exceeds baseline + Kx standard deviation.” N is typically 1.5-2, K is typically 3.
Multi-metric correlation: A single metric deviation isn’t necessarily a problem. Multiple correlated metrics deviating simultaneously is a real problem. E.g., CPU deviation + network retransmit deviation = possibly a network-induced interruption.

Here’s a complete Prometheus baseline alert rule set:

# baseline-alerts.yml
groups:
  - name: baseline_anomaly_alerts
    rules:
      # CPU usage deviates from baseline
      - alert: CpuDeviationFromBaseline
        expr: |
          (
            (100 - avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
            -
            quantile_over_time(0.50, (100 - avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100))[30d:1m]
          )
          >
          3 * stddev_over_time((100 - avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100))[30d:1m]          
        for: 10m
        labels:
          severity: warning
          category: baseline-deviation
        annotations:
          summary: "{{ $labels.instance }} CPU usage deviates from baseline by over 3 standard deviations"
          description: "Current CPU {{ $value }} deviates from historical P50 baseline, sustained for over 10 minutes"

      # Disk I/O wait time deviates from baseline
      - alert: DiskIoAwaitDeviation
        expr: |
          rate(node_disk_io_time_weighted_seconds_total[5m])
          /
          rate(node_disk_io_time_seconds_total[5m])
          >
          quantile_over_time(0.95, 
            rate(node_disk_io_time_weighted_seconds_total[5m]) / rate(node_disk_io_time_seconds_total[5m])
          )[30d:5m] * 1.5          
        for: 15m
        labels:
          severity: critical
          category: baseline-deviation
        annotations:
          summary: "{{ $labels.instance }} disk I/O wait time deviates from baseline"

      # Network drop rate deviates from baseline
      - alert: NetworkDropDeviation
        expr: |
          rate(node_network_receive_drop_total[5m]) + rate(node_network_transmit_drop_total[5m])
          >
          quantile_over_time(0.99,
            rate(node_network_receive_drop_total[5m]) + rate(node_network_transmit_drop_total[5m])
          )[30d:5m]          
        for: 5m
        labels:
          severity: critical
          category: baseline-deviation
        annotations:
          summary: "{{ $labels.instance }} network drop rate exceeds 30-day P99 baseline"

      # Memory available drops below baseline
      - alert: MemoryAvailableDeviation
        expr: |
          node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100
          <
          quantile_over_time(0.05,
            node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100
          )[30d:5m] * 0.5          
        for: 10m
        labels:
          severity: critical
          category: baseline-deviation
        annotations:
          summary: "{{ $labels.instance }} available memory below 50% of historical P5 baseline"

Note the common pattern: all rules use quantile_over_time or stddev_over_time range functions computing 30-day statistics, then compare against current values. Not a single line uses > 80.

Alert Noise Reduction: Don’t Let Baseline Alerts Become the New Noise

Baseline alerts have their own pitfalls. The biggest: baselines themselves drift.

Say your service has been slowly leaking memory. Over 30 days, memory usage crept from 50% to 80%. If you build a baseline from those 30 days, the baseline drifts upward too — 80% memory usage ends up “normal.” The leak goes undetected.

Solutions:

1. Use a longer window for baseline, shorter window for deviation detection.

Build baselines from 30 or even 90 days of data so short-term deterioration gets stretched across the timeline. But detect deviations using the last 5 minutes. The baseline reflects long-term average water level; deviation reflects recent sudden change.

2. Periodically recalibrate baselines manually.

Review baseline data quarterly to confirm it still represents “healthy state.” If the business underwent major changes (new version, new features, architecture changes), rebuild the baseline.

3. Add rate-of-change alerts.

Absolute deviation isn’t enough — also watch the rate of change. A metric’s change over the past hour exceeding its historical P99 is itself an anomaly signal, regardless of whether it crossed the baseline absolute value.

# CPU usage 1-hour change exceeds historical P99
abs(
  (100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
  -
  (100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m] offset 1h)) * 100)
)
>
quantile_over_time(0.99,
  abs(
    (100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
    -
    (100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m] offset 1h)) * 100)
  )
)[30d:1h]

The 60-Second Quick Baseline Checklist

Not every scenario has time to wait 30 days for data to accumulate. Sometimes you’ve just taken over a new machine and need a quick read. Brendan Gregg mentioned a “60-second checklist” in his LISA'19 talk. I’ve adapted it:

#	Command	What to Look At	Anomaly Signals
1	`uptime`	load average (3 columns)	1min > 5min means rising; load > CPU cores means saturated
2	`dmesg -T \| tail -20`	Recent kernel logs	OOM kills, disk errors, hardware alerts
3	`vmstat 1 5`	r column, si/so, wa	r > cores = CPU saturated; si/so > 0 = using swap; wa > 20% = I/O bottleneck
4	`mpstat -P ALL 1 3`	Per-core CPU usage	Single core 100% but overall low = single-thread bottleneck
5	`pidstat 1 5`	Process-level CPU usage	Find the most CPU-hungry process
6	`iostat -xz 1 3`	%util, await, r/s, w/s	%util > 80% or await > 20ms = disk bottleneck
7	`free -h`	available column	available < 10% of total = memory critical
8	`sar -n DEV 1 3`	Per-NIC rxkB/s, txkB/s	Bandwidth near ceiling = network bottleneck
9	`ss -s`	TCP connection count and states	Excessive TIME-WAIT = connection reuse issue
10	`top -b -n 1 \| head -20`	System overview + top processes	Comprehensive assessment of which subsystem is busiest

This checklist isn’t a baseline — it’s the first step before building one. Run these 10 commands and you get a basic read on a machine’s “health baseline.” Then mount sar or your collection script, wait 3-7 days for data, and form an initial baseline.

Production-Grade Deployment: A Complete Baseline Automation Solution

Piecemeal fragments assembled — here’s a complete, production-ready baseline automation solution.

Architecture Design

Four layers:

+-----------------------------------------------------+
|              Alerting & Visualization Layer           |
|   Grafana Baseline Dashboards + Prometheus Alerts    |
+-----------------------------------------------------+
|              Baseline Computation Layer               |
|   Recording Rules (P50/P95/P99) + Python Detection   |
+-----------------------------------------------------+
|              Data Storage Layer                       |
|   Prometheus TSDB (short-term) + Long-term (Thanos/VM)|
+-----------------------------------------------------+
|              Collection Layer                         |
|   node_exporter + custom textfile collector           |
|   + sar (fallback) + collection script (/proc direct)|
+-----------------------------------------------------+

Complete Recording Rules

# /etc/prometheus/rules/baseline-recording-rules.yml
groups:
  - name: cpu_baseline
    interval: 1m
    rules:
      # CPU usage (minus idle)
      - record: instance:cpu_usage:ratio
        expr: 1 - avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m]))
      
      # 30-day P50 baseline
      - record: instance:cpu_usage_baseline:p50
        expr: quantile_over_time(0.50, instance:cpu_usage:ratio[30d:1m])
      
      # 30-day P95 baseline
      - record: instance:cpu_usage_baseline:p95
        expr: quantile_over_time(0.95, instance:cpu_usage:ratio[30d:1m])
      
      # 30-day P99 baseline
      - record: instance:cpu_usage_baseline:p99
        expr: quantile_over_time(0.99, instance:cpu_usage:ratio[30d:1m])
      
      # 30-day standard deviation
      - record: instance:cpu_usage_baseline:stddev
        expr: stddev_over_time(instance:cpu_usage:ratio[30d:1m])

  - name: memory_baseline
    interval: 1m
    rules:
      - record: instance:mem_available:ratio
        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
      
      - record: instance:mem_available_baseline:p50
        expr: quantile_over_time(0.50, instance:mem_available:ratio[30d:1m])
      
      - record: instance:mem_available_baseline:p05
        expr: quantile_over_time(0.05, instance:mem_available:ratio[30d:1m])

  - name: disk_baseline
    interval: 1m
    rules:
      - record: instance:disk_io_util:ratio
        expr: rate(node_disk_io_time_seconds_total[5m])
      
      - record: instance:disk_io_util_baseline:p95
        expr: quantile_over_time(0.95, instance:disk_io_util:ratio[30d:1m])
      
      - record: instance:disk_io_util_baseline:p50
        expr: quantile_over_time(0.50, instance:disk_io_util:ratio[30d:1m])

  - name: network_baseline
    interval: 1m
    rules:
      - record: instance:net_drop_rate
        expr: rate(node_network_receive_drop_total[5m]) + rate(node_network_transmit_drop_total[5m])
      
      - record: instance:net_drop_baseline:p99
        expr: quantile_over_time(0.99, instance:net_drop_rate[30d:1m])

Grafana Baseline Dashboard

Build a baseline dashboard in Grafana. The core idea: plot current values alongside baseline ranges so you can tell at a glance whether current performance falls within the baseline envelope.

Recommended panel configuration:

Panel	Metric	Visualization	Notes
CPU Usage	Current + P50 + P95 + P99	Line chart	Four lines; current crossing P95 turns red
Memory Available	Current + P50 + P05	Line chart	Current below P05 turns red
Disk I/O	Current %util + P50 + P95	Line chart	Current above P95 turns red
Network Drops	Current + P99	Bar chart	Bars exceeding P99 turn red
Deviation Multiplier	(Current - P50) / stddev	Line chart	Threshold line at 3

One-Shot Deployment Script

#!/bin/bash
# deploy-baseline-monitoring.sh
# One-shot deployment of baseline monitoring (Prometheus + node_exporter + Recording Rules)

set -euo pipefail

PROMETHEUS_VERSION="2.55.0"
NODE_EXPORTER_VERSION="1.9.0"
ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')

echo "=== 1. Install node_exporter ==="
cd /tmp
wget -q "https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXPORTER_VERSION}/node_exporter-${NODE_EXPORTER_VERSION}.linux-${ARCH}.tar.gz"
tar xzf "node_exporter-${NODE_EXPORTER_VERSION}.linux-${ARCH}.tar.gz"
cp "node_exporter-${NODE_EXPORTER_VERSION}.linux-${ARCH}/node_exporter" /usr/local/bin/
chmod +x /usr/local/bin/node_exporter

# Create systemd service
cat > /etc/systemd/system/node-exporter.service <<'UNIT'
[Unit]
Description=Node Exporter
After=network.target

[Service]
ExecStart=/usr/local/bin/node_exporter \
    --collector.textfile.directory=/var/node_exporter/textfile \
    --collector.tcpstat \
    --collector.processes
Restart=always
RestartSec=5
User=node_exporter

[Install]
WantedBy=multi-user.target
UNIT

useradd -r -s /bin/false node_exporter 2>/dev/null || true
mkdir -p /var/node_exporter/textfile
chown -R node_exporter:node_exporter /var/node_exporter
systemctl daemon-reload
systemctl enable --now node-exporter

echo "=== 2. Create Recording Rules ==="
mkdir -p /etc/prometheus/rules
# ... (insert baseline-recording-rules.yml content here)
# In actual deployment, write the YAML content to file

echo "=== 3. Create Alert Rules ==="
# ... (insert baseline-alerts.yml content here)

echo "=== 4. Verify Deployment ==="
sleep 2
if curl -s http://localhost:9100/metrics | grep -q "node_cpu_seconds"; then
    echo "node_exporter running normally"
else
    echo "node_exporter failed to start, check: journalctl -u node-exporter"
    exit 1
fi

echo ""
echo "=== Deployment Complete ==="
echo "node_exporter: http://$(hostname -I | awk '{print $1}'):9100/metrics"
echo "Next step: add this host as a scrape target in Prometheus config"
echo "Baseline data needs 3-7 days to accumulate before forming an effective baseline"

Hard-Learned Lessons: 5 Pitfalls in Baseline Deployment

Lesson 1: Don’t build baselines during a change window.

Once built a baseline during a release window. The new version happened to introduce a slow memory leak. The baseline came out showing memory P95 at 85% — that wasn’t “normal,” it was “running sick.” After fixing the bug, the baseline became a source of false positives.

Correct approach: wait at least 7 days of stable operation before starting baseline collection. If a major change just happened, wait a week.

Lesson 2: A longer baseline window isn’t always better.

30 days is a good window. Tried 90 days once and found the baseline became too insensitive to short-term changes — a machine’s disk I/O went from 200 to 800 IOPS, and the baseline took two weeks to “catch up.” During that time, all alerts were missed.

Conversely, 7-day windows don’t work either — weekend vs. weekday differences make baselines too volatile.

In practice, 30-day window + 1-minute collection interval is the most cost-effective combination.

Lesson 3: Some metrics don’t suit baselining.

Error-count metrics (OOM kill count, disk error count, NIC hardware errors) aren’t suited for quantile baselines. These metrics should be 0 or near-zero under normal conditions. Their “normal baseline” is 0 — any non-zero value deserves attention.

For these metrics, “alert on non-zero” is more effective than baselining:

# Error metrics: alert on non-zero
- alert: OomKillDetected
  expr: increase(node_vmstat_oom_kill[5m]) > 0
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: "{{ $labels.instance }} experienced OOM Kill"

Lesson 4: Baseline alerts need dedicated routing.

Baseline alerts have different semantics than traditional threshold alerts — they say “this differs from usual,” not “this crossed a danger line.” Operations handles these two types differently.

Recommend tagging baseline alerts with category: baseline-deviation in Alertmanager and routing to a separate channel, or starting with low-priority notifications (e.g., IM message instead of phone call):

# alertmanager.yml
route:
  receiver: default
  group_by: ['instance', 'category']
  routes:
    - matchers:
        - category = "baseline-deviation"
      receiver: baseline-alerts-channel
      group_wait: 10m       # Aggregate baseline alerts for 10 minutes
      repeat_interval: 4h   # Don't repeat for 4 hours

receivers:
  - name: default
    webhook_configs:
      - url: 'https://hooks.slack.com/...'
  - name: baseline-alerts-channel
    webhook_configs:
      - url: 'https://hooks.slack.com/...'

Lesson 5: Baselines aren’t build-once-and-forget.

Baselines are living things. Business changes, traffic changes, architecture changes — baselines need to change with them. Recommend establishing a quarterly baseline review process:

Each quarter, pull baseline data for all machines and review whether it’s still reasonable
If baseline drift is detected (e.g., CPU baseline crept from 40% to 60%), investigate the cause
After confirming the cause is legitimate, rebuild the baseline from the last 30 days of data
If the cause is illegitimate (e.g., memory leak), fix the problem first, then rebuild

Summary

Performance baselines aren’t a new concept, but many teams haven’t implemented them. The reason isn’t technical difficulty — quantile method + Prometheus Recording Rules will do it — but rather not forming the habit of “build baseline → use baseline → adjust baseline.”

Recommended practical path:

Run the 60-second checklist for a quick read, confirm no obvious issues
Install node_exporter + sar, start collecting data, accumulate at least 7 days
Deploy quantile baseline Recording Rules (Method 1), start with CPU and memory subsystems
Configure baseline alert rules, collect alerts via low-priority channel, observe false positive rate for a week
Gradually expand to disk I/O, network, TCP metrics
After running stable for a month, consider time-bucket method (Method 2) to separate work hours from off-hours
When machine count exceeds 500 or daily alert volume exceeds 100, consider dynamic baselining (Method 3)

Core idea in one sentence: let data tell you what’s normal, instead of guessing thresholds. The upfront investment is modest (node_exporter + a few Recording Rules), but the long-term payoff is substantial — fewer alerts, lower MTTR, and fewer 3 AM phone calls.

References & Acknowledgments

This article references the following materials during writing. Thanks to the original authors for their contributions:

Performance Analysis Methodology — Brendan Gregg, original definition of the USE Method and overview of multiple performance analysis methodologies
Linux Systems Performance (LISA'19) — Brendan Gregg / USENIX, systematic summary of six major areas in Linux performance analysis
perf-tools (GitHub) — Brendan Gregg, performance analysis toolset based on perf_events and ftrace
Chapter 6: System Performance Baseline and Anomaly Detection — CSDN/zhoucoolqi, practical baseline approach from “firefighter” to “health manager”
What Is AIOps? Guide to Artificial Intelligence for IT Operations — phoenixNAP, explanation of dynamic baseline capabilities in AIOps anomaly detection
How to Optimize Linux Device Performance in 2026 — fosslinux/Liam, practical advice on establishing baselines before optimizing

Overview#

Why Static Thresholds Don’t Work#

The USE Method: The Backbone of Baseline Collection#

Baseline Metric Checklist#

Collection Tool Selection: From Manual to Automated#

Layer 1: Lightweight Kernel-Level Collection — sar#

Layer 2: Custom Collection Scripts — collectd / In-House Scripts#

Layer 3: Time-Series Database + Visualization#

How to Define “Normal”: Baseline Modeling Methods#

Method 1: Quantile Method (Most Practical)#

Method 2: Time-Bucket Method (Recommended)#

Method 3: Dynamic Baseline (Advanced)#

From Baseline to Alerts: Engineering Anomaly Detection#

Alert Rule Design Principles#

Alert Noise Reduction: Don’t Let Baseline Alerts Become the New Noise#

The 60-Second Quick Baseline Checklist#

Production-Grade Deployment: A Complete Baseline Automation Solution#

Architecture Design#

Complete Recording Rules#

Grafana Baseline Dashboard#

One-Shot Deployment Script#

Hard-Learned Lessons: 5 Pitfalls in Baseline Deployment#

Summary#

References & Acknowledgments#