Prometheus PromQL: From Beginner to Expert

PromQL (Prometheus Query Language) is the query language of the Prometheus monitoring system and the core of the cloud-native monitoring stack. Whether you’re building Grafana dashboards, writing alerting rules, or running ad-hoc queries during incident troubleshooting, PromQL is indispensable. This article starts from the data model and progressively covers aggregation operations, common functions, practical queries, and advanced techniques like subqueries.

Reference: Prometheus Official Documentation — Querying basics

I. PromQL Data Model

PromQL has four fundamental data types. Understanding them is the prerequisite for writing correct queries:

Type	Description	Example
Instant Vector	A set of time series with current sampled values	`node_cpu_seconds_total`
Range Vector	A set of time series with all samples within a time range	`node_cpu_seconds_total[5m]`
Scalar	A simple numeric value	`3.14`, `1024`
String	A string value (rarely used)	`"hello"`

The two most commonly used:

Instant Vector: The most common in dashboards and alerts, returning the value of each series at the “current moment.”
Range Vector: Used with functions like rate() and increase(), must include a time window [...].

# Instant vector: returns all current series
up

# Range vector: returns all samples from the past 5 minutes
up[5m]

# Scalar
1 - 0.3

II. Basic Queries

2.1 Metric Selection and Label Filtering

Use label selectors to precisely filter target series:

# Select all series named node_cpu_seconds_total
node_cpu_seconds_total

# Filter by the mode label
node_cpu_seconds_total{mode="idle"}

# Multiple label combination (AND)
node_cpu_seconds_total{instance="node-1:9100", mode="idle"}

# Regex label matching
node_cpu_seconds_total{instance=~"node-[0-9]+:9100"}

# Negative label matching (exclude certain values)
node_cpu_seconds_total{mode!="idle"}

# Negative regex matching
node_memory_MemTotal_bytes{instance!~"localhost.*"}

2.2 Range Vectors

Append [time_window] after the metric name to get a range vector. Supported time units: s (seconds), m (minutes), h (hours), d (days), w (weeks), y (years):

# Samples from the past 5 minutes
http_requests_total[5m]

# Past 1 hour
http_requests_total[1h]

# Past 30 seconds
http_requests_total[30s]

III. Aggregation Operations

Aggregation operations summarize multiple groups of time series. Core syntax:

<aggr-op>([parameter,] <vector>) [without|by (<label list>)]

Common Aggregation Operators

Operator	Description
`sum`	Sum
`avg`	Average
`max` / `min`	Maximum / Minimum
`count`	Count
`count_values`	Count by value grouping
`topk` / `bottomk`	Top K / Bottom K
`quantile`	Quantile

by vs. without

by retains specified labels for grouping, while without removes specified labels and groups by the remaining ones:

# Sum CPU idle time grouped by instance
sum by (instance) (node_cpu_seconds_total{mode="idle"})

# Aggregate after removing mode and cpu labels (keeps instance, job, etc.)
sum without (cpu, mode) (node_cpu_seconds_total)

# Top 3 instances by highest CPU usage
topk(3, sum by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m])))

topk Example

# Top 5 endpoints by traffic
topk(5, sum by (handler) (rate(http_requests_total[5m])))

IV. Common Functions

4.1 rate / irate / increase

These three functions only operate on Counter-type metrics:

# rate: average growth rate over the past 5 minutes (recommended for dashboards and alerts)
rate(http_requests_total[5m])

# irate: instantaneous growth rate from the last two samples (suitable for high-precision short-window charts)
irate(http_requests_total[5m])

# increase: absolute increment over the past 5 minutes
increase(http_requests_total[5m])

Selection guidelines:

rate() is suitable for alerts and dashboards; it smooths out data jitter.
irate() is suitable for ultra-high-precision short windows (e.g., [1m]), but is sensitive to missing data.
increase() answers questions like “how much did the total increase over the past hour?”

4.2 histogram_quantile

Histogram quantile calculation, used for P50/P90/P99 latency analysis:

# Calculate P99 latency (single-bucket syntax)
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

# Multi-instance scenario: aggregate by le first, then calculate
histogram_quantile(
  0.99,
  sum by (le, instance) (rate(http_request_duration_seconds_bucket[5m]))
)

Note: If multiple instances expose the same histogram, you must aggregate by le first. Otherwise, histogram_quantile will look for all buckets within a single series, producing incorrect results.

4.3 predict_linear

Predicts future trends based on linear regression, suitable for capacity forecasting alerts:

# Predict disk usage 1 hour from now
predict_linear(node_filesystem_avail_bytes[1h], 4 * 3600)

# Disk will be full within 4 hours
predict_linear(node_filesystem_avail_bytes[2h], 4 * 3600) < 0

4.4 Other High-Frequency Functions

# Time aggregation: max value every 5 minutes over the past hour
max_over_time(up[1h:5m])

# Same time point one day ago
rate(http_requests_total[5m] offset 1d)

# Calculate percentage: ratio of used memory to total memory
1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

# clamp_max to set an upper limit (e.g., filter outliers)
clamp_max(rate(http_requests_total[5m]), 1000)

V. Practical Query Examples

5.1 CPU Usage

# Single-machine CPU usage (%)
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Top 5 machines by CPU usage
topk(5,
  100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
)

Principle: The complement of idle time proportion is the CPU usage. avg is used because Node Exporter exposes data per CPU core.

5.2 Memory Usage

# Memory usage (%)
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes)
  / node_memory_MemTotal_bytes * 100

# Grouped by host
100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

5.3 P99 Latency

# Global P99 latency (seconds)
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

# P99 latency grouped by endpoint
histogram_quantile(0.99,
  sum by (le, handler) (rate(http_request_duration_seconds_bucket[5m]))
)

5.4 Error Rate

# HTTP 5xx error rate (%)
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m])) * 100

# Grouped by service
sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
  / sum by (service) (rate(http_requests_total[5m])) * 100

5.5 Comprehensive Example: Multi-Dimensional Traffic Dashboard Query

# Total QPS
sum(rate(http_requests_total[5m]))

# QPS by status code
sum by (status) (rate(http_requests_total[5m]))

# Success rate (2xx + 3xx proportion)
sum(rate(http_requests_total{status=~"[23].."}[5m]))
  / sum(rate(http_requests_total[5m]))

VI. Advanced Techniques

6.1 Subqueries

Subqueries allow applying a range and evaluation step to any instant query expression. Syntax: <expr>[range:resolution]:

# Max CPU usage every 5 minutes over the past hour
max_over_time(
  100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100
)[1h:5m]

# 5-minute rolling average of per-minute error rate over the past hour
avg_over_time(
  (sum(rate(http_requests_total{status=~"5.."}[5m]))
    / sum(rate(http_requests_total[5m])))[1h:1m]
)

6.2 offset Modifier

offset shifts the query time backward, commonly used for period-over-period analysis:

# Current QPS
sum(rate(http_requests_total[5m]))

# QPS at the same time one week ago
sum(rate(http_requests_total[5m] offset 1w))

# Week-over-week QPS difference
sum(rate(http_requests_total[5m]))
  - sum(rate(http_requests_total[5m] offset 1w))

6.3 @ Modifier (Time Modifier)

The @ modifier anchors a query to an absolute time specified by a UNIX timestamp:

# Query CPU usage at UNIX timestamp 1780000000
node_cpu_seconds_total @ 1780000000

# Anchor to 2 hours ago
rate(http_requests_total[5m] @ (time() - 2 * 3600))

# Combined with offset
rate(http_requests_total[5m] @ (time() - 86400) offset 1h)

The @ modifier is supported since Prometheus v2.25, suitable for building “incident time point retrospective” queries.

6.4 Recording Rules

High-frequency queries should use recording rules for pre-computation, avoiding the need to execute complex expressions on every query:

# prometheus-rules.yaml
groups:
  - name: custom_rules
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_errors:ratio
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))          

      - record: instance:cpu_usage:ratio
        expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

Referencing pre-computed metrics like job:http_requests:rate5m directly in alerts and dashboards can significantly reduce Prometheus query load.

VII. Common Pitfalls

Counter resets: rate() and increase() automatically handle Counter resets (e.g., service restarts resetting to zero), but only if you’re using Counter-type metrics, not Gauges.
Rate window too short: With only 2 sample points in a [1m] window, rate results fluctuate severely. Use at least [5m].
Missing aggregation in histogram_quantile: In multi-instance scenarios, calling it without first aggregating by le leads to incorrect results.
Division by zero: Use clamp_min to protect against a zero denominator:

# Safe ratio calculation
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / clamp_min(sum(rate(http_requests_total[5m])), 1)

Summary

The PromQL learning path can be summarized as: understand the data model → master label filtering → become proficient in aggregation → leverage core functions (rate / histogram_quantile) → advance to subqueries and recording rules. In day-to-day SRE work, 80% of query scenarios revolve around CPU, memory, latency, and error rate. By templating these practical queries and codifying them as recording rules, you can efficiently build a monitoring system.

For more details, see Prometheus Official Documentation — Querying

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

Prometheus Official Documentation — Querying basics — Prometheus Authors, referenced for Prometheus Official Documentation — Querying basics

I. PromQL Data Model#

II. Basic Queries#

2.1 Metric Selection and Label Filtering#

2.2 Range Vectors#

III. Aggregation Operations#

Common Aggregation Operators#

by vs. without#

topk Example#

IV. Common Functions#

4.1 rate / irate / increase#

4.2 histogram_quantile#

4.3 predict_linear#

4.4 Other High-Frequency Functions#

V. Practical Query Examples#

5.1 CPU Usage#

5.2 Memory Usage#

5.3 P99 Latency#

5.4 Error Rate#

5.5 Comprehensive Example: Multi-Dimensional Traffic Dashboard Query#

VI. Advanced Techniques#

6.1 Subqueries#

6.2 offset Modifier#

6.3 @ Modifier (Time Modifier)#

6.4 Recording Rules#

VII. Common Pitfalls#

Summary#

References & Acknowledgments#