Grafana Dashboard Best Practices

Grafana is the most popular visualization platform in the cloud-native era, but there’s a world of difference between “functional” and “effective.” A cluttered dashboard leaves on-call engineers lost in a sea of panels, while a well-designed one conveys system health in 5 seconds. This article starts from design principles, covers the variable system, panel selection, and alerting integration, and ties everything together with a complete SLO dashboard.

Reference: Grafana Official Documentation

I. Dashboard Design Principles

1.1 The Five-Second Rule

A dashboard should answer the most critical question within 5 seconds: Is the system healthy right now? If it takes longer than 5 seconds to understand, the information hierarchy is wrong.

Practical approach:

Top row for global status: Use Stat or Gauge panels to display SLO achievement rate, core error rate, and P99 latency. Green/yellow/red thresholds at a glance.
Middle section for trends: Time series panels showing metric trends over the past 1–6 hours.
Bottom section for detail tables: Table panels listing instance-level details for deep-dive troubleshooting.

1.2 Left to Right, Top to Bottom

Humans read from top-left to bottom-right. A dashboard’s information flow should follow this pattern:

┌─────────────────────────────────────────────┐
│  [SLO]  [Error Rate]  [P99 Latency]  [Traffic]  │  ← Row 1: Status at a glance
├─────────────────────────────────────────────┤
│  CPU Trend           │   Memory Trend         │  ← Row 2: Trends
│  Request Volume      │   Error Rate Trend     │
├─────────────────────────────────────────────┤
│  Instance Detail Table                       │  ← Row 3: Details
└─────────────────────────────────────────────┘

1.3 Other Design Tips

One dashboard, one theme: Don’t mix “database monitoring” and “business metrics” in the same dashboard.
Use threshold colors judiciously: Green = normal, yellow = warning, red = critical. Don’t overuse colors.
Set default time range to “Last 1 hour”: The most common on-call scenario.
Clear naming: Panel titles should say “CPU Usage (%)” not “cpu”.

II. Variable Template System

Variables are the core of dashboard reusability. With variables, you can achieve “one template, multiple environments.”

2.1 Creating Variables

Add variables in Dashboard Settings → Variables. Here are common variable configurations:

Datasource variable $datasource

Type: Datasource
Name: datasource
Query: Prometheus

Server variable $server

Type: Query
Name: server
Query: label_values(node_uname_info, instance)

Instance variable $instance (cascaded from $server)

Type: Query
Name: instance
Query: label_values(node_uncpu_info{instance=~"$server"}, cpu)

Custom variable $environment

Type: Custom
Name: environment
Query: prod, staging, dev

2.2 Variable Reference Syntax

# Reference a variable in a panel query
up{instance=~"$server"}

# Multi-value variable (when Multi-value is enabled)
up{instance=~"$server"}   # $server expands to node-1|node-2|node-3

# Use in panel title
CPU Usage - $server

# Use in dashboard links
/dashboard/sre-overview?var-server=$server

2.3 Variable Chaining

Multi-level variables enable cascading filters like “select environment → select cluster → select node”:

# Level 1: $environment (Custom: prod, staging, dev)

# Level 2: $cluster (depends on $environment)
label_values(kube_node_info{cluster=~"$environment"}, node)

# Level 3: $pod (depends on $cluster)
label_values(kube_pod_info{node="$cluster"}, pod)

III. Panel Type Selection Guide

Panel Type	Use Case	Typical Metrics
Time series	Time-series trend analysis	CPU, memory, QPS, latency trends
Stat	Single key value	SLO achievement rate, current online users
Gauge	Dashboard-style value display	Disk usage, CPU usage
Bar gauge	Multi-instance horizontal comparison	Memory usage comparison across nodes
Table	Structured details	Instance list, alert list
Heatmap	Distributed latency analysis	Request latency distribution
Pie chart	Proportion analysis	Traffic share by status code
State timeline	State change timeline	Node alive/dead status

Selection Decision Tree

Need to display a single key value?
├── Yes → Need a gauge effect? → Gauge
│         No → Stat
└── No → Need trends?
          ├── Yes → Time series
          └── No → Need to compare multiple instances?
                  ├── Yes → Bar gauge
                  └── No → Need details? → Table

Key Panel Configuration Examples

Stat Panel: SLO Achievement Rate

# Query
1 - (
  sum(rate(http_requests_total{status=~"5..", service="$service"}[5m]))
  / sum(rate(http_requests_total{service="$service"}[5m]))
)

# Thresholds
Thresholds:
  - Base: 0    (Green)
  - T1: 0.01   (Yellow)   # Error rate > 1% turns yellow
  - T2: 0.05   (Red)      # Error rate > 5% turns red

# Color mode: Background
# Unit: Percent (0.0-1.0)

Bar Gauge Panel: CPU Comparison Across Nodes

# Query
100 * (1 - avg by (instance) (
  rate(node_cpu_seconds_total{mode="idle", instance=~"$server"}[5m])
))

# Calculation: Last *
# Orientation: Horizontal
# Display mode: Gradient

IV. Alerting Integration: Grafana Alerting

Grafana Unified Alerting supports cross-datasource alerting. Compared to configuring Alertmanager separately on the Prometheus side, it’s better suited for “visual alert management” scenarios.

4.1 Alerting Architecture

Alert Rule → Notification Policy → Contact Point → Notification channel
      ↓
Notification Policy (route matching) → Silences

4.2 Creating Alert Rules

Configure alert rules via UI or Terraform. Here’s a YAML-formatted rule example (Grafana provisioning):

# alerting/alert_rules.yaml
apiVersion: 1

groups:
  - orgId: 1
    name: SLO Alerts
    interval: 60s
    rules:
      - uid: slo-error-rate-high
        title: "SLO Error Rate Alert - {{ $labels.service }}"
        condition: B
        data:
          - refId: A
            relativeTimeRange:
              from: 600   # Past 10 minutes
              to: 0
            datasourceUid: prometheus
            model:
              expr: |
                sum(rate(http_requests_total{status=~"5..", service="$service"}[5m]))
                / sum(rate(http_requests_total{service="$service"}[5m]))                
              instant: true
          - refId: B
            relativeTimeRange:
              from: 600
              to: 0
            datasourceUid: __expr__
            model:
              type: threshold
              expression: A
              conditions:
                - evaluator:
                    params: [0.05]
                    type: gt
        noDataState: NoData
        execErrState: Error
        for: 5m
        annotations:
          summary: "Service {{ $labels.service }} error rate exceeds 5%"
          description: "Current error rate: {{ $values.A }}, SLO threshold: 5%"
        labels:
          severity: critical
          team: sre
        notification_settings:
          group_by: ['service', 'alertname']
          group_wait: 30s
          group_interval: 5m
          repeat_interval: 4h

4.3 Notification Policies and Contact Points

# alerting/notification_policies.yaml
apiVersion: 1

policies:
  - receiver: default
    group_by: ['alertname', 'service']
    routes:
      - receiver: critical-team
        matchers:
          - severity="critical"
        group_wait: 0s
        repeat_interval: 1h

      - receiver: warning-team
        matchers:
          - severity="warning"
        group_wait: 30s
        repeat_interval: 4h

contactPoints:
  - orgId: 1
    name: critical-team
    receivers:
      - uid: webhook-critical
        type: webhook
        settings:
          url: https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=YOUR_KEY
          httpMethod: POST
      - uid: email-critical
        type: email
        settings:
          addresses: ['oncall@example.com']

  - orgId: 1
    name: default
    receivers:
      - uid: default-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/xxx

V. Practical: Building a Complete SLO Dashboard

The following JSON model snippets demonstrate the core structure of an SLO dashboard with multi-panel interaction.

5.1 Dashboard Variable Definitions

{
  "templating": {
    "list": [
      {
        "name": "datasource",
        "type": "datasource",
        "query": "prometheus",
        "current": { "text": "Prometheus", "value": "Prometheus" }
      },
      {
        "name": "service",
        "type": "query",
        "datasource": "$datasource",
        "query": "label_values(http_requests_total, service)",
        "refresh": 2,
        "includeAll": false
      },
      {
        "name": "status_filter",
        "type": "custom",
        "query": "2xx,3xx,4xx,5xx",
        "default": "2xx"
      }
    ]
  }
}

5.2 Panel 1: SLO Achievement Rate (Stat)

{
  "title": "SLO Achievement Rate - $service",
  "type": "stat",
  "datasource": "$datasource",
  "targets": [
    {
      "expr": "1 - (sum(rate(http_requests_total{status=~\"5..\", service=\"$service\"}[5m])) / sum(rate(http_requests_total{service=\"$service\"}[5m])))",
      "legendFormat": "SLO"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "percentunit",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "value": null, "color": "red" },
          { "value": 0.95, "color": "yellow" },
          { "value": 0.99, "color": "green" }
        ]
      }
    }
  },
  "options": {
    "colorMode": "background",
    "reduceOptions": { "calcs": ["lastNotNull"] }
  }
}

5.3 Panel 2: Error Rate Trend (Time Series)

{
  "title": "Error Rate Trend - $service",
  "type": "timeseries",
  "datasource": "$datasource",
  "targets": [
    {
      "expr": "sum(rate(http_requests_total{status=~\"5..\", service=\"$service\"}[5m])) / sum(rate(http_requests_total{service=\"$service\"}[5m])) * 100",
      "legendFormat": "5xx Error Rate (%)"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "percent",
      "custom": {
        "drawStyle": "line",
        "lineInterpolation": "smooth",
        "fillOpacity": 20
      },
      "thresholds": {
        "steps": [
          { "value": null, "color": "green" },
          { "value": 1, "color": "yellow" },
          { "value": 5, "color": "red" }
        ]
      }
    }
  }
}

5.4 Panel 3: P99/P50 Latency Comparison (Time Series)

{
  "title": "Request Latency P50 / P99 - $service",
  "type": "timeseries",
  "targets": [
    {
      "expr": "histogram_quantile(0.50, sum by (le) (rate(http_request_duration_seconds_bucket{service=\"$service\"}[5m])))",
      "legendFormat": "P50"
    },
    {
      "expr": "histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{service=\"$service\"}[5m])))",
      "legendFormat": "P99"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "s",
      "custom": {
        "drawStyle": "line",
        "fillOpacity": 10
      }
    }
  }
}

5.5 Panel 4: QPS Details by Instance (Table)

{
  "title": "Instance QPS Details",
  "type": "table",
  "targets": [
    {
      "expr": "sum by (instance) (rate(http_requests_total{service=\"$service\"}[5m]))",
      "format": "table",
      "instant": true
    }
  ],
  "transformations": [
    {
      "id": "organize",
      "options": {
        "excludeByName": { "Time": true },
        "renameByName": { "Value": "QPS" }
      }
    }
  ]
}

VI. JSON Model Export and Version Management

6.1 Exporting JSON

On the Dashboard page, click the top menu → Share → Export → Save to file to get the complete JSON model.

6.2 Version Management via Git

Store the exported JSON files in a Git repository for dashboard version control:

mkdir -p dashboards/slo
cp exported-dashboard.json dashboards/slo/slo-overview.json
git add dashboards/
git commit -m "feat: add SLO overview dashboard"

6.3 Provisioning Auto-Loading

Grafana supports auto-loading dashboard JSON from the filesystem, without manual import:

# /etc/grafana/provisioning/dashboards/dashboards.yaml
apiVersion: 1

providers:
  - name: SRE Dashboards
    orgId: 1
    folder: SRE
    folderUid: sre-folder
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /var/lib/grafana/dashboards

After placing JSON files in the /var/lib/grafana/dashboards/ directory, Grafana automatically scans and loads them every 30 seconds.

6.4 Managing with Terraform (Recommended for Production)

# terraform/main.tf
resource "grafana_dashboard" "slo" {
  folder       = grafana_folder.sre.uid
  config_json  = file("${path.module}/dashboards/slo-overview.json")
}

resource "grafana_folder" "sre" {
  title = "SRE"
}

terraform plan
terraform apply

Benefits: Infrastructure as code, dashboard changes are auditable and rollback-able, suitable for multi-person collaborative teams.

VII. Performance Optimization Tips

Reduce panel count: Keep panels per dashboard under 12. Too many panels cause browser lag and Prometheus query storms.
Use recording rules: Pre-compute complex PromQL into metrics; dashboards only query the pre-computed results.
Set reasonable query intervals: Panel Interval should not be less than 30s to reduce backend pressure.
Use $__rate_interval: Grafana’s built-in variable that automatically calculates the appropriate rate window based on panel time range and scrape interval:

# Recommended
rate(http_requests_total[$__rate_interval])

# Instead of a fixed window
rate(http_requests_total[5m])

Limit returned series count: Configure Max data points in datasource settings to avoid browser crashes from too many returned series.

Summary

The core of Grafana dashboard design is not “cram every metric onto the screen,” but organizing information around “whether on-call engineers can assess system status in 5 seconds.” Remember three layers: top for status, middle for trends, bottom for details. Combined with the variable template system for multi-environment reuse, and Provisioning or Terraform for version management, you can build professional-grade operations dashboards.

For more details, see Grafana Official Documentation

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

Grafana Official Documentation — Grafana Labs, referenced for Grafana Official Documentation

I. Dashboard Design Principles#

1.1 The Five-Second Rule#

1.2 Left to Right, Top to Bottom#

1.3 Other Design Tips#

II. Variable Template System#

2.1 Creating Variables#

2.2 Variable Reference Syntax#

2.3 Variable Chaining#

III. Panel Type Selection Guide#

Selection Decision Tree#

Key Panel Configuration Examples#

IV. Alerting Integration: Grafana Alerting#

4.1 Alerting Architecture#

4.2 Creating Alert Rules#

4.3 Notification Policies and Contact Points#

V. Practical: Building a Complete SLO Dashboard#

5.1 Dashboard Variable Definitions#

5.2 Panel 1: SLO Achievement Rate (Stat)#

5.3 Panel 2: Error Rate Trend (Time Series)#

5.4 Panel 3: P99/P50 Latency Comparison (Time Series)#

5.5 Panel 4: QPS Details by Instance (Table)#

VI. JSON Model Export and Version Management#

6.1 Exporting JSON#

6.2 Version Management via Git#

6.3 Provisioning Auto-Loading#

6.4 Managing with Terraform (Recommended for Production)#

VII. Performance Optimization Tips#

Summary#

References & Acknowledgments#