Overview
At 2 AM, the payment system alerts exploded. After digging through logs for ages, we discovered the downstream services weren’t down — the API gateway’s connection pool was exhausted. Requests piled up at the gateway layer and never made it through. But our monitoring dashboard only showed CPU and memory curves, completely blind to gateway-level latency, connection counts, and error rates.
This isn’t an isolated case. I’ve seen too many teams treat API gateways as “fancy Nginx” and only monitor nginx_active_connections, only digging into logs when things break. The problem is that the gateway is the throat of all traffic — when this layer fails, the blast radius is global.
This article covers three things clearly: what metrics Kong, APISIX, and Envoy each expose, how to collect them with Prometheus, and how to design alert rules that notify you before failures spread. This isn’t generic “best practices” — it’s the playbook I assembled after hitting production issues.
Why API Gateways Need Dedicated Monitoring
You might wonder: we already have service monitoring behind the gateway, why monitor the gateway separately?
Because the gateway and backend services see different worlds. For example:
| Dimension | Gateway Perspective | Backend Service Perspective |
|---|---|---|
| Request latency | Includes route matching, plugin execution, upstream forwarding — the full chain | Only sees its own processing time |
| Error source | Could be missing route, plugin rejection, upstream timeout, pool exhaustion | Only knows it returned 500 |
| Connection state | Can see upstream connection pool utilization | Only knows how many requests it received |
| Rate limiting | Which rules were triggered at the gateway layer | Services may not even know they were rate-limited |
Simply put, the gateway is the toll booth on the traffic highway. When the toll booth jams, all cars behind pile up. If you don’t install monitoring at the toll booth, by the time the problem propagates to backend services, it’s too late to troubleshoot.
Core Metrics Comparison Across Three Gateways
Kong Metrics Exposure
Kong exposes metrics through the prometheus plugin. Enabling it is straightforward:
# Enable Prometheus plugin (global level)
curl -X POST http://kong-admin:8001/plugins \
--data "name=prometheus" \
--data "config.per_consumer=true" \
--data "config.status_code_metrics=true" \
--data "config.latency_metrics=true" \
--data "config.upstream_health_metrics=true"
After enabling, access http://kong-proxy:8001/metrics to get Prometheus-formatted metrics. Kong’s core metrics fall into four categories:
| Metric Name | Type | Description | Typical Alert Scenario |
|---|---|---|---|
kong_http_requests_total | Counter | Total request count (by service/route/status labels) | Error rate spike |
kong_latency_ms_bucket | Histogram | Latency distribution (includes Kong processing time and upstream response time) | P99 latency exceeds threshold |
kong_upstream_target_health | Gauge | Upstream health status (HEALTHY/UNHEALTHY) | Unhealthy upstream node |
kong_data_plane_version | Gauge | Data plane version number | Version mismatch |
The latency metric is the most critical. Kong breaks latency into three segments: kong_latency is Kong’s own processing time (route matching, plugin execution), upstream_latency is the upstream service response time, and total_latency is the sum. Only by looking at them separately can you determine whether the bottleneck is at the gateway or the backend.
A lesson learned: per_consumer=true causes metric cardinality explosion. If your consumer count exceeds 1,000, Prometheus storage pressure increases dramatically. In production, I recommend disabling this option first and enabling it on demand when needed.
APISIX Metrics Exposure
APISIX uses the prometheus plugin, configured similarly to Kong but with a completely different underlying architecture:
# APISIX prometheus plugin configuration
plugins:
- prometheus
# Enable via Admin API
# Global route level
curl http://127.0.0.1:9180/apisix/admin/routes/1 \
-H 'X-API-KEY: your-key' -X PUT -d '
{
"uri": "/api/*",
"plugins": {
"prometheus": {
"prefer_name": true
}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"127.0.0.1:8080": 1
}
}
}'
APISIX exposes metrics at http://apisix-ip:9091/metrics by default. The key difference from Kong is that APISIX uses etcd as its configuration center with a push-based watch model for config changes, resulting in lower propagation latency. Core metrics:
| Metric Name | Type | Description |
|---|---|---|
apisix_http_status | Counter | HTTP status code counts (by route/service/consumer) |
apisix_bandwidth | Counter | Upstream/downstream bandwidth |
apisix_http_latency_bucket | Histogram | Latency distribution |
apisix_node_info | Gauge | Node info (version, uptime) |
One APISIX advantage is the prefer_name: true option, which uses route names instead of UUIDs as labels — much more intuitive for queries. Kong defaults to service IDs, resulting in PromQL queries full of service="4e3f2a1b-..." gibberish that requires a lookup table during troubleshooting.
Pitfall: APISIX’s prometheus plugin doesn’t support per-consumer latency distribution. If your business needs per-consumer latency analysis, you’ll need to handle it at the log layer. Don’t wait until something breaks to discover you can’t query it.
Envoy Metrics Exposure
Envoy has the richest metrics system among the three gateways, but also the most complex. Envoy exposes hundreds of metrics, and newcomers easily get lost. The core metrics you actually need in production:
| Metric Name | Type | Description |
|---|---|---|
envoy_cluster_upstream_rq_total | Counter | Total upstream requests |
envoy_cluster_upstream_rq_2xx/4xx/5xx | Counter | Request counts by status code |
envoy_cluster_upstream_rq_time_bucket | Histogram | Upstream response latency distribution |
envoy_cluster_circuit_breakers_default_cx_pool_open | Gauge | Whether connection pool circuit breaker is open |
envoy_listener_downstream_cx_total | Counter | Total downstream connections |
envoy_server_live | Gauge | Server liveness (1=alive) |
Envoy exposes metrics through the admin interface (default http://envoy-ip:9901/stats) or stats sink (Prometheus format at http://envoy-ip:9901/stats/prometheus).
Configuring stats sink:
# Envoy config - enable Prometheus stats sink
stats_sinks:
- name: envoy.stat_sinks.statsd
typed_config:
"@type": type.googleapis.com/envoy.config.metrics.v3.StatsdSink
address:
socket_address:
address: 127.0.0.1
port_value: 9125
prefix: envoy.
# Or use Prometheus format (recommended)
admin:
address:
socket_address:
address: 0.0.0.0
port_value: 9901
The pitfall with Envoy metrics is cardinality management. Envoy tags every cluster and every endpoint by default. If you have many routes (say, hundreds of API paths), metric cardinality explodes. The solution is to use stats_config to filter out unnecessary tags:
stats_config:
stats_tags:
- tag_name: cluster_name
regex: "^cluster\\.((.+?)\\.)"
stats_matcher:
inclusion_list:
patterns:
- prefix: "cluster."
- prefix: "listener."
- prefix: "server."
- prefix: "http."
This keeps only cluster, listener, server, and http related metrics, cutting out a pile of unnecessary internal statistics.
Prometheus Scrape Configuration
Kong Scrape
# prometheus.yml - Kong scrape config
scrape_configs:
- job_name: 'kong'
metrics_path: /metrics
static_configs:
- targets:
- 'kong-gateway:8001' # Kong Admin API port
relabel_configs:
- source_labels: [__address__]
target_label: instance
replacement: 'kong-prod-01'
Kong 3.x separates Admin API and Proxy ports — make sure metrics_path points to the correct port. In Kong 2.x, the /metrics endpoint is on the Admin API (8001) by default.
APISIX Scrape
# prometheus.yml - APISIX scrape config
scrape_configs:
- job_name: 'apisix'
metrics_path: /apisix/prometheus/metrics
static_configs:
- targets:
- 'apisix:9091'
relabel_configs:
- source_labels: [__address__]
target_label: instance
replacement: 'apisix-prod-01'
Note that APISIX’s metrics port defaults to 9091 with path /apisix/prometheus/metrics, not the standard /metrics. Get this wrong and Prometheus will keep hitting 404s.
Envoy Scrape
# prometheus.yml - Envoy scrape config
scrape_configs:
- job_name: 'envoy'
metrics_path: /stats/prometheus
static_configs:
- targets:
- 'envoy-proxy:9901'
Envoy’s admin port defaults to 9901, with Prometheus-formatted metrics at /stats/prometheus. If your Envoy runs in Kubernetes, use Pod Monitor for auto-discovery:
# Kubernetes PodMonitor (kube-prometheus-stack)
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: envoy-gateway
namespace: monitoring
spec:
selector:
matchLabels:
app: envoy-gateway
podMetricsEndpoints:
- port: admin
path: /stats/prometheus
interval: 15s
Core Alert Rule Design
This is the most practical section. I’ve organized five alert layers by fault detection priority, from P0 to P4. You can copy these directly into your alert rule files.
Layer 1: Gateway Availability (P0)
groups:
- name: gateway-availability
rules:
# Gateway process alive
- alert: GatewayDown
expr: up{job=~"kong|apisix|envoy"} == 0
for: 1m
labels:
severity: critical
team: sre
annotations:
summary: "Gateway {{ $labels.instance }} unreachable"
description: "Prometheus can no longer scrape metrics from {{ $labels.instance }}, gateway may be down"
# Gateway 5xx error rate (exceeds 5% within 1 minute)
- alert: GatewayHighErrorRate
expr: |
sum(rate(kong_http_requests_total{status=~"5.."}[1m])) by (service)
/
sum(rate(kong_http_requests_total[1m])) by (service)
> 0.05
for: 2m
labels:
severity: critical
team: sre
annotations:
summary: "Kong gateway 5xx error rate exceeds 5%"
description: "Service {{ $labels.service }} 5xx error rate is currently {{ $value | humanizePercentage }}"
Why for: 2m instead of immediate alerting? Because transient spikes (GC pauses, connection resets) can briefly push error rates up before self-recovering. A 2-minute sustained condition filters out 80% of the noise.
Layer 2: Latency Anomaly (P1)
- name: gateway-latency
rules:
# Kong P99 latency exceeds 500ms
- alert: GatewayHighLatency
expr: |
histogram_quantile(0.99,
sum(rate(kong_latency_ms_bucket[5m])) by (service, le)
) > 500
for: 5m
labels:
severity: warning
team: sre
annotations:
summary: "Kong gateway P99 latency exceeds 500ms"
description: "Service {{ $labels.service }} P99 latency is currently {{ $value }}ms"
# Upstream latency dominant (Kong's own latency < 20% of total means bottleneck is upstream)
- alert: GatewayUpstreamLatencyDominant
expr: |
histogram_quantile(0.95,
sum(rate(kong_latency_ms_bucket{type="upstream"}[5m])) by (service, le)
)
/
histogram_quantile(0.95,
sum(rate(kong_latency_ms_bucket{type="total"}[5m])) by (service, le)
)
> 0.8
for: 10m
labels:
severity: info
team: sre
annotations:
summary: "Upstream latency accounts for over 80%, bottleneck is in backend services"
description: "Service {{ $labels.service }} upstream latency ratio is {{ $value | humanizePercentage }}, investigate backend services"
The GatewayUpstreamLatencyDominant alert is one I added. Many teams only look at total latency — when they see high latency, they investigate the gateway first and waste time before discovering the backend is slow. This alert splits latency: if upstream accounts for over 80%, the bottleneck isn’t at the gateway, pointing investigation directly at the backend. In practice, this cuts troubleshooting time in half.
Layer 3: Upstream Health (P2)
- name: gateway-upstream-health
rules:
# Unhealthy upstream node
- alert: GatewayUpstreamUnhealthy
expr: kong_upstream_target_health{state="UNHEALTHY"} == 1
for: 1m
labels:
severity: warning
team: sre
annotations:
summary: "Upstream target {{ $labels.target }} unhealthy"
description: "Upstream {{ $labels.upstream }} target {{ $labels.target }} is currently UNHEALTHY"
# Envoy cluster circuit breaker open
- alert: EnvoyCircuitBreakerOpen
expr: envoy_cluster_circuit_breakers_default_cx_pool_open == 1
for: 30s
labels:
severity: critical
team: sre
annotations:
summary: "Envoy cluster {{ $labels.cluster_name }} connection pool circuit breaker is open"
description: "Cluster connection pool is full, new requests are being rejected"
A circuit breaker opening means the upstream can no longer handle the load. This usually comes with a cascade effect — one slow service exhausts the connection pool, preventing requests to other healthy services. When this alert fires, immediately check upstream service status and connection pool configuration.
Layer 4: Traffic Anomaly (P3)
- name: gateway-traffic-anomaly
rules:
# Request volume drop (possible route config issue or total upstream failure)
- alert: GatewayTrafficDrop
expr: |
sum(rate(kong_http_requests_total[5m])) by (service)
<
sum(rate(kong_http_requests_total[5m] offset 1h)) by (service) * 0.3
for: 10m
labels:
severity: warning
team: sre
annotations:
summary: "Service {{ $labels.service }} traffic dropped over 70%"
description: "Current request volume is {{ $value }} compared to 1 hour ago"
# Request volume spike (possible attack or legitimate traffic peak)
- alert: GatewayTrafficSpike
expr: |
sum(rate(kong_http_requests_total[5m])) by (service)
>
sum(rate(kong_http_requests_total[5m] offset 1h)) by (service) * 3
for: 5m
labels:
severity: info
team: sre
annotations:
summary: "Service {{ $labels.service }} traffic increased over 200%"
description: "Current request volume is {{ $value }} times higher than 1 hour ago, verify if this is a legitimate traffic peak"
Traffic drops are more alarming than spikes. A spike might just be a business peak, but a drop almost always means something broke — either the route configuration was corrupted, or all upstreams went down.
Layer 5: Resource Levels (P4)
- name: gateway-resource
rules:
# Kong data plane connection count too high
- alert: KongHighConnections
expr: |
sum by (instance) (
kong_nginx_http_connections_total{state="active"}
) > 10000
for: 5m
labels:
severity: warning
team: sre
annotations:
summary: "Kong instance {{ $labels.instance }} active connections exceed 10000"
description: "Current active connections: {{ $value }}, scaling may be needed"
# APISIX etcd unreachable
- alert: APISIXEtcdUnreachable
expr: apisix_etcd_reachable == 0
for: 30s
labels:
severity: critical
team: sre
annotations:
summary: "APISIX cannot connect to etcd"
description: "APISIX node {{ $labels.instance }} cannot connect to etcd, configuration changes will not sync"
etcd connectivity loss is a critical failure. APISIX stores all configuration in etcd — losing connection means route rules can’t be updated and new configs can’t be pushed. While existing configs remain cached in memory and traffic forwarding continues temporarily, you’ve lost control of the gateway.
Grafana Dashboard Design
Dashboard Layout Principles
A good API gateway monitoring dashboard should let ops answer three questions at a glance: Is traffic normal? Is latency high? Are there errors?
I recommend a top-to-bottom four-row layout:
┌─────────────────────────────────────────────┐
│ Row 1: Total QPS, error rate, P99 latency, │ ← Global overview
│ active connections │
├─────────────────────────────────────────────┤
│ Row 2: Per-service QPS comparison + │ ← Who has issues
│ error rate comparison │
├─────────────────────────────────────────────┤
│ Row 3: Latency distribution heatmap + │ ← Latency details
│ latency percentile trends │
├─────────────────────────────────────────────┤
│ Row 4: Upstream health status + │ ← Underlying state
│ circuit breaker status + pool │
└─────────────────────────────────────────────┘
Core Panel Configuration
Row 1: Global Overview
# Total QPS (aggregated by gateway instance)
sum(rate(kong_http_requests_total[5m]))
# Error rate
sum(rate(kong_http_requests_total{status=~"5.."}[5m]))
/
sum(rate(kong_http_requests_total[5m]))
# P99 latency
histogram_quantile(0.99,
sum(rate(kong_latency_ms_bucket[5m])) by (le)
)
# Active connections
sum(kong_nginx_http_connections_total{state="active"})
Row 3: Latency Heatmap
Grafana’s Heatmap panel is excellent for showing latency distribution. Configure the Histogram buckets properly, and you can visually spot when latency starts drifting:
# Latency heatmap data source
sum(rate(kong_latency_ms_bucket[5m])) by (le, service)
A heatmap lets you see at a glance that “latency didn’t rise uniformly — a batch of requests suddenly slowed down.” This pattern is invisible on line charts but stands out clearly on heatmaps. The first time I used a heatmap to troubleshoot latency, I discovered that one service had normal P50 but tail latency spiked regularly at 14:00 every day. Turned out the backend was running a scheduled task that exhausted the thread pool.
Row 4: Upstream Health Table
Use a Table panel to display each upstream target’s health status:
kong_upstream_target_health
Add Value Mappings — map 1 to red UNHEALTHY and 0 to green HEALTHY. Much more intuitive than reading numbers.
Three Gateway Monitoring Comparison and Selection Guide
| Dimension | Kong | APISIX | Envoy |
|---|---|---|---|
| Metrics exposure | Prometheus plugin | Prometheus plugin | Admin API / Stats Sink |
| Metric richness | Medium (sufficient) | Medium (sufficient) | High (hundreds of metrics) |
| Latency granularity | Kong + upstream + total | Mostly total | Very fine (per cluster/route) |
| Label readability | UUID by default, needs config | prefer_name uses route names | Requires manual stats_filter |
| Connection pool monitoring | Nginx-layer metrics | Nginx-layer metrics | Built-in circuit breaker metrics |
| Config storage | PostgreSQL / Cassandra | etcd | xDS (Istio/Kubernetes) |
| Cardinality risk | Explodes with per_consumer | Moderate with many routes | High by default, must filter |
Selection recommendations:
- Kong: Conservative teams, existing Kong ops experience, uncomplicated metric needs. Kong’s plugin ecosystem is mature, but manage cardinality carefully with per_consumer
- APISIX: Teams prioritizing dynamic configuration and sensitive to config propagation latency. etcd’s watch mechanism makes config changes effective in seconds, but etcd itself needs HA and monitoring
- Envoy: Deep Service Mesh (Istio) usage, need fine-grained traffic observation. Envoy has the most complete metrics system but the steepest learning curve — suited for teams with dedicated SREs
Production Pitfalls
Pitfall 1: Kong Metric Cardinality Explosion
Last year, a business team onboarded 2,000+ consumers with per_consumer=true enabled. Prometheus storage jumped from 5GB/day to 47GB/day, and queries slowed to a crawl. Each consumer × each route × each status code generates a time series: 2,000 × 50 × 5 = 500,000 time series.
Fix: Disabled per_consumer. For consumer-level analysis, query gateway logs instead. If you truly need fine-grained metrics, use consumer_label_filter to keep only the Top N consumers.
Pitfall 2: APISIX Config Changes Not Synced to Monitoring
APISIX’s Prometheus plugin is configured at the route level. Once a colleague added a new route but forgot to attach the prometheus plugin — the new route’s traffic was completely invisible to monitoring until users reported occasional timeouts on that API.
Fix: Added a CI/CD check — all new routes must include the prometheus plugin configuration, otherwise the pipeline blocks. Also added an “unmonitored route detection” panel in Grafana that periodically compares the route list from APISIX Admin API against routes with metrics in Prometheus. The difference is the missing ones.
Pitfall 3: Envoy Stats Endpoint OOM Killed
Envoy exposes all statistics by default, including numerous internal counters. With many routes, the /stats/prometheus response can reach tens of MB. Each Prometheus scrape caused Envoy memory to spike, eventually getting OOM killed.
Fix: Used stats_matcher to expose only needed metric prefixes (cluster, listener, http, server). Response body dropped from 47MB to 3MB. Also increased scrape interval from 15s to 30s to reduce collection frequency.
Pitfall 4: One-Size-Fits-All Alert Thresholds
Initially, we set P99 > 500ms alerts for all services. Some batch APIs normally take 2-3 seconds, triggering alerts constantly. SREs got used to alert noise and ignored real incidents.
Fix: Grouped thresholds by service type. Online transaction: P99 > 200ms; query: P99 > 800ms; batch: P99 > 5s. Used service labels for conditional branching in Prometheus:
# Different latency thresholds by service type
- alert: GatewayHighLatencyOnline
expr: |
histogram_quantile(0.99,
sum(rate(kong_latency_ms_bucket{service=~"order-.*|payment-.*"}[5m])) by (service, le)
) > 200
and on(service)
kong_latency_ms_count{service=~"order-.*|payment-.*"} > 0
for: 5m
labels:
severity: warning
- alert: GatewayHighLatencyBatch
expr: |
histogram_quantile(0.99,
sum(rate(kong_latency_ms_bucket{service=~"report-.*|export-.*"}[5m])) by (service, le)
) > 5000
and on(service)
kong_latency_ms_count{service=~"report-.*|export-.*"} > 0
for: 10m
labels:
severity: warning
Pitfall 5: APISIX etcd Single-Point Failure Causing Config Loss
APISIX depends entirely on etcd for configuration. Once, an etcd cluster node ran out of disk space. APISIX continued forwarding traffic (configs cached in memory), but all new configurations were undeliverable. Worse, Prometheus’s apisix_etcd_reachable showed healthy — because APISIX was connected to a different etcd node that was alive but had inconsistent data.
Fix: Beyond monitoring apisix_etcd_reachable, also monitor etcd cluster health independently (etcd_server_has_leader, etcd_mvcc_db_total_size_in_bytes, etcd_server_proposals_failed_total). Don’t just trust the gateway’s “I can reach etcd” report — verify from the etcd side.
Automated Inspection Script
Passive alerts aren’t enough — active inspection catches issues before they escalate. Here’s a script that checks gateway key metrics:
#!/usr/bin/env python3
"""API gateway health inspection script.
Queries gateway key metrics via Prometheus API, outputs inspection report.
"""
import requests
import sys
from datetime import datetime
PROMETHEUS_URL = "http://prometheus:9090"
# Inspection config: name -> query -> threshold -> severity
CHECKS = [
{
"name": "Gateway Alive",
"query": 'up{job=~"kong|apisix|envoy"}',
"expect": 1,
"severity": "P0"
},
{
"name": "5xx Error Rate",
"query": """
sum(rate(kong_http_requests_total{status=~"5.."}[5m]))
/ sum(rate(kong_http_requests_total[5m]))
""",
"threshold": 0.01,
"severity": "P1",
"compare": "gt"
},
{
"name": "P99 Latency",
"query": """
histogram_quantile(0.99,
sum(rate(kong_latency_ms_bucket[5m])) by (le)
)
""",
"threshold": 500,
"severity": "P2",
"compare": "gt"
},
{
"name": "Unhealthy Upstream Count",
"query": 'count(kong_upstream_target_health{state="UNHEALTHY"} == 1)',
"threshold": 0,
"severity": "P2",
"compare": "gt"
},
{
"name": "Active Connections",
"query": 'sum(kong_nginx_http_connections_total{state="active"})',
"threshold": 8000,
"severity": "P3",
"compare": "gt"
},
]
def query_prometheus(query):
"""Execute PromQL query"""
resp = requests.get(
f"{PROMETHEUS_URL}/api/v1/query",
params={"query": query},
timeout=10
)
resp.raise_for_status()
data = resp.json()
if data["status"] != "success":
raise ValueError(f"Query failed: {data}")
return data["data"]["result"]
def run_checks():
"""Run all inspection items"""
print(f"\n{'='*60}")
print(f"API Gateway Health Inspection - {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
print(f"{'='*60}\n")
issues = []
for check in CHECKS:
try:
results = query_prometheus(check["query"])
if not results:
print(f"[SKIP] {check['name']}: No data")
continue
if check.get("expect") is not None:
value = float(results[0]["value"][1])
status = "OK" if value == check["expect"] else "FAIL"
print(f"[{status}] {check['name']}: {value}")
if status == "FAIL":
issues.append((check["severity"], check["name"], value))
elif check.get("threshold") is not None:
value = float(results[0]["value"][1])
if check.get("compare") == "gt":
exceeded = value > check["threshold"]
else:
exceeded = value < check["threshold"]
status = "OK" if not exceeded else "WARN"
print(f"[{status}] {check['name']}: {value:.2f} (threshold: {check['threshold']})")
if exceeded:
issues.append((check["severity"], check["name"], value))
except Exception as e:
print(f"[ERR] {check['name']}: {e}")
issues.append(("P0", check["name"], str(e)))
print(f"\n{'='*60}")
if issues:
print(f"Found {len(issues)} issue(s):")
for sev, name, val in issues:
print(f" [{sev}] {name}: {val}")
else:
print("All inspection items normal")
print(f"{'='*60}\n")
return len(issues) == 0
if __name__ == "__main__":
ok = run_checks()
sys.exit(0 if ok else 1)
This script runs hourly via cron, with results posted to the ops team channel. Compared to passive alerting, active inspection catches issues when metrics “start degrading but haven’t crossed the alert threshold yet.” For example, active connections slowly rising from 3,000 to 7,000 won’t trigger the 10,000 alert, but the inspection script flags it yellow: “connections are rising, time to investigate.”
Summary
API gateway monitoring isn’t “just install a Prometheus plugin and call it done.” Three key takeaways:
Layer your metric collection. Don’t just look at totals — split latency into gateway processing time and upstream response time, categorize error rates by status code, and distinguish connection states. Only by separating them can you quickly identify whether the bottleneck is at the gateway or the backend. In practice, using upstream latency ratio as an independent alert condition cuts troubleshooting time in half — it directly tells you “stop wasting time on the gateway, check the backend.”
Differentiate alert thresholds by service. Different service types have vastly different latency characteristics. Using the same threshold for online transactions and batch exports only creates noise. The most absurd configuration I’ve seen was a uniform P99 > 100ms alert for all services — a report export endpoint triggered alerts daily until SREs muted the channel, and then nobody saw the real incident when it happened.
Monitor configuration storage components independently. Whether you use Kong+PostgreSQL or APISIX+etcd, when storage fails the gateway won’t immediately die, but you lose control of it. Don’t just trust the gateway’s “I can reach storage” report — verify independently from the storage side. Metrics like etcd’s server_has_leader and PostgreSQL’s pg_is_in_recovery are far more reliable than the gateway’s reachability check.
One final note on dashboard design. A good monitoring dashboard isn’t about stacking chart quantity — it’s about answering “is the system normal?” in 5 seconds. The four-row layout — global overview, service comparison, latency details, underlying state — is the result of my iterative refinement, far more usable than the initial version with twenty-plus panels.
References & Acknowledgments
This article referenced the following materials during writing. Thanks to the original authors for their contributions:
- Kong Prometheus Plugin Documentation — Kong Inc., usage and configuration parameters for the Prometheus metrics plugin
- Apache APISIX Prometheus Plugin — Apache APISIX, Prometheus plugin documentation and metric descriptions
- Envoy Statistics Documentation — Envoy Proxy, statistics overview and configuration methods
- Ultimate Guide: 6 Core Metrics and Alert Practices for Kong API Gateway — CSDN, Kong monitoring core metrics analysis and alert rule design
- Envoy Gateway Monitoring and Operations: 10 Key Metrics and Best Practices — CSDN, Envoy Gateway monitoring metrics explained
- Prometheus Alerting in Practice: Writing Alert Rules, Using Alertmanager for Routing and Inhibition — CSDN, Prometheus alert rule writing and Alertmanager routing configuration