Cache Hit Rate from 58% to 94%: Building a CDN Monitoring System and 5 Production-Grade Troubleshooting Decisions

Overview

After deploying a CDN, many teams stop at glancing at the hit rate number on the vendor’s console. Hit rate at 90%? Looks fine, moving on. Then one day the origin bandwidth gets maxed out, pages take 8 seconds to load, user complaints flood in — and you discover that 90% was the request hit rate, not the byte hit rate. Large files were all going back to origin, while small requests inflated the number.

This article breaks down a CDN monitoring system I’ve deployed in production. It’s not a pile of metrics — it’s organized around three core dimensions: cache hit rate, origin bandwidth, and edge node health. For each metric, I explain how to collect it, how to read it, and how to troubleshoot when things go wrong. Five real-world cases tie it together, each with specific data and decision logic.

Here’s the key difference between CDN monitoring and regular application monitoring: application components are things you deploy yourself with full control over metrics; CDN monitoring targets third-party or self-built edge networks where many metrics must be pulled from vendor APIs or computed from logs. The collection chain is longer, and troubleshooting has more noise.

1. Core Metric System for CDN Monitoring

1.1 Three Dimensions, Don’t Overdo It

CDN involves 30+ metrics, but only three categories truly matter for business decisions:

Many teams only look at hit rate, ignoring origin bandwidth. That’s like watching how many checkout lanes a supermarket has open without checking if the warehouse door is being emptied. A 95% hit rate sounds fine, but if the 5% of cache misses are all 50MB video files, origin bandwidth gets saturated regardless.

1.2 Request Hit Rate vs. Byte Hit Rate: 90% of People Get This Wrong

These two metrics are frequently confused, and vendor consoles typically display request hit rate by default — because the number looks better.

• Request hit rate = cache hit requests / total requests. A 2KB CSS cache hit and a 50MB video cache miss have the same weight.
• Byte hit rate = cache hit bytes / total transferred bytes. Large file misses are amplified, reflecting origin pressure more accurately.

Real case: An e-commerce platform experienced slow page loads. The console showed 92% hit rate — looked fine. Investigation revealed request hit rate was indeed 92%, but byte hit rate was only 58% — 300 product detail page hero images in 1080P were all going to origin, each 3-5MB, saturating origin bandwidth.

Nearly everyone doing CDN monitoring for the first time hits this trap. A pretty console number doesn’t mean everything’s fine. Request hit rate and byte hit rate must be monitored together.

My recommendation: Alert when byte hit rate drops below 80%, regardless of request hit rate. Low byte hit rate means large amounts of data are hitting origin, and origin bandwidth pressure won’t be small.

1.3 Collection Approach Comparison

CDN metric collection has three paths, each with its own fit:

In production, I recommend a combination: API pull for core metrics, log analysis for troubleshooting drill-down, probing for availability backup. Three approaches cover each other’s blind spots.

Related: Monitoring Data Governance: From Metrics Explosion to Precision Observability — If you’re struggling with “which CDN metrics to collect,” this metrics governance methodology can help.

2. Prometheus Collection Implementation

2.1 Pulling CDN Metrics from Vendor API

Major CDN vendors provide monitoring APIs, but Prometheus has no off-the-shelf CDN Exporter. You need to write a lightweight Exporter that periodically calls the API and converts results to Prometheus format.

Here’s the core logic for an Alibaba Cloud CDN Exporter (Go implementation). For other vendors, just swap the API calls:

package main

import (
    "fmt"
    "net/http"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

// CDN metric definitions
var (
    // Cache hit rate, label distinguishes request hit and byte hit
    cacheHitRate = prometheus.NewGaugeVec(
        prometheus.GaugeOpts{
            Name: "cdn_cache_hit_rate",
            Help: "CDN cache hit rate, hit_type=request|byte",
        },
        []string{"domain", "hit_type"},
    )
    // Origin QPS
    originQPS = prometheus.NewGaugeVec(
        prometheus.GaugeOpts{
            Name: "cdn_origin_qps",
            Help: "CDN origin requests per second",
        },
        []string{"domain"},
    )
    // Origin bandwidth in Mbps
    originBps = prometheus.NewGaugeVec(
        prometheus.GaugeOpts{
            Name: "cdn_origin_bps",
            Help: "CDN origin bandwidth in Mbps",
        },
        []string{"domain"},
    )
    // Edge node HTTP status code distribution
    edgeStatus = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "cdn_edge_status_total",
            Help: "CDN edge response status codes",
        },
        []string{"domain", "code"},
    )
)

func init() {
    prometheus.MustRegister(cacheHitRate, originQPS, originBps, edgeStatus)
}

// Pull CDN monitoring data every 5 minutes
// Note: Alibaba Cloud CDN API has a 5-minute data delay; pulling more frequently is pointless
func collectCDNMetrics() {
    domains := getCDNDomains() // Read accelerated domain list from config
    for _, domain := range domains {
        // Call Alibaba Cloud CDN API for monitoring data
        // Time window: last 5 minutes
        endTime := time.Now().UTC().Format("2006-01-02T15:04:00Z")
        startTime := time.Now().Add(-5 * time.Minute).UTC().Format("2006-01-02T15:04:00Z")

        // Request hit rate
        reqHitRate := callCDNAPI(domain, "RequestHitRate", startTime, endTime)
        cacheHitRate.WithLabelValues(domain, "request").Set(reqHitRate)

        // Byte hit rate
        byteHitRate := callCDNAPI(domain, "TrafficHitRate", startTime, endTime)
        cacheHitRate.WithLabelValues(domain, "byte").Set(byteHitRate)

        // Origin QPS and bandwidth
        originQPSVal := callCDNAPI(domain, "OriginQPS", startTime, endTime)
        originQPS.WithLabelValues(domain).Set(originQPSVal)

        originBpsVal := callCDNAPI(domain, "OriginBps", startTime, endTime)
        originBps.WithLabelValues(domain).Set(originBpsVal)
    }
}

func main() {
    // Collect every 5 minutes, aligned with API data delay
    go func() {
        ticker := time.NewTicker(5 * time.Minute)
        for {
            select {
            case <-ticker.C:
                collectCDNMetrics()
            }
        }
    }()
    // Initial collection
    collectCDNMetrics()

    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":9105", nil)
}

Key design decisions:

1. 5-minute collection interval, not the common 15 seconds. CDN API returns aggregated data with a minimum granularity of 5 minutes. More frequent pulls just trigger API rate limits.
2. Domain as label, not node. One domain has hundreds of edge nodes behind it. Node-level granularity creates data explosion and is meaningless — when troubleshooting, you care about “this domain’s hit rate dropped,” not “Guangzhou Telecom node 03’s hit rate dropped.”
3. Request hit rate and byte hit rate as separate metrics, distinguished by hit_type label. Both need individual alerting.

2.2 Probing Supplement: Using Blackbox Exporter for CDN Reachability

API-pulled metrics have a 5-15 minute delay. If an edge node goes down, you won’t know for 15 minutes? Unacceptable.

Add a probing layer: use Blackbox Exporter to probe CDN domains every 30 seconds, monitoring HTTP status codes and response time.

# prometheus.yml probing config
scrape_configs:
  - job_name: 'cdn-probe'
    scrape_interval: 30s
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://cdn.example.com/health-check.html  # Health check file on CDN
        - https://www.example.com/                   # Direct origin comparison
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

Probing doesn’t replace API metrics — it provides “detect within 1 minute” fast alerting. API metrics handle trend analysis and 5-minute-level alerting; probing handles second-level reachability alerting.

Related: Blackbox Exporter: External Probing and Uptime Monitoring — Probing configuration details can be found here.

2.3 Origin-Side Supplementary Monitoring

CDN vendor APIs tell you “origin QPS is X, origin bandwidth is Y,” but not whether the origin can handle it. Add a monitoring layer on the origin Nginx, cross-referencing with CDN API data:

# Nginx log marking CDN origin pull requests
# Identify CDN requests via $http_x_forwarded_for
log_format cdn_origin '$status $body_bytes_sent $request_time '
                      '$http_x_forwarded_for $remote_addr '
                      '$upstream_cache_status $request_uri';

server {
    listen 443 ssl;
    server_name origin.example.com;

    # Only log CDN origin requests
    access_log /var/log/nginx/cdn_origin.log cdn_origin;

    location / {
        proxy_pass http://backend;
        # Add cache status header for CDN origin responses
        add_header X-Origin-Cache-Status $upstream_cache_status;
    }
}

Pair with mtail or a custom Go script to parse Nginx logs, pushing origin response times and status code distributions to Prometheus. This data answers “CDN says origin QPS is 200, but origin Nginx sees 800” — the gap is usually caused by cache purges, prefetch, or CDN config changes triggering burst origin pulls.

3. Grafana Dashboard Design

3.1 Three-Layer Dashboard Structure

CDN monitoring dashboards shouldn’t be crammed onto one big screen. Split into three layers:

3.2 Core Panel PromQL

// Byte hit rate trend (last 1 hour)
cdn_cache_hit_rate{hit_type="byte"}

// Origin bandwidth ratio (origin bandwidth / total bandwidth)
cdn_origin_bps / (cdn_origin_bps + cdn_edge_bps) * 100

// Origin QPS spike detection (current vs 1-hour average)
cdn_origin_qps > (avg_over_time(cdn_origin_qps[1h]) * 2)

// 5xx error rate
rate(cdn_edge_status_total{code=~"5.."}[5m]) /
rate(cdn_edge_status_total[5m]) * 100

// Probe response time P95
histogram_quantile(0.95, rate(probe_duration_seconds_bucket[5m]))

3.3 An Overlooked Panel: Hit Rate and Origin Bandwidth Correlation

Looking at the hit rate curve alone reveals nothing; same for origin bandwidth alone. Overlay both on one chart, and anomalies become obvious.

Normal: hit rate rises → origin bandwidth falls, two curves mirror each other. Abnormal: hit rate unchanged → origin bandwidth spikes, meaning the origin pulls aren’t regular requests but large file pulls. Hit rate didn’t drop, but byte hit rate did.

This panel helped me locate 3 instances of “hit rate unchanged but origin bandwidth exploded.” Each root cause was large files not being cached.

3.4 Cost Monitoring Panel: CDN Billing Model Determines Monitoring Focus

CDN billing directly impacts which metrics to watch:

With 95th percentile billing, one bandwidth spike from a large file prefetch could double your monthly bill.

Add a “bandwidth peak tracker” panel showing where the current 5-minute bandwidth ranks in the monthly chart. If approaching the top 5%, investigate for abnormal origin traffic immediately.

Cost monitoring isn’t about saving money — it’s about preventing runaway costs. CDN traffic costs can 3-5x during major sales. Without a monitoring panel, you have no idea where the money went, let alone what’s normal growth vs. wasted spend from cache strategy failures.

4.5. CDN Cache Strategy Configuration: The Foundation of Monitoring

No matter how good your monitoring is, if the cache strategy is wrong, metrics only tell you “how bad it is.” Get the cache strategy right first, then monitoring becomes meaningful.

Cache Time Tiering

Different resource types need vastly different cache times. Here’s my production cache strategy template:

Critical pitfall: Don’t set cache time for API endpoints. Some teams set a blanket 1-hour cache for convenience, and user sessions, inventory data all get cached by CDN — returning another user’s data. API endpoints must have Cache-Control: no-cache or be configured as “do not cache dynamic paths” on the CDN.

Cache-Control Headers: Origin and CDN Must Agree

CDN node caching behavior is controlled by two layers: CDN console cache rules and origin Cache-Control response headers. When they conflict, different vendors handle it differently — some prioritize CDN config, some prioritize origin headers.

Pitfall case: Set 30-day cache for .pdf on CDN console, but origin Nginx accidentally returned Cache-Control: no-store. Edge nodes didn’t cache PDFs at all; every request went to origin. Hit rate looked fine (low request volume), but origin bandwidth kept climbing.

Troubleshooting method: Use curl -v to check response headers:

# Check origin response headers
curl -v -o /dev/null "https://origin.example.com/static/doc.pdf" 2>&1 | grep -i "cache-control"

# Check CDN edge response headers
curl -v -o /dev/null "https://cdn.example.com/static/doc.pdf" 2>&1 | grep -i "cache-control\|x-cache\|age"

If origin returns no-store or no-cache, CDN likely won’t cache it. My recommendation: Set Cache-Control by Content-Type on origin Nginx, don’t let the application layer manage it — a casual no-cache in application code is invisible to you.

# Origin Nginx unified cache strategy by Content-Type
location ~* \.(jpg|jpeg|png|gif|webp|svg)$ {
    add_header Cache-Control "public, max-age=604800";  # 7 days
}

location ~* \.(css|js)$ {
    add_header Cache-Control "public, max-age=2592000";  # 30 days
}

location ~* \.(pdf|mp4|zip)$ {
    add_header Cache-Control "public, max-age=2592000";  # 30 days
}

# API paths not cached
location /api/ {
    add_header Cache-Control "no-cache, no-store, must-revalidate";
}

Cache Key Rules: Query Parameters Are the #1 Killer

This was mentioned in Case 1, but deserves its own treatment — because query parameter-induced cache hit rate drops are extremely common.

CDN defaults to using the full URL (including query parameters) as the cache key. This means /image.jpg?v=1 and /image.jpg?v=2 are two separate cache entries. If your URLs carry parameters that don’t affect content, hit rates drop off a cliff:

Alibaba Cloud CDN: Console → Cache Configuration → Cache key → “Ignore parameters.” Tencent Cloud: cache key rule settings. Only ignore parameters that don’t affect content — don’t ignore everything — if pagination and language parameters get ignored, users see other people’s data.

5. Five Production-Grade Troubleshooting Decisions

The following 5 cases are all from real production environments. Each starts with the symptom, then the investigation process, and finally the decision.

5.1 Case 1: Byte Hit Rate 58%, Request Hit Rate 92%

Symptom: Pre-sale load testing for an e-commerce platform. Console showed 92% request hit rate, but origin bandwidth stayed at 800Mbps. Nginx logs showed heavy origin pulls.

Investigation:

Step 1: Confirm whether it’s request hit rate or byte hit rate. Console default was request hit rate at 92%; switching to byte hit rate showed only 58%. The gap came from 1080P product hero images — each 3-5MB, all going to origin.

Step 2: Check origin URL distribution. Group Nginx logs by $request_uri for Top 20 origin pull URLs:

# Extract origin pull URLs, group by path prefix
awk '{print $7}' /var/log/nginx/cdn_origin.log \
  | sed 's/\?.*//' \
  | sort | uniq -c | sort -rn | head -20

80% of origin pulls pointed to /static/images/products/ large images. CDN cache config had .jpg cached for 1 hour, but URLs carried ?v=timestamp, causing full origin pulls on every version change.

Decision:

1. Ignore query parameters for cache key: ?v= version number doesn’t affect image content. Configure CDN cache key to “ignore all query parameters.”
2. Large image cache time from 1 hour to 7 days: Product images update infrequently. 7 days is enough. Purge on new release.
3. Origin URL monitoring panel: Add a Top 20 origin pull URL panel, so you don’t need to dig through logs next time.

After changes: byte hit rate went from 58% to 94%, origin bandwidth from 800Mbps to 120Mbps.

5.2 Case 2: Hit Rate Normal But Users Report Slowness

Symptom: Hit rate and origin metrics all normal, but users from multiple regions reported slow page loads. CDN vendor console showed everything fine.

Investigation:

CDN vendor console metrics are globally aggregated, masking individual region or ISP issues. Comparing probe data in Grafana: China Telecom probe latency P95 jumped from 30ms to 180ms, while China Unicom was normal.

Further investigation of CDN node distribution revealed a core node in the Telecom region was under maintenance. Traffic was rerouted to distant nodes, increasing physical distance and latency. The vendor’s aggregated metrics averaged out this regional issue.

Decision:

1. Group probe nodes by ISP: Previously only looking at global P95, now track Telecom/Unicom/Mobile separately.
2. Add latency divergence alert: Alert when same domain’s P95 latency differs by more than 100ms across ISPs.
3. Supplement with vendor node-level API data: Pull node monitoring API data into Prometheus for core node latency.

After this, similar issues were pinpointed within 3 minutes — “which ISP in which region has latency anomaly” — without waiting for user complaints.

5.3 Case 3: Hit Rate Crash After Cache Purge, Not Recovering

Symptom: After a new release, full CDN purge was executed. Hit rate dropped from 94% to 30%. Expected recovery in 30 minutes, but still at 40% after 2 hours.

Investigation:

Normal post-purge hit rate dips briefly, then recovers as requests repopulate cache. But recovery was too slow. Checking origin URLs revealed 200+ JS chunk files introduced by the new build, each with a unique URL. CDN nodes needed to fetch each one from origin.

Root cause: Frontend build split a single vendor.js into 200+ small chunks. After full purge, each chunk required one origin pull. CDN node cache population couldn’t keep up.

Decision:

1. Active prefetch after release: Don’t wait for user requests to trigger cache population. Call CDN vendor prefetch API to batch-push new release URLs to edge nodes.

#!/bin/bash
# Post-release batch prefetch script
# Extract URLs needing prefetch from build artifacts
DOMAIN="cdn.example.com"
BUILD_DIR="/data/releases/latest"

# Extract all static resource URLs
find "$BUILD_DIR" -type f \( -name "*.js" -o -name "*.css" -o -name "*.png" -o -name "*.jpg" \) \
  | sed "s|$BUILD_DIR|https://$DOMAIN|" \
  > /tmp/prefetch_urls.txt

# Call prefetch API (Alibaba Cloud example)
while IFS= read -r url; do
    echo "$url" >> /tmp/batch.txt
    if [ $(wc -l < /tmp/batch.txt) -eq 50 ]; then
        aliyun cdn PushObjectCache --ObjectPaths "$(cat /tmp/batch.txt | tr '\n' ',')" --Area mainland
        > /tmp/batch.txt
        sleep 2  # API rate limit protection
    fi
done < /tmp/prefetch_urls.txt

# Handle remaining URLs
if [ -s /tmp/batch.txt ]; then
    aliyun cdn PushObjectCache --ObjectPaths "$(cat /tmp/batch.txt | tr '\n' ',')" --Area mainland
fi

echo "Prefetch complete, $(wc -l < /tmp/prefetch_urls.txt) URLs"

2. Full purge → incremental purge: Only purge changed files. Build script outputs changed file list, purge only those.
3. Add purge impact alert: Alert if hit rate doesn’t recover to 70% within 1 hour of purge.

After changes: release flow went from “purge → wait 2 hours → hit rate recovers” to “incremental purge → prefetch → hit rate recovers to 90%+ within 10 minutes.”

5.4 Case 4: Intermittent Origin 5xx

Symptom: CDN edge 5xx rate fluctuated between 0.1%-0.5%, no pattern. Origin Nginx logs normal, no 5xx.

Investigation:

CDN 5xx but origin has no 5xx — the problem is on the CDN → origin pull link. Checking CDN origin logs, 5xx clustered around specific time windows, coinciding with origin Nginx keepalive_timeout expiry.

Root cause: CDN used HTTP Keep-Alive to reuse connections, but origin Nginx keepalive_timeout was 65 seconds while CDN connection reuse timeout was 60 seconds. The mismatch occasionally caused CDN to reuse a connection already closed by Nginx, resulting in origin pull failure.

Decision:

1. Unify Keep-Alive timeout: Origin Nginx keepalive_timeout from 65s to 90s, 30 seconds longer than CDN’s 60s, ensuring CDN never reuses an expired connection.

# Origin Nginx config
http {
    # Keep-Alive timeout must be 30+ seconds longer than CDN side
    # Otherwise CDN reuses connections already closed by origin, causing 5xx
    keepalive_timeout 90s;
    keepalive_requests 1000;
}

2. Add origin link monitoring: Add origin connection error rate metric to Exporter, cross-reference with Nginx logs.
3. Origin 5xx alert threshold from 1% to 0.3%: Intermittent 5xx also needs alerting. Don’t ignore because volume is small.

5.5 Case 5: Multi-CDN Metric Alignment

Symptom: Using two CDN vendors for failover. But their API hit rate metrics had different definitions — Vendor A’s “hit rate” was request hit rate, Vendor B’s was byte hit rate. Putting both numbers on the same dashboard made them incomparable.

Investigation:

This isn’t a technical problem, it’s a metric alignment problem. Two vendors define API fields differently. Without unified semantics, side-by-side data is misleading.

Decision:

1. Unify semantics at Exporter level: Regardless of vendor API field names, Exporter outputs hit_type=request and hit_type=byte labels. Vendor A’s HitRate maps to hit_type=request, Vendor B’s TrafficHitRate maps to hit_type=byte.
2. Add provider label: Each metric gets provider=a|b, making same-domain data from both vendors individually visible.
3. Failover decisions don’t rely on single metric: When switching, look at probe latency, 5xx rate, and hit rate together — not just hit rate.

5.6 Bonus: CDN Log Analysis — Metrics Show Trends, Logs Show Root Cause

Metrics tell you “hit rate dropped.” Logs tell you “which URLs are going to origin, how slow origin responses are, what the status code distribution is.” For troubleshooting, logs matter more than metrics.

# 1. Status code distribution
awk '{print $9}' cdn_log_20260924.txt | sort | uniq -c | sort -rn

# 2. Top 20 origin pull URLs (strip query params)
grep "MISS" cdn_log_20260924.txt \
  | awk '{print $7}' \
  | sed 's/\?.*//' \
  | sort | uniq -c | sort -rn | head -20

# 3. Hourly origin pull volume — find burst origin pull time windows
grep "MISS" cdn_log_20260924.txt \
  | awk '{print $4}' \
  | cut -d: -f1-2 \
  | sort | uniq -c

# 4. Slow origin pull URLs (>2 seconds)
awk '$10 > 2.0 {print $7, $10}' cdn_log_20260924.txt | sort -k2 -rn | head -20

# 5. Referer distribution for origin pulls — check for hotlinking
grep "MISS" cdn_log_20260924.txt \
  | awk '{print $11}' \
  | sort | uniq -c | sort -rn | head -10

Script the log analysis, run it on a schedule, output to Grafana panels or Feishu daily reports. Log analysis doesn’t need to be real-time — once a day is enough, for discovering previous day’s trend changes and anomaly patterns.

6. Pitfall Summary: 6 CDN Monitoring Anti-Patterns

Summary

CDN monitoring’s core isn’t about how many metrics you collect, but building actionable alerts on the right dimensions. Three takeaways from my years of experience:

1. Byte hit rate matters more than request hit rate. Request hit rate lies; byte hit rate doesn’t. Alert when byte hit rate drops below 80%.
2. Probing is a necessary supplement to API monitoring. API has 5-15 min delay; probing catches reachability issues within 1 minute. Both layers are indispensable.
3. Origin URL distribution is the most valuable troubleshooting panel. When hit rate drops, don’t guess — look at Top 20 origin URLs to see which resources aren’t cached.

Practical advice: if you’re just starting CDN monitoring, don’t build the full suite at once. Do three things first — byte hit rate monitoring, probe reachability, origin bandwidth alerting. These cover 80% of problem scenarios. Once stable, gradually add origin URL distribution, ISP dimensions, multi-vendor alignment.

CDN monitoring system isn’t a one-time project but a continuously tuned process as business changes. Cache strategy changed, new CDN vendor added, new domains, frontend build pipeline modified — each change can affect cache hit rate and origin patterns. The monitoring system must evolve alongside, or the next problem will leave you blind again.

References & Acknowledgments

The following resources were referenced during the writing of this article. Thanks to the original authors for their contributions:

1. What is a cache hit ratio? — Cloudflare, definition and formula for cache hit ratio, helped clarify the distinction between request hit rate and byte hit rate
2. CDN Cache Troubleshooting — Alibaba Cloud CDN official docs, troubleshooting process for cache misses and low hit rates. Case 1’s investigation path references these official recommendations
3. CDN Hit Rate Low? Cache Rules and Cache-Control Optimization — Tencent Cloud Developer Community, analysis of Cache-Control headers and cache key pollution, helped understand query parameter impact on hit rate
4. CDN Hit Rate Monitoring and Alerting System — Hostease, practical approach to CDN alerting and release coordination. The tiered alerting design in this article draws from their layered philosophy
5. Prometheus Production-Grade Monitoring Deep Practice — Tencent Cloud Developer Community, Prometheus layered collection architecture design. This article’s Exporter design borrows from their collection-storage separation approach