Synthetic Monitoring: Proactively Safeguarding User Experience

┌──────────────────────────────────────────────────────┐
│                Synthetic Monitoring Workflow           │
│                                                      │
│  ┌──────────────┐                                   │
│  │ Probe Node    │  ← Globally distributed nodes     │
│  │ Cluster       │                                   │
│  │ (Beijing/    │                                   │
│  │  Shanghai/   │                                   │
│  │  Overseas)   │                                   │
│  └──────┬───────┘                                   │
│         │                                            │
│         │ Execute probe scripts on schedule          │
│         ▼                                            │
│  ┌──────────────┐    ┌──────────────┐              │
│  │  Target       │    │  Record      │              │
│  │  System       │───→│  Results     │              │
│  └──────────────┘    │  • Status    │              │
│                      │  • Response  │              │
│                      │  • Content   │              │
│                      │  • Screenshot│              │
│                      │  • Trace     │              │
│                      └──────┬───────┘              │
│                             │                       │
│                             ▼                       │
│                      ┌──────────────┐              │
│                      │ Alerting +    │              │
│                      │ Dashboard     │              │
│                      └──────────────┘              │
└──────────────────────────────────────────────────────┘

Passive and synthetic monitoring complement each other:

Passive monitoring:
  → Answers "What is the system's current state?"
  → CPU 80%, QPS 1000, error rate 0.1%
  → Discovers symptoms of existing problems

Synthetic monitoring:
  → Answers "Can users use the system normally?"
  → Is the homepage accessible? Login working? Payment flow smooth?
  → Discovers problems before users do

Both combined = complete availability assurance

E-commerce critical paths:

1. Homepage access → page load < 3s
2. Product search → search results < 2s
3. Product detail → page render < 2s
4. Add to cart → API response < 1s
5. Checkout flow → full flow < 10s
6. Payment completion → payment callback < 5s

Each step is a critical path — any step failing affects conversion rate.

# API synthetic monitoring (Checkly / Grafana Synthetics)
steps:
  - name: "User Login"
    request:
      url: https://api.example.com/auth/login
      method: POST
      headers:
        Content-Type: application/json
      body:
        email: synthetic@test.com
        password: ${SECRET_PASSWORD}
    assertions:
      - status_code == 200
      - response_time < 2000
      - body.token != null

  - name: "Get User Profile"
    request:
      url: https://api.example.com/user/profile
      method: GET
      headers:
        Authorization: Bearer {{steps[0].body.token}}
    assertions:
      - status_code == 200
      - response_time < 1000

  - name: "Search Products"
    request:
      url: https://api.example.com/products/search?q=phone
      method: GET
      headers:
        Authorization: Bearer {{steps[0].body.token}}
    assertions:
      - status_code == 200
      - response_time < 2000
      - body.products.length > 0

// Playwright script — simulates user shopping flow
const { chromium } = require('playwright');

async function syntheticTest() {
    const browser = await chromium.launch();
    const page = await browser.newPage();

    try {
        // Step 1: Visit homepage
        await page.goto('https://www.example.com', { timeout: 10000 });
        await page.waitForLoadState('networkidle');
        const pageTitle = await page.title();
        assert(pageTitle.includes('Store'), 'Homepage title correct');

        // Step 2: Search for products
        await page.fill('#search-box', 'phone');
        await page.click('#search-button');
        await page.waitForSelector('.product-list .product-item', { timeout: 5000 });

        const productCount = await page.locator('.product-item').count();
        assert(productCount > 0, 'Search results not empty');

        // Step 3: Click product detail
        await page.click('.product-item:first-child');
        await page.waitForSelector('.product-detail', { timeout: 5000 });

        // Step 4: Add to cart
        await page.click('#add-to-cart');
        await page.waitForSelector('.cart-success', { timeout: 3000 });

        // Step 5: Proceed to checkout
        await page.click('#checkout');
        await page.waitForURL('**/checkout/**', { timeout: 5000 });

        console.log('✓ Synthetic monitoring passed: shopping flow normal');
        await page.screenshot({ path: 'success.png' });

    } catch (error) {
        console.error('✗ Synthetic monitoring failed:', error.message);
        await page.screenshot({ path: 'failure.png' });
        throw error;
    } finally {
        await browser.close();
    }
}

module.exports = syntheticTest;

┌─────────────────────────────────────────────────────┐
│                   Monitoring Layer Model              │
│                                                     │
│  Layer 4: Synthetic Monitoring (Active)              │
│  → Simulates user behavior, discovers "can users use it"│
│  → Discovers problems before users                   │
│                                                     │
│  Layer 3: Passive Monitoring (Metrics)               │
│  → Collects system metrics, discovers "is system healthy"│
│  → Real-time system state reflection                 │
│                                                     │
│  Layer 2: Log Monitoring (Logs)                      │
│  → Searches log content, discovers "what went wrong" │
│  → Provides troubleshooting evidence                 │
│                                                     │
│  Layer 1: Distributed Tracing (Traces)               │
│  → Traces request paths, discovers "where is the problem"│
│  → Pinpoints failure locations                       │
└─────────────────────────────────────────────────────┘

Synthetic monitoring detects slow homepage response (> 5s)
    │
    ▼
Review synthetic monitoring's Trace info
    │
    ├── DNS resolution 0.5s (normal)
    ├── TCP connection 0.2s (normal)
    ├── TLS handshake 0.3s (normal)
    ├── Server processing 3.5s (anomalous!)
    └── Page rendering 0.5s (normal)
    │
    ▼
Jump to distributed tracing (via TraceID correlation)
    │
    ├── API Gateway: 0.1s
    ├── User Service: 0.2s
    ├── Product Service: 2.8s ← Bottleneck!
    │   └── Database Query: 2.5s ← Root cause!
    └── Cart Service: 0.3s
    │
    ▼
Check Product Service logs
    │
    └── Found slow query: SELECT * FROM products WHERE ...
    │
    ▼
Fix: Add index / optimize SQL

Blind spots of single-region probing:

Probe node in Beijing
  → Beijing access normal ✓
  → Shanghai users may be unable to access ✗ (not visible)
  → Guangzhou users may experience slow access ✗ (not visible)
  → Overseas users may have DNS resolution errors ✗ (not visible)

Multi-region probing:

Probe nodes in Beijing, Shanghai, Guangzhou, overseas
  → Beijing access normal ✓
  → Shanghai access normal ✓
  → Guangzhou detects latency > 3s ⚠️
  → Overseas detects DNS resolution failure 🔴
  → Every region's user experience is covered

┌─── Beijing Probe Node ────────────┐
│  API tests + browser tests        │
│  → Probes national entry point    │
└────────────────────────────────────┘

┌─── Shanghai Probe Node ───────────┐
│  API tests + browser tests        │
│  → Probes East China entry        │
└────────────────────────────────────┘

┌─── Guangzhou Probe Node ──────────┐
│  API tests + browser tests        │
│  → Probes South China entry       │
└────────────────────────────────────┘

┌─── Overseas Probe Node ───────────┐
│  API tests + browser tests        │
│  → Probes overseas entry          │
└────────────────────────────────────┘

         │
         ▼
  ┌──────────────┐
  │  Centralized  │
  │  Storage      │
  │  (Prometheus) │
  └──────┬───────┘
         │
         ▼
  ┌──────────────┐
  │   Grafana    │
  │  Multi-region │
  │  comparison  │
  └──────────────┘

groups:
  - name: synthetic-multi-region
    rules:
      # All regions fail → critical
      - alert: SyntheticAllRegionsDown
        expr: |
          count by(target) (synthetic_probe_success == 0) >= 4          
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "All regions probe failed: {{ $labels.target }}"
          description: "4 regions simultaneously failed probing, service may be completely unavailable"

      # Most regions fail → critical
      - alert: SyntheticMostRegionsDown
        expr: |
          count by(target) (synthetic_probe_success == 0) >= 3          
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "Most regions probe failed: {{ $labels.target }}"

      # Single region fails → warning
      - alert: SyntheticSingleRegionDown
        expr: |
          synthetic_probe_success == 0
          and on(target)
          count by(target) (synthetic_probe_success == 0) < 3          
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Single region probe failed: {{ $labels.target }} ({{ $labels.region }})"
          description: "Only {{ $labels.region }} failed probing, likely a regional network issue"

      # High latency variance between regions
      - alert: SyntheticLatencyVariance
        expr: |
          (max by(target) (synthetic_probe_duration_seconds) -
           min by(target) (synthetic_probe_duration_seconds)) > 3          
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High latency variance between regions: {{ $labels.target }}"
          description: "Difference between fastest and slowest region exceeds 3 seconds"

# GitHub Actions integrated with synthetic monitoring
name: Synthetic Monitoring

on:
  schedule:
    - cron: '*/5 * * * *'  # Every 5 minutes
  workflow_dispatch:        # Manual trigger

jobs:
  synthetic-test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        region: [beijing, shanghai, guangzhou]
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm install playwright @playwright/test

      - name: Run synthetic test
        env:
          REGION: ${{ matrix.region }}
          BASE_URL: ${{ vars[format('BASE_URL_{0}', matrix.region)] }}
        run: node synthetic-tests/critical-path.js

      - name: Upload screenshots on failure
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: failure-screenshots-${{ matrix.region }}
          path: screenshots/

// synthetic-tests/critical-path.js
const { chromium } = require('playwright');
const { PrometheusPushgateway } = require('prom-client');

const pg = new PrometheusPushgateway('http://pushgateway:9091');

async function runSyntheticTest() {
    const startTime = Date.now();
    const browser = await chromium.launch({
        args: ['--no-sandbox']
    });

    let success = 1;
    let errorMessage = '';

    try {
        const page = await browser.newPage();

        // Inject performance monitoring
        await page.route('**/*', async (route) => {
            const response = await route.fetch();
            const timing = Date.now() - startTime;
            console.log(`${route.request().url()}: ${response.status()} (${timing}ms)`);
            route.fulfill({ response });
        });

        // Step 1: Visit homepage
        const homeStart = Date.now();
        await page.goto(process.env.BASE_URL, { timeout: 15000 });
        await page.waitForLoadState('networkidle');
        const homeDuration = (Date.now() - homeStart) / 1000;

        if (homeDuration > 3) {
            throw new Error(`Homepage loading too slow: ${homeDuration}s`);
        }

        // Step 2: Verify key elements
        const heroSection = await page.$('.hero-section');
        if (!heroSection) {
            throw new Error('Homepage key element missing');
        }

        // Step 3: Test search functionality
        const searchStart = Date.now();
        await page.fill('#search-input', 'test product');
        await page.click('#search-submit');
        await page.waitForSelector('.search-results', { timeout: 5000 });
        const searchDuration = (Date.now() - searchStart) / 1000;

        if (searchDuration > 2) {
            throw new Error(`Search response too slow: ${searchDuration}s`);
        }

        console.log('✓ All synthetic monitoring passed');

    } catch (error) {
        success = 0;
        errorMessage = error.message;
        console.error('✗ Synthetic monitoring failed:', errorMessage);

        // Save screenshot
        const page = await browser.newPage();
        await page.goto(process.env.BASE_URL);
        await page.screenshot({ path: `screenshots/failure-${Date.now()}.png` });
    } finally {
        await browser.close();
    }

    // Push metrics to Prometheus
    const duration = (Date.now() - startTime) / 1000;
    await pg.pushAdd({
        synthetic_probe_success: {
            value: success,
            labels: {
                target: process.env.BASE_URL,
                region: process.env.REGION
            }
        },
        synthetic_probe_duration_seconds: {
            value: duration,
            labels: {
                target: process.env.BASE_URL,
                region: process.env.REGION
            }
        }
    });

    if (success === 0) {
        process.exit(1);
    }
}

runSyntheticTest();

# Grafana Synthetic Monitoring configuration
probes:
  - id: probe-us-east
    region: us-east-1
  - id: probe-eu-west
    region: eu-west-1
  - id: probe-ap-south
    region: ap-south-1

checks:
  - name: "Homepage HTTP"
    type: HTTP
    config:
      url: https://www.example.com
      method: GET
      headers:
        User-Agent: "Grafana-Synthetic"
    assertions:
      - status_code == 200
      - response_time < 3000
    frequency: 60s
    probes: [probe-us-east, probe-eu-west, probe-ap-south]

  - name: "Login API"
    type: HTTP
    config:
      url: https://api.example.com/auth/login
      method: POST
      body: '{"email":"test@example.com","password":"***"}'
    assertions:
      - status_code == 200
      - response_time < 2000
    frequency: 300s
    probes: [probe-us-east, probe-eu-west]

  - name: "Checkout Flow"
    type: Browser
    config:
      script: |
        await page.goto('https://www.example.com');
        await page.fill('#search', 'phone');
        await page.click('#search-btn');
        await page.waitForSelector('.product-item');
        await page.click('.product-item:first-child #add-to-cart');
        await page.waitForSelector('.cart-success');        
    frequency: 600s
    probes: [probe-us-east]

# Prometheus config — Blackbox as synthetic monitoring
scrape_configs:
  - job_name: 'synthetic-api'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://www.example.com              # Homepage
          - https://api.example.com/health        # API health check
          - https://api.example.com/v1/products   # Product API
        labels:
          synthetic: 'true'
          type: 'api'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115

What are your synthetic monitoring needs?

Only need API/TCP probing?
  ├── Yes → Blackbox Exporter (free, sufficient)
  └── No → Need browser testing?

Need browser testing?
  ├── Yes → Already have Grafana ecosystem?
  │         ├── Yes → Grafana Synthetic Monitoring
  │         └── No → Checkly (focused, good value)
  └── No → Only need API probing + global nodes?
            ├── Yes → Pingdom (simple, reliable)
            └── No → Already using Datadog/New Relic?
                      ├── Yes → Use platform's built-in Synthetics
                      └── No → Grafana Synthetic Monitoring

# Availability SLO (30-day window)
avg_over_time(synthetic_probe_success[30d]) * 100

# By target and region
avg by(target, region) (avg_over_time(synthetic_probe_success[30d])) * 100

# Response time SLO
histogram_quantile(0.95,
  sum by(le, target) (rate(synthetic_probe_duration_seconds_bucket[5m]))
)

# Success rate trend
avg_over_time(synthetic_probe_success[1h]) * 100

groups:
  - name: synthetic-slo
    rules:
      # Availability SLO breach (30-day < 99.9%)
      - alert: SyntheticAvailabilitySLOBreach
        expr: |
          avg by(target) (avg_over_time(synthetic_probe_success[30d])) < 0.999          
        for: 5m
        labels:
          severity: warning
          slo: availability-999
        annotations:
          summary: "Synthetic monitoring availability SLO breach: {{ $labels.target }}"
          description: "Past 30-day availability below 99.9%"

      # Response time SLO breach (P95 > 3s)
      - alert: SyntheticLatencySLOBreach
        expr: |
          histogram_quantile(0.95,
            sum by(le, target) (rate(synthetic_probe_duration_seconds_bucket[5m]))
          ) > 3          
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Synthetic monitoring latency SLO breach: {{ $labels.target }}"
          description: "P95 response time exceeds 3 seconds"

synthetic-tests/
├── api/
│   ├── health-check.js        # API health check
│   ├── auth-flow.js           # Login flow
│   ├── product-search.js      # Product search
│   └── checkout-flow.js       # Checkout flow
├── browser/
│   ├── homepage.js            # Homepage loading
│   ├── search-and-buy.js     # Search to purchase full flow
│   └── mobile-responsive.js   # Mobile responsiveness
├── config/
│   ├── environments.json      # Environment config
│   └── thresholds.json        # Threshold config
└── lib/
    ├── prometheus.js          # Metrics pushing
    └── alerting.js            # Alerting logic

# Identify synthetic monitoring traffic in applications
# Method 1: Special User-Agent
headers:
  User-Agent: "Synthetic-Monitor/1.0"

# Method 2: Special Header
headers:
  X-Synthetic-Monitor: "true"
  X-Synthetic-Test-ID: "checkout-flow-001"

# Method 3: Dedicated test account
auth:
  email: synthetic-monitor@example.com
  password: ${SYNTHETIC_PASSWORD}

// Exclude synthetic monitoring traffic
if (request.getHeader("X-Synthetic-Monitor") != null) {
    // Don't count in business metrics
    // Don't write to business logs
    // Don't trigger A/B tests
}

┌─────────────────────────────────────────────────────────┐
│              Synthetic Monitoring Dashboard               │
│                                                         │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐             │
│  │ Total    │  │ Success  │  │ Avg      │             │
│  │ Probes   │  │ Rate     │  │ Latency  │             │
│  │   8,640  │  │  99.5%   │  │  1.2s    │             │
│  └──────────┘  └──────────┘  └──────────┘             │
│                                                         │
│  ┌─────────────────────────────────────────────┐       │
│  │  Path Success Rates (30-day trend)           │       │
│  │  Homepage  ━━━━━━━━━━━━━━━━━ 100%           │       │
│  │  Search    ━━━━━━━━━━━━━━━━━ 99.8%          │       │
│  │  Login     ━━━━━━━━━━━━━━━━━ 99.5%          │       │
│  │  Checkout  ━━━━━━━━━━━━━━━━━ 98.2% ⚠️       │       │
│  │  Payment   ━━━━━━━━━━━━━━━━━ 99.9%          │       │
│  └─────────────────────────────────────────────┘       │
│                                                         │
│  ┌─────────────────────────────────────────────┐       │
│  │  Multi-Region Latency Comparison             │       │
│  │  Beijing   ━━━━━━━━━━━━ 0.8s                │       │
│  │  Shanghai  ━━━━━━━━━━━━━━ 1.0s              │       │
│  │  Guangzhou ━━━━━━━━━━━━━━━━━━ 1.5s          │       │
│  │  Overseas  ━━━━━━━━━━━━━━━━━━━━━━━━ 2.5s ⚠️│       │
│  └─────────────────────────────────────────────┘       │
│                                                         │
│  ┌──────────────────────┐ ┌──────────────────────┐    │
│  │ Recent Failures      │ │ SSL Certificate Days │    │
│  │ Target | Time | Error│ │ Domain | Days Left   │    │
│  └──────────────────────┘ └──────────────────────┘    │
└─────────────────────────────────────────────────────────┘

# Tiered alerting
- alert: SyntheticAPIFailure
  expr: synthetic_probe_success{type="api"} == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "API probe failed: {{ $labels.target }}"

- alert: SyntheticBrowserFailure
  expr: synthetic_probe_success{type="browser"} == 0
  for: 3m   # Browser tests allow more retry time
  labels:
    severity: warning
  annotations:
    summary: "Browser probe failed: {{ $labels.target }}"

- alert: SyntheticLatencyDegradation
  expr: |
    avg_over_time(synthetic_probe_duration_seconds[1h]) >
    2 * avg_over_time(synthetic_probe_duration_seconds[7d])    
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "Response time degradation: {{ $labels.target }}"
    description: "Current latency is more than 2x the 7-day average"