New Service OOMed 2 Hours After Launch: Replacing Manual Approval with a 7-Dimension Production Readiness Gate

Overview

1:47 AM. Phone buzzed. Alert channel exploded.

A newly launched recommendation service at a ride-hailing project went OOMKilled on all 3 Pods less than 2 hours after launch. Memory spiked from 800MB to the 2GB limit, the kernel killed the process, Kubernetes restarted the Pod, and it got killed again. Loop. Users were routed to the fallback path. P99 latency shot from 80ms to 2000ms.

The postmortem found the root cause. The dev team wrote comprehensive unit tests, passed code review, ran load tests — but nobody asked one basic question: Is this service actually “ready” for production?

“Ready” has a formal name in the SRE world: Production Readiness Review (PRR). Google defines it in Chapter 32 of the SRE Book as the first step before SRE takes on operational responsibility for a service — only after passing PRR does the SRE team agree to own that service.

Here is the problem. Traditional PRR is a document. An SRE engineer checks items one by one, fills out a form, signs off. That works when you have a handful of services. When an organization has 50+ microservices all iterating simultaneously, manual PRR becomes a deployment bottleneck. Dev teams complain “approval is too slow.” SRE teams complain “too many services to review.” Eventually PRR turns into a rubber stamp — sign here, deploy anyway, crash anyway.

This article covers how to turn PRR from a manual document into an automated gate — 7 dimensions of machine-checkable items replacing human judgment, making “ready” a gate on the CI/CD pipeline instead of a meeting in a conference room.

What Production Readiness Review Actually Is

Let me explain in plain terms.

Production Readiness Review is: a set of checks a service must pass before it receives real user traffic. The core question is — “if this service breaks in production, can we detect it, can we locate it, can we recover it?”

This is not code review. Code review asks “is the code correct.” PRR asks “can this service survive in production and be rescued when things go wrong.” A service with beautiful code but zero monitoring can pass code review but not PRR.

Google’s PRR model covers several core dimensions: architecture resilience, capacity planning, monitoring and alerting, disaster recovery, deployment process, security compliance. I expanded it to 7 dimensions in practice, because the two extras — operational ownership and cost visibility — turned out to be non-negotiable after I learned the hard way.

Why Manual PRR Fails

When I rolled out PRR at a ride-hailing project, I started with manual review. The SRE team wrote a 40-page checklist. Before each new service launch, the dev team filled the form, SRE reviewed line by line. Three weeks in, three problems surfaced:

First, check standard drift. Different SREs interpreted the same check item differently. Take “are resource limits configured reasonably” — one person thought 2 CPU / 4GB was reasonable, another insisted on load test data. Inconsistent standards meant non-repeatable review results.

Second, check timing lag. PRR typically happens after development, before launch. By then, architecture is locked in. If PRR finds an architectural issue (like all replicas in one availability zone), the fix is expensive. Google’s SRE Book emphasizes this too — the earlier SRE engages, the faster quality improves, and the better the final result (Related: SRE and Development Team Collaboration Models).

Third, check results expire. Services iterate rapidly after launch. Passing PRR once does not mean the tenth release still meets the bar. A service configures PDB (PodDisruptionBudget) on first launch, someone removes it in a later release, no re-PRR, and nobody finds out until a node maintenance event evicts all Pods and takes the service down.

The root cause: manual PRR is a one-time snapshot, but production environments change continuously. You need continuous checking, not a one-shot review.

7-Dimension Gate Design: From Document to Code

Turning PRR from document to code, the core idea is: automate everything machine-checkable, keep items requiring human judgment as explicit review points.

Below is the design rationale and pitfalls for each dimension.

Dimension 1: Architecture Resilience

Architecture resilience checks “will a single point of failure take the service down.” The most common anti-pattern: Deployment configured with 3 replicas, but all 3 land on the same node. Node goes down, service goes down.

Automated checks:

# PRR Architecture Resilience Check Rules
architecture_resilience:
  - id: ARC-001
    name: "Replica count >= 2"
    check: "kubectl get deploy {svc} -o jsonpath='{.spec.replicas}'"
    assert: ">= 2"
    severity: critical
    description: "Production services need at least 2 replicas for redundancy"
    
  - id: ARC-002
    name: "PDB configured"
    check: "kubectl get pdb {svc} -o name"
    assert: "not_empty"
    severity: critical
    description: "Must configure PodDisruptionBudget to prevent all Pods being evicted during node maintenance"
    
  - id: ARC-003
    name: "Topology spread constraints"
    check: "kubectl get deploy {svc} -o jsonpath='{.spec.template.spec.topologySpreadConstraints}'"
    assert: "not_empty"
    severity: warning
    description: "Recommend topologySpreadConstraints to avoid replicas concentrating in one AZ"
    
  - id: ARC-004
    name: "Anti-affinity configured"
    check: "kubectl get deploy {svc} -o jsonpath='{.spec.template.spec.affinity.podAntiAffinity}'"
    assert: "not_empty"
    severity: warning
    description: "Recommend podAntiAffinity to avoid replicas on the same node"

One pitfall here. I initially set ARC-003 (topology spread constraints) to critical. A bunch of existing services failed it, the gate blocked everything, and the dev team pushed back hard. I changed it to warning — existing services can be non-compliant temporarily, but new services must configure it. This compromise was necessary: the gate exists to surface problems, not to block releases.

When rolling this out, I added a mechanism for warning-level checks: they don’t block deployment, but automatically create a remediation ticket in the issue tracker, bound to the service owner, with a deadline. This way, problems don’t disappear into the void.

Results: 3 months in, across 120+ microservices, PDB coverage went from 31% to 96%, topology spread constraint coverage from 0 to 78%. Not through manual nagging — through the gate automatically exposing problems.

Dimension 2: Resource Quotas

Resource quota checks “are CPU and memory configs reasonable.” The OOM incident at the start? Root cause: dev team set 800MB memory request, 2GB memory limit, but actual peak usage hit 2.3GB — the limit capped at 2GB, exceeded, OOMKilled.

Automated checks:

import subprocess
import json

def check_resource_quota(service_name, namespace="prod"):
    """PRR Resource Quota Check: requests/limits validity + historical utilization"""
    issues = []
    
    # 1. Check if requests and limits are configured
    deploy_json = json.loads(
        subprocess.check_output(
            ["kubectl", "get", "deploy", service_name, 
             "-n", namespace, "-o", "json"]
        )
    )
    
    containers = deploy_json["spec"]["template"]["spec"]["containers"]
    for c in containers:
        resources = c.get("resources", {})
        requests = resources.get("requests", {})
        limits = resources.get("limits", {})
        
        # Must have both requests and limits
        if not requests.get("cpu") or not requests.get("memory"):
            issues.append({
                "id": "RES-001",
                "severity": "critical",
                "msg": f"Container {c['name']} missing cpu/memory requests"
            })
        if not limits.get("cpu") or not limits.get("memory"):
            issues.append({
                "id": "RES-002",
                "severity": "critical", 
                "msg": f"Container {c['name']} missing cpu/memory limits"
            })
        
        # 2. Check request/limit ratio
        # Request should be 50%-80% of limit
        # Too low: node oversubscription risk
        # Too high: resource waste
        if requests.get("memory") and limits.get("memory"):
            req_mem = parse_quantity(requests["memory"])
            lim_mem = parse_quantity(limits["memory"])
            ratio = req_mem / lim_mem
            if ratio < 0.5:
                issues.append({
                    "id": "RES-003",
                    "severity": "warning",
                    "msg": f"Container {c['name']} memory request/limit ratio "
                           f"{ratio:.0%}, below 50% recommended, "
                           f"may cause node oversubscription"
                })
    
    # 3. Query Prometheus for historical utilization
    # If actual usage approaches limit, the limit is too low
    utilization = query_prometheus_utilization(service_name)
    for container, stats in utilization.items():
        if stats["memory_p95"] > stats["memory_limit"] * 0.85:
            issues.append({
                "id": "RES-004",
                "severity": "critical",
                "msg": f"Container {container} memory P95 "
                       f"{stats['memory_p95']:.0f}MB exceeds 85% of limit "
                       f"(current limit: {stats['memory_limit']:.0f}MB)"
            })
    
    return issues

The key check here is RES-004 — predicting OOM risk from historical utilization. Traditional PRR only checks what is written in the config file, not what the service actually uses at runtime. I integrated Prometheus queries, pulling 7-day memory P95. If it exceeds 85% of the limit, that is critical.

Back to the opening OOM case. If this check existed, the gate would have caught it in CI: load test data showed peak memory at 2.3GB, limit set at 2GB, P95/limit ratio at 115%, far above the 85% threshold. The dev team would have seen “this service has OOM risk, adjust limit or optimize memory usage” in CI — not at 1:47 AM when the alert fired.

Dimension 3: Deployment Safety

Deployment safety checks “can you roll back quickly when a deploy goes bad.” While doing K8s migration at a new-energy logistics platform, the most common issue was dev teams configuring rolling updates without maxUnavailable. The default 25% means up to 25% of replicas can be unavailable during an update. A 3-replica service has 1 unavailable during update — if that coincides with a traffic spike, users feel the latency.

Automated checks:

deployment_safety:
  - id: DEP-001
    name: "RollingUpdate strategy configured"
    check: "kubectl get deploy {svc} -o jsonpath='{.spec.strategy}'"
    assert: "type == 'RollingUpdate'"
    severity: critical
    
  - id: DEP-002
    name: "maxUnavailable <= 25%"
    check: |
      kubectl get deploy {svc} -o \
      jsonpath='{.spec.strategy.rollingUpdate.maxUnavailable}'      
    assert: "<= 25%"
    severity: warning
    description: "Recommend maxUnavailable=0 or low value to ensure availability during updates"
    
  - id: DEP-003
    name: "Health probes configured"
    check: |
      kubectl get deploy {svc} -o \
      jsonpath='{.spec.template.spec.containers[0].readinessProbe}'      
    assert: "not_empty"
    severity: critical
    description: "Must configure readinessProbe, otherwise traffic routes to unready Pods"
    
  - id: DEP-004
    name: "readinessProbe and livenessProbe separated"
    check: |
      Check if readinessProbe.httpGet.path 
      differs from livenessProbe.httpGet.path      
    assert: "different_paths"
    severity: warning
    description: "Ready and live probes should check different endpoints"

DEP-004 was added after I got burned. An order service at a ride-hailing project kept restarting. Investigation found that dev had configured readinessProbe and livenessProbe on the same /health endpoint. When the database connection pool saturated, /health returned 500. readinessProbe removed the Pod from Endpoints (traffic stopped coming in), but livenessProbe also failed, so kubelet restarted the Pod. After restart, the pool was still saturated. Restart again. Death loop.

The correct approach: readinessProbe and livenessProbe check different dimensions. Readiness checks “can we accept traffic” (e.g., DB connectivity). Liveness checks “is the process alive” (e.g., a lightweight /live endpoint that just returns 200 without deep checks). The distinction between these probes also matters for K8s log collection strategy selection (Related: K8s Log Collection Strategy).

Dimension 4: Observability

Observability checks “can you detect and locate issues when things break.” A service without monitoring is driving blindfolded — you don’t know if it’s healthy until users complain.

Google SRE’s four golden signals — Latency, Traffic, Errors, Saturation — are the minimum standard for observability. But in practice, I found many teams “have monitoring” but are not “observable”: Prometheus scrapes tons of metrics, Grafana has tons of panels, but alert rules are misconfigured — either no alerts, or alert storms.

Automated checks:

def check_observability(service_name):
    """PRR Observability Check: Four Golden Signals + Alert Rules + Tracing"""
    issues = []
    
    # 1. Check if four golden signals metrics exist
    golden_signals = {
        "latency": f'histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{{service="{service_name}"}}[5m]))',
        "traffic": f'rate(http_requests_total{{service="{service_name}"}}[5m])',
        "errors": f'rate(http_requests_total{{service="{service_name}",status=~"5.."}}[5m])',
        "saturation": f'container_cpu_usage_seconds_total{{pod=~"{service_name}-.*"}}'
    }
    
    for signal, query in golden_signals.items():
        result = query_prometheus(query)
        if not result:
            issues.append({
                "id": "OBS-001",
                "severity": "critical",
                "msg": f"Missing {signal} metric, golden signals incomplete"
            })
    
    # 2. Check if alert rules exist
    alerts = query_alertmanager_rules(service_name)
    if not alerts:
        issues.append({
            "id": "OBS-002",
            "severity": "critical",
            "msg": "No alert rules for this service, failures cannot be auto-detected"
        })
    
    # 3. Check for Runbook links
    # Alerts must link to Runbooks, otherwise on-call has no guidance
    for alert in alerts:
        if not alert.get("annotations", {}).get("runbook_url"):
            issues.append({
                "id": "OBS-003",
                "severity": "warning",
                "msg": f"Alert {alert['name']} missing runbook_url annotation, "
                       f"on-call cannot quickly find handling guide"
            })
    
    # 4. Check if tracing is enabled
    traces = check_jaeger_service(service_name)
    if not traces:
        issues.append({
            "id": "OBS-004",
            "severity": "warning",
            "msg": "Tracing not integrated, cross-service call chains invisible"
        })
    
    return issues

OBS-003 — alerts must link to Runbooks — was added during alert system optimization. Previously, alert delivery took 5 seconds, fault localization dropped from 30 minutes to 8 minutes (Related: MTTR Optimization Practice), and the key was every alert linked to a Runbook. On-call sees the alert, immediately knows “what this alert means, what to check, how to handle it.” An alert without a Runbook is noise — you know something broke, but not what to do.

Dimension 5: Disaster Recovery

Disaster recovery checks “can you recover quickly when things break.” Key insight: backup is not recovery capability. Having a backup does not mean you can restore — an untested backup is an unverified assumption.

Automated checks:

recovery:
  - id: REC-001
    name: "Database backup frequency"
    check: "Query backup system API for last successful backup time"
    assert: "last_backup_age < 24h"
    severity: critical
    description: "Database should be backed up at least daily"
    
  - id: REC-002
    name: "Backup restore test"
    check: "Query restore test records for last successful restore test"
    assert: "last_restore_test_age < 30d"
    severity: warning
    description: "Recommend monthly restore drills to verify backup usability"
    
  - id: REC-003
    name: "Graceful termination configured"
    check: |
      kubectl get deploy {svc} -o \
      jsonpath='{.spec.template.spec.terminationGracePeriodSeconds}'      
    assert: ">= 30"
    severity: warning
    description: "Recommend terminationGracePeriodSeconds >= 30 for graceful shutdown"
    
  - id: REC-004
    name: "preStop hook configured"
    check: |
      kubectl get deploy {svc} -o \
      jsonpath='{.spec.template.spec.containers[0].lifecycle.preStop}'      
    assert: "not_empty"
    severity: warning
    description: "Recommend preStop hook to deregister from service discovery before exit"

REC-002 is the check I consider most essential but least implemented. When designing cross-datacenter failover, I set RPO under 5 minutes and RTO under 30 minutes (Related: Cross-Datacenter Failover Architecture). But whether that RTO is achievable depends on whether you have actually drilled the recovery process. An untested RTO is a wish, not a commitment.

Dimension 6: Security Compliance

Security compliance checks “are permissions controlled, are images clean.” A common trap: security review is not the same as security scanning. Scanning finds vulnerabilities. Review confirms security policies are in place.

Automated checks:

def check_security_compliance(service_name, namespace="prod"):
    """PRR Security Compliance Check"""
    issues = []
    
    # 1. NetworkPolicy configured
    np = kubectl_get_networkpolicy(service_name, namespace)
    if not np:
        issues.append({
            "id": "SEC-001",
            "severity": "warning",
            "msg": "No NetworkPolicy configured, inter-Pod network not isolated"
        })
    
    # 2. Image scanned
    image = get_deploy_image(service_name, namespace)
    scan_result = query_harbor_scan(image)
    if not scan_result or scan_result.get("scan_status") != "success":
        issues.append({
            "id": "SEC-002",
            "severity": "critical",
            "msg": f"Image {image} has not passed security scan"
        })
    elif scan_result.get("vulnerabilities", {}).get("critical", 0) > 0:
        issues.append({
            "id": "SEC-003",
            "severity": "critical",
            "msg": f"Image {image} has "
                   f"{scan_result['vulnerabilities']['critical']} critical vulnerabilities"
        })
    
    # 3. ServiceAccount permission check
    sa = get_service_account(service_name, namespace)
    if sa == "default":
        issues.append({
            "id": "SEC-004",
            "severity": "critical",
            "msg": "Using default ServiceAccount, should create dedicated SA with minimal RBAC"
        })
    
    # 4. Pod Security Standards
    pod_security = check_pod_security(service_name, namespace)
    if pod_security.get("privileged"):
        issues.append({
            "id": "SEC-005",
            "severity": "critical",
            "msg": "Container running in privileged mode, security isolation bypassed"
        })
    
    return issues

SEC-004 (using default SA) came from real experience. During MLPS 2.0 (China’s cybersecurity compliance standard) compliance at an e-commerce project, I found 80% of services used the default ServiceAccount. That means all services share the same permissions. Once one service is compromised, the attacker can access all cluster resources using that SA. MLPS 2.0 requires least-privilege — each service must have its own SA (Related: K8s RBAC and Security Contexts).

Dimension 7: Operational Ownership

Operational ownership checks “who to call when it breaks.” This dimension looks simplest but is the most frequently overlooked. When rolling out PRR, I encountered this scenario: alert fires, on-call sees “order-service CPU 100%” — and then what? Who owns this service? Is there a Runbook?

Automated checks:

ownership:
  - id: OWN-001
    name: "Service catalog registration"
    check: "Query service catalog API for service registration"
    assert: "registered"
    severity: critical
    description: "Service must be registered in service catalog with owner info"
    
  - id: OWN-002
    name: "Owner identified"
    check: "Query service catalog for non-empty owner field"
    assert: "owner_not_empty"
    severity: critical
    
  - id: OWN-003
    name: "Runbook link"
    check: "Query service catalog for non-empty runbook_url"
    assert: "runbook_not_empty"
    severity: critical
    description: "Every alert must link to a Runbook (see Alerting Strategy Design)"
    
  - id: OWN-004
    name: "On-Call rotation"
    check: "Query rotation system for service inclusion"
    assert: "in_rotation"
    severity: critical
    description: "Service must have a clear on-call owner"

OWN-004 connects directly to on-call rotation design. When designing on-call rotations, I found the biggest problem was not rotation rules — it was that some services were never put into rotation at all (Related: On-Call Rotation Design). Alerts fired for hours with nobody responding, because the service was not in the rotation system. Automatically checking “is this service in the rotation system” solves the problem at the root.

Gate Engine Implementation: A PRR Gate in Go

The 7 dimensions of checks need an engine to execute them. I wrote a PRR Gate in Go, embedded in the CI/CD pipeline, running all checks automatically before each deployment.

package main

import (
    "encoding/json"
    "fmt"
    "os"
)

// Severity check item severity level
type Severity string

const (
    Critical Severity = "critical"
    Warning  Severity = "warning"
    Info     Severity = "info"
)

// CheckResult check result
type CheckResult struct {
    ID       string   `json:"id"`
    Name     string   `json:"name"`
    Severity Severity `json:"severity"`
    Passed   bool     `json:"passed"`
    Message  string   `json:"message"`
    Evidence string   `json:"evidence"`
}

// PRRGate Production Readiness Review gate
type PRRGate struct {
    ServiceName string
    Namespace   string
    Checks      []Check
}

// Check interface
type Check interface {
    ID() string
    Name() string
    Severity() Severity
    Execute(svc, ns string) (CheckResult, error)
}

// Run executes all checks
func (g *PRRGate) Run() (*GateReport, error) {
    report := &GateReport{
        Service: g.ServiceName,
        Total:   len(g.Checks),
    }
    
    for _, check := range g.Checks {
        result, err := check.Execute(g.ServiceName, g.Namespace)
        if err != nil {
            result = CheckResult{
                ID:       check.ID(),
                Name:     check.Name(),
                Severity: check.Severity(),
                Passed:   false,
                Message:  fmt.Sprintf("Check execution failed: %v", err),
            }
        }
        report.Results = append(report.Results, result)
        
        if !result.Passed {
            if result.Severity == Critical {
                report.CriticalFail++
            } else {
                report.WarningFail++
            }
        }
    }
    
    // Gate policy: critical failures block deployment
    report.GatePassed = report.CriticalFail == 0
    return report, nil
}

// GateReport gate report
type GateReport struct {
    Service      string        `json:"service"`
    Total        int           `json:"total"`
    CriticalFail int          `json:"critical_fail"`
    WarningFail  int          `json:"warning_fail"`
    GatePassed   bool          `json:"gate_passed"`
    Results      []CheckResult `json:"results"`
}

func main() {
    gate := &PRRGate{
        ServiceName: os.Getenv("SERVICE_NAME"),
        Namespace:   os.Getenv("NAMESPACE"),
        Checks: registerChecks(),
    }
    
    report, err := gate.Run()
    if err != nil {
        fmt.Printf("PRR gate execution failed: %v\n", err)
        os.Exit(2)
    }
    
    output, _ := json.MarshalIndent(report, "", "  ")
    fmt.Println(string(output))
    
    if !report.GatePassed {
        fmt.Printf("\n❌ PRR gate failed: %d critical checks failed\n",
            report.CriticalFail)
        os.Exit(1)
    }
    
    if report.WarningFail > 0 {
        fmt.Printf("\n⚠️ PRR gate passed, but %d warnings need attention\n",
            report.WarningFail)
    } else {
        fmt.Println("\n✅ PRR gate all passed")
    }
}

Key design decisions:

First, pluggable checks. Each Check implements an interface, registered via registerChecks(). Adding a new dimension does not require engine changes — just a new Check implementation. When I rolled this out at a ride-hailing project, I started with 4 dimensions and expanded to 7 over time. Each addition was a new Check class, no core logic changes.

Second, tiered gate policy. Critical fails block deployment (exit 1). Warning fails report without blocking (exit 0). This matters — if every check blocked deployment, the gate becomes a bottleneck, and dev teams find ways around it. Tiering separates “must pass” from “should pass” and keeps the gate from being circumvented.

Third, JSON report output. Not for aesthetics — it lets the CI/CD platform parse results, auto-create tickets, and notify owners. I pushed reports to an internal dashboard where all services’ PRR status was visible at a glance.

Gate Position in CI/CD

# PRR Gate position in GitLab CI / Jenkins Pipeline
stages:
  - build
  - test
  - scan          # security scanning
  - prr-gate      # ← production readiness review gate
  - deploy-staging
  - integration-test
  - deploy-prod   # only reached if PRR passed
  - verify

prr-gate:
  stage: prr-gate
  image: prr-gate:latest
  variables:
    SERVICE_NAME: $CI_PROJECT_NAME
    NAMESPACE: prod
  script:
    - /app/prr-gate
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: on_success
  artifacts:
    reports:
      json: prr-report.json

The key is PRR Gate sits before deploy-staging. The service has not been deployed anywhere yet, and PRR checks first. If critical checks fail, the pipeline stops. The dev team sees exactly which items failed and how to fix them — in CI, not at 2 AM after launch.

7-Dimension Gate vs Traditional Manual PRR: Measured Comparison

I ran a data comparison 6 months after rolling out PRR Gate at a ride-hailing project:

Key findings:

Pass rate dropping is good. Manual PRR pass rate was 89%, looks healthy. In reality, it was because manual review standards were inconsistent and enforcement was lax. Automation dropped it to 67%, meaning 22% of services that “passed manual sign-off” actually failed production readiness standards. The gate caught them before launch, preventing post-deployment incidents.

Fix speed improved 2.3x. Because the automated gate gives specific, actionable check items — “missing PDB,” “memory P95 exceeds 85% of limit” — the dev team knows exactly what to fix. Manual PRR gave vague statements like “suggest improving disaster recovery plan” — the dev team had no idea where to start.

Review time dropped from days to seconds. The biggest efficiency gain. The SRE team of 3, each reviewing 8 services/month, was at capacity. After automation, no limit — 100 services running PRR Gate simultaneously was fine. SRE time freed up for higher-value work — chaos engineering, SLO framework setup.

Rollout Path: From 0 to 1 with PRR Gate

To roll out PRR Gate from scratch, I recommend 3 phases, one month each:

Month 1: Inventory baseline. Run all checks without blocking any deployment, just generate reports. The goal is to map the current state — how many services lack PDB, how many lack monitoring, how many lack Runbooks. This data is the most powerful weapon for driving improvement.

Month 2: New services mandatory. All newly launched services must pass PRR Gate. Existing services are not forced, but marked red on the dashboard. Dev teams start adapting to the gate’s existence and begin proactively meeting checks for new services.

Month 3: Full enforcement. All releases (including existing service iterations) must pass PRR Gate critical checks. Warning items do not block but create remediation tickets. Dev team resistance is expected here — the key message is “the gate is not here to block you, it is here to keep you from being woken up at 3 AM.”

The biggest resistance during rollout comes from “pass rate suddenly dropping” — services that used to get manual sign-off now fail machine checks. My strategy: run the gate in CI first without blocking, just reporting, for two weeks. Let everyone get used to seeing check items, then enforce. Sudden enforcement and gradual enforcement have very different outcomes.

Relationship with Error Budget Gates

PRR Gate and error budget gates are complementary, not substitutes.

PRR Gate checks “the service’s own configuration and operational readiness” — static, pre-checkable. Error budget gates check “the service’s current reliability state” — dynamic, based on runtime data.

Specifically: a service with all-green PRR Gate (config is fine) may still not be deployable — if its error budget is depleted, the service is currently unstable and should not receive new changes. Conversely, a service with ample error budget (currently stable) but red PRR Gate (e.g., PDB was removed) should also not deploy — config problems are latent risks that will eventually cause incidents.

I chained the two at a ride-hailing project: PRR Gate runs first, then error budget check. Both must pass before deployment. This reduced change-related incidents by 60% (Related: Replacing Manual CAB with Error Budget Gates).

Runtime PRR is the third layer — periodically scanning production for configuration drift. If a release removes PDB, the CI-stage PRR Gate will not catch it (it was changed after deployment), but runtime PRR will detect and alert. Three layers form defense in depth.

Summary

The essence of production readiness review is not filling out forms and signing off. It is answering one question: if this service breaks in production, can you rescue it.

The three fatal flaws of manual PRR — standard drift, timing lag, result expiration — stem from one-time snapshots in a continuously changing environment. The automated 7-dimension gate turns checks from documents to code, from one-time to continuous, from human judgment to machine-verifiable.

The 7 dimensions — architecture resilience, resource quotas, deployment safety, observability, disaster recovery, security compliance, operational ownership — cover the core scenarios of “can this service survive in production and be rescued when it breaks.” Every check item comes from real production incidents, not textbook theory.

The key to rolling out PRR Gate is not technical implementation — it is policy design: critical blocks, warning reports but creates tickets, existing services not forced but flagged red. The gate exists to make problems visible, trackable, fixable. If the dev team feels the gate is obstructing them, they will find ways around it — that is human nature, not an attitude problem.

One last honest statement: do not wait for the crash to do PRR. The OOM incident at the start — if the RES-004 check (memory P95 exceeds 85% of limit blocks deployment) existed, it would have been caught in CI. The 1:47 AM alert should never have fired.

References & Acknowledgments

This article referenced the following materials during writing. Thanks to the original authors for their contributions:

1. Production Readiness Is Easier to Inspect Than to Debate — Temitayo, systematically articulates the methodology of turning production readiness from debate into inspectable conditions. The “inspect rather than debate” approach in this article was inspired by this piece.
2. Top 8 SRE best practices for reliable delivery in 2026 — N-iX Solutions Architect Rafal Barzycki, 2026 SRE best practices covering SLO/error budgets/four golden signals/automation/progressive delivery. The observability dimension and error budget linkage in this article referenced methodologies from this piece.
3. Beyond Automation: Conducting a Production Readiness Assessment for a CI/CD Pipeline — Gravox, production readiness assessment from a CI/CD pipeline perspective. The health check layering design in the deployment safety dimension referenced this article.
4. Google SRE Book Chapter 32: The Evolution of SRE Engagement Models — Google SRE Team, defines the PRR model’s position in the service lifecycle. The PRR definition and “earlier engagement means better quality” principle in this article originate from this chapter.
5. Agentic CI/CD: Kubernetes Deployment Gates with Elastic MCP Server — Elastic Observability Labs, covers embedding deployment gates and resource health checks in CI/CD. The gate engine’s position in CI/CD design referenced this article.