Waiting on Three Approvals to Fight a Fire at 3 AM: 6 Engineering Decisions for Production Break-Glass Access and Audit Trails

Overview

At 3:17 AM, my phone rang during a shift at an e-commerce platform. Payment success rate had dropped from 99.7% to 89%, and the order pipeline was starting to back up.

The on-call engineer quickly pinned down the root cause: during the previous canary rollout, a single network ACL rule had been deleted by mistake, so payment callback traffic could no longer reach the backend cluster. The fix itself was trivial — add the rule back, or revert that config commit. Three minutes of work.

But those three minutes got stuck on permissions. Write access to the production network device was held by three people: the project lead, the security lead, and the platform manager. We called all three. One phone was off, one person was on a plane, one was muted in a meeting in another city. By the time the third one called back and signed off, it was 3:57 AM.

Forty minutes had gone by, and the window to stop the bleeding had closed. That day’s MTTR stretched from the usual 8 minutes to 48 minutes, and when we did the post-mortem math, the economic loss from that delay dwarfed the cost of the rule itself.

In the retro, someone proposed the obvious fix: just give the on-call engineer standing access — wouldn’t that be faster? Someone else proposed the opposite extreme: pre-approve every change, no exceptions, nobody gets to bypass anything. I’ve seen both approaches in different projects, and I’ve seen both blow up.

The genuinely mature answer is the path between those two extremes — Zero Standing Access (ZSA) + Just-in-Time (JIT) elevation (granted only when needed, revoked the moment it expires) + a pre-staged break-glass emergency channel + end-to-end operational auditing.

The term break-glass comes from the fire box: a fire extinguisher behind glass that no one can reach day to day, but that anyone can use by smashing the glass when there’s a real fire. What it solves is “privileges must be tight under normal conditions, but in extreme conditions you can’t wait for approval.” And what operational auditing solves is “once you’ve smashed the glass, you have to be able to say exactly what you did with the extinguisher.”

I’ve rolled this system out across several rounds — MLPS 2.0 compliance remediation, a K8s migration, and multi-region production environments — converging unplanned elevation events from 40 a month down to 6. The potholes along the way were more numerous than I expected. This article breaks it into 6 engineering decisions:

• Zero standing access + JIT elevation: why a permanent root account must be converged, and how far to converge
• The break-glass channel must be pre-staged: credential sealing, an independent second channel, regular drills, and alert-on-use
• Time-boxed and least-privilege: a TTL of 15–30 minutes instead of 4–8 hours, plus a complete implementation of automatic revocation on expiry
• Trigger means broadcast: a tiered decision table where SEV1 skips approval and only broadcasts, while SEV3 goes through approval
• End-to-end audit trail: bastion host entry point, command-level auditing, session recording, and correlation with the incident timeline
• Mandatory post-incident review and expired-permission sweeps: review within 24 hours, and automatically surfacing permissions nobody remembered to revoke

If you’re working on privilege convergence, MLPS compliance, or building out an on-call rotation: on-call duty and incident classification are the other half of emergency response. This article covers the permissions-and-audit half. Only with both halves do you have a complete emergency response capability.

Zero Standing Access: Why a Permanent root Account Is the Biggest Technical Debt

Let me state one premise up front: the problem with emergency access isn’t rooted in the emergency — it’s rooted in the steady state. If nobody holds standing high privilege day to day, then “who can elevate during an outage” becomes a question worth designing for. If everyone is root all the time, then “elevation” isn’t the problem at all — the problem becomes “anyone can do whatever they want in production.”

The Cost of Standing Access

Standing access means an account holds production write permission long-term, and it’s there the moment you log in. Its hidden costs come in four layers:

1. Not attributable: when several people share an account, you can’t tell whether the admin in the audit log was Alice or Bob
2. A larger credential exposure surface: leaking account A puts your entire asset inventory at risk, not just the small slice it was meant to manage
3. Permission rot: someone did one network change three months ago, the high privilege they left behind never got revoked, and six months later they’ve transferred teams with the permission still attached
4. Compliance that doesn’t comply: MLPS 2.0 explicitly requires “least privilege” and “auditable operations,” and a standing root account satisfies neither

During a K8s migration of 120+ microservices at a new-energy logistics platform, I did the convenient thing first: I handed six people on the platform team a kubeconfig with cluster-admin. Three months of migration with zero incidents — but when we took inventory for a privilege audit, we found problems: two of the six kubeconfigs had been copied onto a shared directory on a jump host, and one token had never been rotated. The migration’s stability record was pristine; its permission hygiene was a disaster. It took two weeks to replace every cluster-admin with namespace-scoped Roles plus JIT elevation to make up that lesson.

Google emphasizes in Emergency Response that incident response needs clear authorization and well-defined operational boundaries, not “anyone can touch production.” That’s about the response process, but the underlying premise is that permissions must be controllable.

The ZSA Privilege Convergence Model

My convergence model has three layers, with privilege increasing and hold time decreasing from bottom to top:

L0 is the critical design decision. Many people assume that “tightening permissions” means engineers can’t see anything, so when something breaks they’re staring into the void and have to go find someone with access to query on their behalf — which is slower, not faster. The right move is to open up read-only access wide enough — logs, metrics, distributed tracing, read-only APIs all exposed, and only write permission tightened. In projects I’ve owned, opening production read-only access to all engineers cut fault localization time from 30 minutes to 8, because nobody had to “wait for someone to grant access to read the logs” anymore.

This layering is the same idea Microsoft lays out in RBAC best practices: bind permissions to roles, not to people, because roles are enumerable and reviewable, while people and ad-hoc groups inevitably drift. The only difference is that Microsoft’s guidance is about role assignment in a cloud identity system, while this is about converging production operations channels.

Related: From “@ Ops in Chat” to Self-Service: 6-Layer Architecture and 4 Landing Phases for Platformizing 20+ Operations is exactly about the L1 layer — packaging high-frequency, idempotent, low-risk operations into platform self-service so engineers click the button themselves instead of hunting someone down. L2 is the focus of this article.

Choosing Between Three Privilege Models

There are three mainstream approaches, and none of them is a silver bullet. This is the table I actually used to weigh the trade-offs during selection:

My recommendation: as long as your system has 7×24 on-call and an explicit availability target, use JIT + break-glass. Full pre-approval looks like the safest option, but in practice it breeds two bad behaviors: dressing up urgent changes as “routine changes” to get them approved in advance, or simply not fixing anything during an incident while waiting for approval. Both make the risk worse. Standing high privilege is only defensible in one scenario: a sandbox that is fully isolated, whose data can be rebuilt, and where a failure has zero blast radius. On production, I do not accept standing high privilege.

One thing has to be said plainly: JIT and break-glass are not the same thing, and many teams blur them together and end up doing neither well. Here’s the difference:

The design principle in one sentence: JIT is the daily commuter tool; break-glass is the fire axe. You don’t use a fire axe to open delivery boxes every day, but you also can’t discover in the middle of a fire that the case is empty.

The break-glass Channel Must Be Pre-Staged, Not Improvised

Iron rule number one: an emergency channel must be an essay you wrote beforehand, not one you compose in the exam hall.

The most classic failure I’ve seen goes like this: during an incident, the on-call engineer realized the primary SSO was down (many outages take the IdP down with them), so they tried to bypass SSO with a local account — and discovered the local emergency account had never been enabled, while its password lived in a departed colleague’s password manager. By the time they tracked down the account, reset the password, and confirmed the permission scope, an hour had passed.

Why Improvised Channels Always Fail

Because “building a channel” is itself an action that depends on a system that is very likely also down. The primary auth system fails, the network partitions, CI/CD is down, the bastion host is down — any of these can happen at the same time. An emergency channel has to be the one path that still works when all of those systems are down. Which means it must:

• Be independent of the primary auth system (not “another SSO cluster,” but a completely separate credential system)
• Be independent of the primary authorization path (no dependency on CI/CD, no dependency on a platform API)
• Be stored independently (not alongside other credentials in the same Vault — otherwise when Vault goes down, it goes down with it)

Credential Sealing (Sealed Envelope)

The core implementation of break-glass is called a “sealed envelope”: emergency credentials are sealed offline, nobody knows about them under normal conditions, using them requires breaking the seal, and the act of breaking the seal itself triggers an alert.

My approach in practice has three stages:

1. Credential generation: generate a high-entropy credential once (for example a 32-byte random password, or a short-lived certificate with an expirationSeconds value), never stored in a database and never shown to anyone
2. Sealing: encrypt the credential with GPG, store the ciphertext in an isolated, heavily monitored bucket, and have the decryption private key held in shards by 2–3 core members (Shamir’s Secret Sharing, e.g., 2-of-3)
3. Monitoring the seal: any read, download, or metadata access against the ciphertext object fires a real-time alert straight to the on-call channel and the security lead

The key here is the “sharding” in step 2: any single member can’t decrypt on their own; at least two have to cooperate. That is both technical two-person control and the “major operations require multi-person authorization” that MLPS compliance demands.

An Independent Second Channel

An independent channel isn’t just “another login entry point” — it includes the network path and the operational entry point:

• It does not go through the primary bastion host (which may also be down), but through a separate jump path
• The credential is not “superuser rights on the main account,” but a small set of pre-defined, specific permissions — for example “can only modify network ACLs and roll back K8s Deployments,” not cluster-admin

My recommendation: use an independent second channel for break-glass, rather than “loosening RBAC just a little.” The cost of loosening RBAC is that it stays loose — the extra permissions you granted for emergencies lie dormant 99% of normal time, get misused in daily work, and get exploited by attackers. An independent channel is “closed by default, opened in an emergency”; its existence doesn’t change the standing permission surface. The price is one extra step to go through the independent channel during an emergency, but that extra step buys you a clean steady state.

Regular Drills: An Undrilled Emergency Channel Is No Channel at All

This is the most expensive pothole I’ve hit in a real project: we deployed a thorough break-glass sealing mechanism and never used it for two years. Then, during a real drill, we opened the sealed envelope and found — the sealed credential had not been rotated for 18 months, and the emergency account it was bound to had been disabled long ago as part of a company reorganization.

What a drill needs to do is clear:

• At least once a quarter, walk the whole path yourself: unseal → log in → perform one harmless verification action → re-seal → rotate the credential
• Every drill verifies: whether the seal is intact, whether the private-key shards are all present, whether the credential is still valid, whether the target system is reachable, and whether the alert fires
• Force a credential rotation and re-seal after every drill (a drill is itself a “use,” and anything used must be replaced)

The script below drops straight into Cron to run an integrity check on the sealed credential:

#!/usr/bin/env bash
# file: seal-check.sh
# Purpose: check the integrity, validity period, and rotation age of break-glass sealed credentials
# Dependencies: gpg, jq, awscli (or s3cmd, depending on the storage backend), curl (for the alert webhook)
# Input: environment variables SEAL_BUCKET / SEAL_KEY / ALERT_WEBHOOK
# Output: prints the check result to stdout; on failure exits 1 and pushes to the alert webhook
# Suggested frequency: once a day. Why daily: credential rotation itself is infrequent, but accidents
# where "a sealed object was deleted or altered by mistake" need to be found fast, and a daily check
# keeps the maximum detection latency under 24 hours at low cost.

set -euo pipefail

SEAL_BUCKET="${SEAL_BUCKET:-sre-breakglass-seal}"
SEAL_KEY="${SEAL_KEY:-prod/emergency/envelope.gpg}"
ALERT_WEBHOOK="${ALERT_WEBHOOK:-}"
STALE_DAYS=180   # More than 180 days without rotation counts as a failure. Why 180: MLPS requires
                 # periodic rotation of key-type credentials, and emergency credentials are used
                 # extremely rarely; six months is a compromise that covers the common compliance
                 # interpretation without inflating the rotation cost too much.
EXIT=0

alert() {
  echo "[ALERT] $1"
  if [[ -n "${ALERT_WEBHOOK}" ]]; then
    # use --data-urlencode so special characters in the message don't break the JSON
    curl -sS -X POST -H 'Content-Type: application/json' \
      --data-urlencode "text=break-glass seal check failure: $1" \
      "${ALERT_WEBHOOK}" >/dev/null || true
  fi
  EXIT=1
}

# 1) Does the sealed object exist
if ! aws s3api head-object --bucket "${SEAL_BUCKET}" --key "${SEAL_KEY}" >/dev/null 2>&1; then
  alert "sealed object does not exist: s3://${SEAL_BUCKET}/${SEAL_KEY}"
  exit "${EXIT}"
fi

# 2) Read object metadata: last modified time + whether it is marked as the latest version
META=$(aws s3api head-object --bucket "${SEAL_BUCKET}" --key "${SEAL_KEY}" \
        --query '{lm:LastModified,ver:VersionId}' --output json)
LAST_MODIFIED=$(echo "${META}" | jq -r '.lm')
echo "sealed object last modified: ${LAST_MODIFIED}"

# 3) Days since the last rotation
LAST_TS=$(date -u -jf "%Y-%m-%dT%H:%M:%S" "${LAST_MODIFIED%%.*}" +%s 2>/dev/null || \
          date -u -d "${LAST_MODIFIED}" +%s)
NOW_TS=$(date -u +%s)
AGE_DAYS=$(( (NOW_TS - LAST_TS) / 86400 ))
echo "days since last rotation: ${AGE_DAYS}"
if (( AGE_DAYS > STALE_DAYS )); then
  alert "sealed credential has not been rotated for ${AGE_DAYS} days, exceeding the ${STALE_DAYS}-day threshold"
fi

# 4) Are all private-key shards present (shards live in different members' isolated storage; here we only verify the objects exist)
for i in 1 2 3; do
  if ! aws s3api head-object --bucket "${SEAL_BUCKET}" --key "shares/shard-${i}.gpg" >/dev/null 2>&1; then
    alert "private key shard missing: shard-${i}"
  fi
done

# 5) Verify that the ciphertext is decipherable (public-key trial only, no plaintext written to disk)
# Why check only "decipherability": a full decryption would produce plaintext material, and the check
# should never hold plaintext. Here we use gpg's --list-packets to verify the ciphertext structure is
# intact, without emitting any plaintext.
if ! gpg --list-packets <(aws s3 cp "s3://${SEAL_BUCKET}/${SEAL_KEY}" - 2>/dev/null) >/dev/null 2>&1; then
  alert "sealed ciphertext structure is abnormal; it may be corrupted"
fi

echo "check complete, exit code ${EXIT}"
exit "${EXIT}"

Alert on Use: Every Use Must Be a Visible Event

Any “read” against the sealed object must alert in real time, and it must reach multiple channels such that the same person cannot ignore it: the on-call channel, the security lead, and the incident management board. The reason is straightforward: using break-glass means “someone is about to make a production change while bypassing the normal permission system,” which is by itself one of the highest-severity events that requires awareness.

The first principle of incident communication in Managing Incidents is “let the right people know what’s happening.” A break-glass alert is really broadcasting “the permission boundary is being broken” as a first-class incident message.

Time-Boxed and Least-Privilege: A 15–30 Minute TTL With Automatic Revocation

With “alert on use” settled, the next decision is: how long, and how much scope?

Why the TTL Is 15–30 Minutes, Not 4–8 Hours

In an early version I set the TTL to 8 hours, reasoning that “it stays available for a whole shift and spares people from re-applying.” That decision cost me later: a sweep found 3 “zombie emergency accounts” that had lived on long-term — the incidents they belonged to had been retroed long ago, yet the permissions were still attached, and one of those accounts’ credentials showed up in an unauthorized scanning attempt.

The problem with 8 hours is threefold:

• It covers most of a working shift, effectively degrading into “quasi-standing access”
• Incident retros are usually essentially settled within 1–2 hours after the incident, and the cases where you genuinely need to keep changing things for 8 hours are rare
• Once someone forgets to revoke it (and people always forget), it becomes a zombie permission

After switching to 15–30 minutes, unplanned elevation events converged from 40 a month to 6. That number looks counterintuitive — tighten the permissions and elevation events actually go down? The reason is that with a short TTL, engineers tend to finish the job inside the elevation window rather than “apply first and keep it around just in case.” Permissions that get requested but never used drop sharply, and unrevoked permissions fall to nearly zero.

The same judgment shows up beyond operations channels. Microsoft’s Just-in-Time (JIT) VM access makes “grant on demand + close automatically on expiry” the core of JIT design, rather than “open long-term + audit afterwards” — which is essentially an admission that people will forget to revoke, so revocation must be the system’s job, not a human’s.

My recommendation: use a 15-minute TTL for routine emergency access, and 30 minutes for complex changes that span multiple hosts or require verification. I don’t recommend 4–8 hours, and I don’t recommend anything under 5 minutes either — a TTL that’s too short forces engineers to re-apply repeatedly mid-window, which just fragments the work and adds noise to the audit records. The test is simple: the TTL should be slightly longer than what the action actually takes, not a measure of how much you’d like to get done this shift.

Least Privilege: Tighten All Three Dimensions at Once

The TTL governs time; the scope governs space. The scope has to tighten along three dimensions simultaneously:

1. Object: a single host / a single namespace / a single resource name (use a resourceNames allowlist, not an entire API group)
2. Action: grant only the verbs needed to finish the current task (e.g., patch on deployment, no delete)
3. Identity: one issuance maps to exactly one person and one incident; never allow “one credential shared by several people”

Of the three, the third is the most easily overlooked and the most fatal — a shared credential throws away attributability and voids every audit investment you made upstream.

Full Implementation: A Time-Boxed Elevation and Auto-Revocation Service

The code below is a zero-external-dependency, directly runnable JIT authorization service written with the Python standard library (http.server + sqlite3), so you can stand it up on any machine in 30 seconds to validate the logic, then swap in your own backend as needed.

#!/usr/bin/env python3
# file: jit_grantd.py
"""
jit-grantd -- a minimally runnable time-boxed elevation (JIT) and auto-revocation service.

[Dependencies] Python 3.9+ standard library only; no pip packages required.
               In production, replace http.server with FastAPI/uvicorn and terminate mTLS at an
               upstream gateway, but the core "issue - expire - revoke - audit" state machine is
               identical to what's shown here.

[Run]
    python3 jit_grantd.py            # listens on 127.0.0.1:8787
[Input]
    POST /grant   {"subject":"alice","resource":"ns/payment","verbs":["patch"],
                   "ttl_seconds":900,"reason":"INC-20261027-0317","severity":"SEV1"}
    POST /revoke  {"grant_id":"g-xxxx"}
    GET  /grants  list currently active grants
[Output]
    JSON. Every operation is written to the audit_log table with who/what/when/why for post-hoc audit.
"""

import json
import sqlite3
import secrets
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import urlparse

DB_PATH = "jit_grants.db"
LISTEN = ("127.0.0.1", 8787)

# TTL bounds. Why clamp on the server side: client parameters can't be trusted, and if arbitrary TTLs
# were allowed, one fat-fingered 86400 would degrade it into standing access — so the bounds must be
# enforced server-side.
TTL_MIN, TTL_MAX = 300, 1800   # 5 minutes ~ 30 minutes


def init_db():
    con = sqlite3.connect(DB_PATH)
    con.executescript(
        """
        -- grants: currently active temporary grants
        CREATE TABLE IF NOT EXISTS grants (
            grant_id    TEXT PRIMARY KEY,
            subject     TEXT NOT NULL,      -- who
            resource    TEXT NOT NULL,      -- over what (least-privilege scope)
            verbs       TEXT NOT NULL,      -- what they can do (JSON array)
            issued_at   INTEGER NOT NULL,
            expire_at   INTEGER NOT NULL,   -- must be revoked on expiry
            severity    TEXT NOT NULL,
            status      TEXT NOT NULL DEFAULT 'active'  -- active/expired/revoked
        );
        -- audit_log: append-only; no UPDATE/DELETE allowed, to keep the audit chain intact
        CREATE TABLE IF NOT EXISTS audit_log (
            id        INTEGER PRIMARY KEY AUTOINCREMENT,
            ts        INTEGER NOT NULL,
            action    TEXT NOT NULL,        -- grant/revoke/expire
            grant_id  TEXT,
            subject   TEXT,
            detail    TEXT NOT NULL         -- JSON: contains reason and other "why" fields
        );
        CREATE INDEX IF NOT EXISTS idx_grants_expire ON grants(expire_at, status);
        """
    )
    con.commit()
    con.close()


def audit(con, action, grant_id, subject, detail):
    """Audit writes and business writes must share one transaction, so we never "grant access but leave no trace"."""
    con.execute(
        "INSERT INTO audit_log(ts, action, grant_id, subject, detail) VALUES (?,?,?,?,?)",
        (int(time.time()), action, grant_id, subject, json.dumps(detail, ensure_ascii=False)),
    )


def issue(subject, resource, verbs, ttl_seconds, reason, severity):
    ttl = max(TTL_MIN, min(TTL_MAX, int(ttl_seconds)))
    now = int(time.time())
    gid = "g-" + secrets.token_hex(6)
    con = sqlite3.connect(DB_PATH)
    with con:  # transaction: the business write and the audit write live and die together
        con.execute(
            "INSERT INTO grants(grant_id,subject,resource,verbs,issued_at,expire_at,severity)"
            " VALUES (?,?,?,?,?,?,?)",
            (gid, subject, resource, json.dumps(verbs), now, now + ttl, severity),
        )
        audit(con, "grant", gid, subject,
              {"resource": resource, "verbs": verbs, "ttl": ttl, "reason": reason})
    con.close()
    return {"grant_id": gid, "expire_at": now + ttl, "ttl": ttl}


def revoke(gid, by="system"):
    con = sqlite3.connect(DB_PATH)
    with con:
        cur = con.execute(
            "UPDATE grants SET status='revoked' WHERE grant_id=? AND status='active'", (gid,)
        )
        if cur.rowcount:
            audit(con, "revoke", gid, by, {"by": by})
    con.close()
    return {"grant_id": gid, "revoked": True}


def sweeper():
    """
    Expiry reclamation thread: scans every 5 seconds, flips expired active grants to expired, and leaves a trace.
    Why a dedicated thread instead of reclaiming opportunistically inside a request: a grant can expire
    naturally without any subsequent request (for example, the engineer elevated and then did nothing).
    If reclamation only fired on request, that "unused permission" would linger forever — exactly the
    zombie permission we're out to eliminate.
    """
    while True:
        try:
            con = sqlite3.connect(DB_PATH)
            now = int(time.time())
            rows = con.execute(
                "SELECT grant_id, subject FROM grants WHERE status='active' AND expire_at<=?",
                (now,),
            ).fetchall()
            if rows:
                with con:
                    for gid, subject in rows:
                        con.execute("UPDATE grants SET status='expired' WHERE grant_id=?", (gid,))
                        audit(con, "expire", gid, subject, {"expired_at": now})
            con.close()
        except Exception as exc:  # the reclamation thread must never die from a single exception
            print("[sweeper] error:", exc)
        time.sleep(5)


class Handler(BaseHTTPRequestHandler):
    def _json(self, code, payload):
        body = json.dumps(payload, ensure_ascii=False).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json; charset=utf-8")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def do_GET(self):
        if urlparse(self.path).path != "/grants":
            return self._json(404, {"error": "not found"})
        con = sqlite3.connect(DB_PATH)
        rows = con.execute(
            "SELECT grant_id,subject,resource,verbs,expire_at,severity FROM grants"
            " WHERE status='active' AND expire_at>?", (int(time.time()),),
        ).fetchall()
        con.close()
        self._json(200, {"grants": rows})

    def do_POST(self):
        path = urlparse(self.path).path
        length = int(self.headers.get("Content-Length", 0))
        try:
            data = json.loads(self.rfile.read(length) or b"{}")
        except json.JSONDecodeError:
            return self._json(400, {"error": "invalid json"})

        if path == "/grant":
            missing = [k for k in ("subject", "resource", "reason") if not data.get(k)]
            if missing:
                # reason is required: a grant with no "why" is worthless in a post-hoc audit
                return self._json(400, {"error": f"missing: {','.join(missing)}"})
            return self._json(200, issue(
                data["subject"], data["resource"], data.get("verbs", ["patch"]),
                data.get("ttl_seconds", 900), data["reason"], data.get("severity", "SEV3"),
            ))
        if path == "/revoke":
            return self._json(200, revoke(data.get("grant_id", ""), data.get("by", "system")))
        self._json(404, {"error": "not found"})

    def log_message(self, *args):
        pass  # the default access log is noisy; auditing is handled by the audit_log table


if __name__ == "__main__":
    init_db()
    threading.Thread(target=sweeper, daemon=True).start()
    print(f"jit-grantd listening on http://{LISTEN[0]}:{LISTEN[1]}  (db={DB_PATH})")
    ThreadingHTTPServer(LISTEN, Handler).serve_forever()

Let’s run it and verify:

# 1) Request a 15-minute temporary elevation (the server clamps the TTL to the 300-1800 second range)
curl -s -X POST http://127.0.0.1:8787/grant \
  -H 'Content-Type: application/json' \
  -d '{"subject":"alice","resource":"ns/payment","verbs":["patch"],
       "ttl_seconds":900,"reason":"INC-20261027-0317","severity":"SEV1"}'
# => {"grant_id":"g-3f9a1c2b7e01","expire_at":1761500000,"ttl":900}

# 2) List the currently active grants
curl -s http://127.0.0.1:8787/grants

# 3) A grant with a 10-minute TTL will be flipped to expired by the sweeper after 10 minutes, leaving an expire audit record

This service doesn’t operate on systems directly; it’s an “authorization ledger.” Actual permission is granted by a downstream enforcer that reads this service’s state (for example, a Kubernetes admission controller, or a bastion host’s command proxy). That layering matters: once authorization logic and enforcement logic are decoupled, revocation becomes “flip a ledger state” rather than “chase down every system to delete a permission.”

Landing It on K8s: RBAC Has No TTL, So a Controller Must Backstop It

The fatal flaw of native Kubernetes RBAC is that it has no expiry. Once a RoleBinding is created it’s valid forever, so JIT has to add a controller layer on top of RBAC. Two approaches:

1. Short-lived tokens: use the TokenRequest API to issue a ServiceAccount token with expirationSeconds; the token dies when it expires. Note that the minimum validity Kubernetes allows is 600 seconds (10 minutes), so don’t set a TTL below that.
2. Controller-driven reclamation: a CronJob or a custom Controller that scans RoleBindings carrying a TTL annotation and deletes them on expiry.

# file: jit-rolebinding.yaml
# A one-off, namespace-scoped grant with a TTL annotation. Note that verbs only include patch,
# not delete -- emergency configuration recovery almost never needs delete rights, just patch or
# rollout restart.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: jit-patch-ns-payment
  namespace: payment
  annotations:
    sre.wang/issued-for: "alice"
    sre.wang/incident: "INC-20261027-0317"
    sre.wang/ttl-seconds: "900"
    sre.wang/expire-at: "1761500000"   # backfilled by jit-grantd; the controller revokes based on it
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["patch", "get", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jit-patch-ns-payment-binding
  namespace: payment
  annotations:
    sre.wang/ttl-seconds: "900"
    sre.wang/expire-at: "1761500000"
subjects:
  - kind: User
    name: alice
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: jit-patch-ns-payment
  apiGroup: rbac.authorization.k8s.io

Use OPA Gatekeeper or Kyverno as a checkpoint to stop anyone from creating a binding without a TTL annotation and bypassing the system:

# file: require-ttl-rolebinding.yaml (Kyverno policy)
# Why block at the admission layer: if you rely only on post-hoc sweeps, the window within a sweep
# cycle (possibly hours) is the hole. The admission layer guarantees that "non-compliant grants can't
# be created at all" -- the only point where you can close the gap the instant creation happens.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-ttl-on-jit-rolebinding
spec:
  validationFailureAction: Enforce
  rules:
    - name: jit-binding-must-have-ttl
      match:
        any:
          - resources:
              kinds: ["RoleBinding"]
      preconditions:
        all:
          - key: "{{ request.object.metadata.annotations.\"sre.wang/issued-for\" || '' }}"
            operator: NotEquals
            value: ""
      validate:
        message: "A JIT RoleBinding must carry the sre.wang/expire-at annotation, with a validity of no more than 1800 seconds"
        pattern:
          metadata:
            annotations:
              sre.wang/expire-at: "?*"
              sre.wang/ttl-seconds: "?*"

Trigger Means Broadcast: Awareness Outranks Blocking

The third recurring argument is: should emergency elevation go through approval? My answer depends on the severity level — there’s no one-size-fits-all rule.

Why SEV1 Should Not Wait for Approval

SEV1 means “the core path is down, affecting a large number of users.” At that point, every extra minute spent waiting for approval costs you something linear and quantifiable. In that 40-minute case earlier, the cost far outweighed the potential risk of a single misstep.

But “no waiting for approval” is not the same as “no notification.” There’s a key shift here: in a SEV1 scenario, what we give up is “pre-hoc blocking,” and what we get in return is “instant broadcast + post-hoc audit.” There has never been a scheme that is both strictly controlled and zero-latency; the essence of an engineering decision is picking the side whose cost you can accept.

My recommendation: SEV1 goes “no approval, broadcast on trigger,” with a mandatory audit review within 24 hours; SEV3 and below go through normal JIT approval. The reasoning: approval is a form of “pre-hoc blocking,” and its value lies in stopping elevations that are “unnecessary but convenient,” a scenario that is common at low severity levels. By the time you reach SEV1, the incident itself has already answered the question of whether it’s necessary — blocking at that point doesn’t stop risk, it stops time-to-mitigation. The value of broadcasting is that it turns “post-hoc audit” into “real-time oversight”: everyone can see who used what permission at what time, so abuse gets questioned within minutes rather than waiting for an audit report a month later.

The SEV Tiered Decision Table

Every column in this table maps to an engineering constraint — don’t casually change one column without touching the others. For example, if you stretch SEV1’s TTL from 30 minutes to 4 hours, you have to simultaneously widen the broadcast scope to include continuous monitoring — otherwise the combination of “no approval + long TTL + inadequate broadcast” leaves production write access open, unattended, for an extended period.

Broadcast Content Design: A Broadcast Must Carry Verifiable Fields

A qualified break-glass broadcast can’t just say “someone used emergency access.” It must include:

• Who (subject — a real person, never a shared account)
• Over what (resource — down to the namespace/host/device level)
• What they can do (verbs)
• Why (the associated incident number, INC-xxxx, plus a one-line reason)
• Until when (expire_at)

I built one unglamorous but very effective detail here: the expire_at in the broadcast is an absolute timestamp, not a relative phrase like “in 15 minutes.” Because the people in the on-call channel are scattered across time zones, a relative description makes everyone do their own arithmetic; an absolute timestamp is what makes “why hasn’t this been revoked yet?” visible at a glance.

With Prometheus you can turn “an emergency grant is in use” into an alertable rule and integrate it seamlessly with your existing alerting:

# file: breakglass-alert-rules.yaml
# Dependencies: jit-grantd exposes /metrics (add an exporter to the sample service in this article),
# or a check script pushes the grant count as a Pushgateway metric.
groups:
  - name: break-glass.rules
    rules:
      # Rule 1: any single break-glass use alerts immediately (no waiting for a for duration)
      # Why for: 0s: the use of emergency access is itself a first-class event and doesn't need to persist for N minutes to fire.
      - alert: BreakGlassGrantIssued
        expr: increase(jit_grant_issued_total{channel="breakglass"}[5m]) > 0
        for: 0s
        labels:
          severity: critical
          category: privileged_access
        annotations:
          summary: "A production break-glass elevation was used"
          description: >-
            {{ $value }} break-glass elevations in the last 5 minutes.
            subject={{ $labels.subject }} resource={{ $labels.resource }}
            incident={{ $labels.incident }}. Go to the incident channel now to confirm what was done.            
          runbook_url: "https://www.sre.wang/posts/sre-break-glass-emergency-access/"

      # Rule 2: abnormal spike in the number of grants (someone may be bulk-applying, or the system is being abused)
      - alert: JitActiveGrantsTooMany
        expr: jit_grant_active > 20
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Abnormally many active JIT grants ({{ $value }})"
          description: "The normal watermark should be below 5. Check whether there's a bulk application or unrevoked grants."

      # Rule 3: still in use after expiry (revocation failure = the most dangerous signal)
      # Why a separate rule: failing to revoke on expiry means the ledger state and the actual capability
      # have drifted, which is worse than "too much permission was granted" -- because it means the
      # reclamation mechanism itself is broken.
      - alert: JitGrantOverdueStillActive
        expr: jit_grant_overdue_active > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "{{ $value }} grants have expired but are still active"
          description: "Automatic revocation may be failing. Revoke manually now and investigate the sweeper."

      # Rule 4: no review submitted within 24 hours of an emergency elevation
      - alert: BreakGlassReviewOverdue
        expr: jit_breakglass_without_review_seconds > 86400
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "A break-glass use has gone more than 24 hours without a review"
          description: "incident={{ $labels.incident }}. Please submit the post-incident review record."

End-to-End Audit Trails: From the Bastion Host Door to the Incident Timeline

No matter how well permissions are designed, if the audit trail is incomplete you still can’t explain anything after the fact. I got burned by exactly this in one retro — because command-level auditing wasn’t complete at the time, all we had was “someone ran a change on a jump host,” with no way to tell which backend host it actually hit, so the retro stalled on root-cause localization for a full two days.

A Single Entry Point: All Production Operations Go Through One Door

The first principle of auditing is a single entry point. Every production write operation (including break-glass) must go through the same entry point — a bastion host, or at minimum a single command proxy. The reason: an audit chain is only complete under a single entry point. If two or more paths in exist, you can never be sure “no operation bypassed auditing.”

In practice I enforce this at the network layer: management ports on production hosts accept traffic only from the bastion host’s source IP, so any attempt to connect directly and bypass the bastion simply doesn’t route. That makes the audit entry point an architectural fact rather than a policy that relies on discipline.

The “management-plane network isolation + minimal exposure” described in Related: Linux Security Hardening Checklist: From Minimal Installation to Compliance is two sides of the same idea as the single entry point here: one side shrinks the attack surface, the other guarantees audit completeness.

Command-Level Auditing Plus Session Recording

Auditing has three layers, each deeper than the last:

Account-level auditing alone is not enough. The “couldn’t tell which machine was changed” trap earlier happened precisely because we had account-level auditing but not command-level. In command-level auditing, pay special attention to recording the target object identifier — for example the -n namespace in a kubectl command, a --target argument, the destination host of an SSH session, the resource ID of an API call. Without it, the audit record is just “someone typed some characters,” not “someone did something to a particular object.”

Don’t turn session recording on for everything by default; its storage cost is high. My approach: standing accounts get command-level auditing only, and full session recording is forced on only for break-glass elevations. The core scenario for session recording is “retracing the complete sequence of operations under an anomalous grant,” and that happens almost exclusively during emergency elevation.

On the compliance side, MLPS 2.0 imposes retention requirements on audit records, which in engineering terms generally means no less than 6 months. Tencent Cloud Developer Community’s Bastion Host and Data Audit Practices for MLPS Compliance gives a fairly complete walkthrough of engineering details like “audit record retention + centralized management + tamper resistance,” which you can use as a reference for the compliance path. Microsoft’s Microsoft Cloud Security Benchmark: Privileged Access likewise lists “privileged access must be auditable and traceable” as a core control and explicitly recommends JIT authorization over standing privileges.

Audit Table Design: Store the “Why” Too

The field most easily left out of an audit trail is “why.” An audit table with only who/what/when can’t answer “was this operation reasonable at the time” during a retro. So when you design an audit table, reason and the incident-linkage field have to be first-class citizens.

-- file: audit_schema.sql
-- Design notes: the audit table is append-only -- no updates, no deletes; every write must share a
-- transaction with the business operation.
-- The field design centers on the five questions a retro needs: who, over what, did what, why, and when.

CREATE TABLE access_audit (
    id              BIGSERIAL PRIMARY KEY,
    ts              TIMESTAMPTZ NOT NULL DEFAULT now(),   -- when
    actor           TEXT        NOT NULL,                  -- who (must be a specific person; shared accounts forbidden)
    actor_source_ip INET        NOT NULL,                  -- where from (supports attribution)
    channel         TEXT        NOT NULL,                  -- channel: jit / breakglass / standby
    grant_id        TEXT,                                  -- linked grant record
    resource_type   TEXT        NOT NULL,                  -- object type: host/namespace/device
    resource_id     TEXT        NOT NULL,                  -- object identifier (the key to retro localization)
    action          TEXT        NOT NULL,                  -- what was done: patch/rollback/restart
    command         TEXT,                                  -- the raw command (command-level audit)
    session_id      TEXT,                                  -- session ID (links to session recording)
    incident_id     TEXT,                                  -- linked incident number (threads the timeline together)
    reason          TEXT        NOT NULL,                  -- why (required for a retro)
    outcome         TEXT        NOT NULL,                  -- success/failure
    approval_ref    TEXT                                   -- approval ticket number (non-null when approval applies)
);

-- Query pattern: during a retro, aggregate all operations by incident number and order by time to reconstruct the timeline
CREATE INDEX idx_audit_incident ON access_audit (incident_id, ts);
-- Query pattern: during an audit, look up all production operations by a person over a time range
CREATE INDEX idx_audit_actor_time ON access_audit (actor, ts DESC);

-- The audit table must be tamper-resistant. In production: revoke UPDATE/DELETE from the ordinary
-- application account, keep INSERT only, and drop a read-only copy via database auditing or an
-- external log sink.
REVOKE UPDATE, DELETE ON access_audit FROM app_rw;

During a retro, one SQL query pulls out the complete operation timeline for an incident to compare against the alerts and incident records:

-- Reconstruct all production operations under a given incident, ordered by time, to compare against the alert timeline
SELECT
    a.ts            AS operation_time,
    a.actor         AS operator,
    a.channel       AS channel,
    a.resource_id   AS object,
    a.action        AS action,
    a.outcome       AS result,
    a.reason        AS reason
FROM access_audit a
WHERE a.incident_id = 'INC-20261027-0317'
ORDER BY a.ts;

Correlating With the Incident Timeline

An audit trail sitting alone in a database has limited value. Its value comes from being linked to alerts, changes, and incident records to form a timeline you can compare against. I require every break-glass grant to carry an incident_id, so that:

• The alert fires at 03:17
• break-glass is issued at 03:19 (the reason hangs off the same incident)
• The actual change runs from 03:20 to 03:24 (command-level audit)
• The grant is automatically revoked at 03:34
• The retro is submitted the next day

Once this timeline lines up, the retro discussion is no longer “I think at the time it was roughly…” but “the audit record shows the grant was issued at 3:19 and the actual action happened at 3:20 — what were we waiting for during that one minute?” That level of precision is critical for improving the emergency process.

Post-Incident Review and Expired-Permission Sweeps

The last link in an emergency access system is the wrap-up. Without it, the system slowly rots — permissions forgotten, retros unwritten, drills skipped, and six months later it’s as good as nonexistent.

Mandatory Review Within 24 Hours

A retro must be submitted within 24 hours of any break-glass use, covering four questions:

1. Why weren’t normal permissions enough? (This one is the easiest to skip, but it points straight at the gaps in your permission model)
2. Were all the emergency operations within the granted scope? (Any overreach? Any continued operation in another way after the TTL lapsed?)
3. Was there a less privileged alternative? (Would JIT have been enough next time, without break-glass?)
4. What problems did this use expose in the channels or the drills?

I care especially about question 1. If a certain class of failure repeatedly needs break-glass, it means there’s a problem with the scope or the approval speed of normal JIT, and it needs adjusting. The frequency of break-glass use is itself a KPI — it should sit stable at a low level: never zero for a long stretch (which would mean nobody has ever verified the channel), and never frequent (which would mean the standing permission design is broken).

Automated Sweeps for Expired Permissions

People will always forget to revoke permissions — that’s a certainty, not an exception. So there has to be an automated sweep to backstop it. The Python script below scans several common permission sources to find grants that are past their expiry but never revoked:

#!/usr/bin/env python3
# file: stale_grant_audit.py
"""
Expired-permission sweep -- scans permission sources for grants that are "expired but still present" and alerts.

[Dependencies] standard library + requests (pip install requests); kubectl/aws and similar CLIs must be on PATH.
[Input] none (configured in SOURCES at the top of the script).
[Output] a list of expired grants; exits 1 when anything is found. Best run daily from a scheduled job,
         or wired into the on-call daily report.
[Why a sweep as backstop] The auto-revocation service handles reclamation on the "normal path," but if the
    sweeper crashes, if a permission is created directly in K8s bypassing jit-grantd, or if a grant is added
    by hand through a console, none of those are covered by automatic revocation. The sweep is the last line
    of defense, similar in spirit to periodic consistency checks on a database.
"""

import json
import subprocess
import sys
import time
from datetime import datetime, timezone

import requests

JIT_API = "http://127.0.0.1:8787"
GRACE_SECONDS = 300  # allow a 5-minute grace period so we don't race the sweeper's scan cycle


def stale_from_jit():
    """Check whether jit-grantd has any grants that are expired but still active (a revocation-failure signal)."""
    stale = []
    try:
        grants = requests.get(f"{JIT_API}/grants", timeout=5).json().get("grants", [])
    except requests.RequestException as exc:
        # An unreachable API is itself an anomaly, but it shouldn't fail the whole sweep -- it just means this source can't be covered
        return [{"source": "jit", "error": f"api unreachable: {exc}"}]
    now = time.time()
    for g in grants:
        # field order matches the SELECT in jit_grantd
        gid, subject, resource, _verbs, expire_at, _sev = g
        if expire_at + GRACE_SECONDS < now:
            stale.append({
                "source": "jit", "grant_id": gid, "subject": subject,
                "resource": resource,
                "expired_for_seconds": int(now - expire_at),
            })
    return stale


def stale_from_k8s():
    """Scan for RoleBindings that carry the sre.wang/expire-at annotation but have already expired."""
    stale = []
    cmd = [
        "kubectl", "get", "rolebindings", "--all-namespaces", "-o", "json",
    ]
    try:
        out = subprocess.run(cmd, capture_output=True, text=True, check=True, timeout=30).stdout
    except (subprocess.CalledProcessError, subprocess.TimeoutExpired) as exc:
        return [{"source": "k8s", "error": str(exc)}]
    now = int(time.time())
    for item in json.loads(out).get("items", []):
        ann = (item.get("metadata", {}).get("annotations") or {})
        expire_at = ann.get("sre.wang/expire-at")
        if not expire_at:
            continue
        try:
            if int(expire_at) + GRACE_SECONDS < now:
                stale.append({
                    "source": "k8s",
                    "binding": f'{item["metadata"]["namespace"]}/{item["metadata"]["name"]}',
                    "issued_for": ann.get("sre.wang/issued-for", "unknown"),
                    "expired_for_seconds": now - int(expire_at),
                })
        except (TypeError, ValueError):
            continue
    return stale


def main():
    findings = []
    for fn in (stale_from_jit, stale_from_k8s):
        findings.extend(fn())

    ts = datetime.now(timezone.utc).isoformat()
    print(f"[{ts}] expired-permission sweep, {len(findings)} finding(s)")
    for f in findings:
        print("  -", json.dumps(f, ensure_ascii=False))

    if findings:
        # Hook alerting in here: push a webhook, post to the on-call channel, or write to the daily report
        print("Expired permissions found; handle them now: revoke manually first, then investigate why auto-revocation failed.")
        return 1
    print("No expired permissions.")
    return 0


if __name__ == "__main__":
    sys.exit(main())

The Verification Loop for Permission Revocation

Finding a problem in the sweep isn’t enough — you also have to verify that “the revocation actually took effect.” In my projects I require a real verification after every revocation: once a grant is marked revoked, a probe attempts the revoked action to confirm it is genuinely denied.

This verification matters because it catches the case where “the ledger state updated but the actual permission was never removed” — for instance, a K8s RoleBinding was deleted, but somewhere a long-lived token is still cached. Ledger-only auditing can’t see this at all, because the ledger says “revoked” and everything looks fine. Auditing should verify actual capability, not ledger state. That lesson comes from the “zombie emergency accounts” incident mentioned earlier: at the time, the audit table showed those accounts as revoked, but the actual permissions were still there, because the revocation had only changed the ledger without changing the real binding.

Summary

Back to that 3:17 AM scene. If this system had been in place, the flow would have been:

1. The alert fires, and the on-call engineer confirms it’s SEV1
2. A 30-minute emergency grant is obtained through the break-glass independent channel (no approval, but broadcast to the on-call channel, the security lead, and the incident board)
3. Under forced session recording on a sentinel host, the narrowest possible permission (patch the network ACL only) puts the rule back, recovering in 3 minutes
4. The grant is automatically revoked after 30 minutes, with the sweeper leaving a trace
5. A retro is submitted the next day; SQL pulls the complete timeline and confirms every step stayed inside the granted scope
6. The sweep script confirms no permissions were left behind; if several network-related incidents in a row all needed break-glass, that’s the signal that pre-authorization for network devices should be added to normal JIT

The value of this system isn’t how complex it is — it’s that it takes the instinctively conflicting goals of “security” and “speed” and breaks them down, through engineering, to the point where they can coexist. Of the six decisions, the two I think are most underestimated are number 2 (the pre-staged channel) and number 6 (the wrap-up): the former determines whether the system is any use during a real incident, and the latter determines whether the system survives long-term.

If I could only give three things to do right now:

1. Get the TTL down to 15–30 minutes first. It’s nearly zero-cost — a config change drops zombie permissions to near zero, and the effect shows immediately
2. Run one break-glass drill, even if it’s just walking through the unsealing process. I’d bet a lot of teams have sealed credentials that are expired, missing shards, or bound to disabled accounts — you’ll never find out without a drill
3. Add a “target object identifier” to command-level auditing. Whether a retro can localize the root cause quickly is 80% determined by this one column

Once the system is built, wiring it together with change gates, the on-call mechanism, and error budgets gives you a complete production stability governance system. Related: 3 Days of Approval, Still Crashed: Replacing Manual CAB with Error Budget Gates Cut Change-Related Incidents by 60% covers automated gating on the change side, and this article covers emergency access — together they span the two paths of “planned changes” and “unplanned emergencies.”

References & Acknowledgments

1. Emergency Response — Google SRE Book, Chapter 13. Referenced for the emergency response workflow, incident command role separation, and the “clear authorization, clear boundaries” response principle
2. Managing Incidents — Google SRE Book, Chapter 14. Referenced for the first principle of incident communication — “keep the right people informed” — used to design the awareness scope of break-glass broadcasts
3. Microsoft Cloud Security Benchmark: Privileged Access — Microsoft. Referenced for the core control requirements in the privileged access baseline: “JIT authorization takes precedence over standing privileges” and “privileged operations must be auditable”
4. RBAC best practices — Microsoft Entra ID documentation. Referenced for least-privilege principles and role assignment best practices, used to design the privilege convergence model
5. Just-in-Time (JIT) VM access — Microsoft Defender for Cloud. Referenced for the implementation approach to time-boxed JIT access and the design pattern of limited-duration authorization
6. Bastion Host and Data Audit Practices for MLPS Compliance — Tencent Cloud Developer Community. Referenced for engineering details on centralized, tamper-resistant audit retention and MLPS 2.0 compliance rollout

The real-world cases and figures in this article have all been generalized and do not refer to any specific enterprise, client, or project. Thanks to the colleagues who polished this emergency access mechanism with me across multiple production environments — your retros and your insistence on rigor are what made these decisions hold up.