The 3 AM Phone Rings Again: 7 Engineering Decisions for On-Call Rotation Design and Fatigue Management

Overview

3 AM. Your phone buzzes. P1 alert — order service 5xx error rate spiking. You drag yourself out of bed, fire up the laptop, pull logs, check Grafana, trace it to a saturated upstream database connection pool. Restart, scale up, recover. It’s 4:30 AM. The next morning, nursing dark circles in the standup, someone mentions that last night’s alert was actually a false positive.

If you’ve worked in operations, you know this feeling.

On-Call is an unavoidable part of SRE work. But many teams run it as “whoever gets woken up deals with it” — no rotation, no backup, no handoff, and certainly no fatigue management. The result: key engineers quit, alert channels become white noise, and incident response gets slower and slower.

This article covers 7 engineering decisions in On-Call rotation design. Not theoretical frameworks — these are the trade-offs and hard lessons from building SRE on-call systems from scratch. In a previous article on Incident Management and On-Call Mechanism Design, I covered the incident response process. This article focuses on the rotation itself: shift cadence, primary-secondary design, alert tiering, fatigue quantification, handoffs, new hire onboarding, and compensation.

If you’re a tech lead who just took over an SRE team, or you’re being crushed by chaotic on-call practices, this should save you about six months of trial and error.

1. What Problem Does On-Call Actually Solve?

Let’s be clear about what On-Call is before discussing how to design it.

On-Call is: when systems break outside working hours, someone is available to respond and restore service immediately.

Think of it like a hospital ER. The ER can’t close at night. Doctors work day shifts, but someone has to be on duty at night. And that someone isn’t the same person pulling a week-long all-nighter — that’s how you get a sleep-deprived surgeon with shaky hands during the next day’s operation. Hospitals solve this with rotation: day shift, night shift, backup shift, cycling through staff so every shift has a rested doctor.

On-Call works the same way. Your systems run 24/7, but engineers can’t stare at dashboards 24/7. On-Call rotation ensures “the right person gets the alert at the right time” — not “everyone gets the alert” or “nobody gets the alert.”

Google SRE’s On-Call Principles

Google SRE dedicates three chapters to On-Call in Site Reliability Engineering. The core principle: toil must not exceed 50% of working time.

What’s “toil”? Google defines it as repetitive, automatable, tactical work with no lasting value — manually restarting services, manually scaling, manually digging through logs to find the problem. On-Call is the biggest source of toil.

An SRE must spend at least half their time on engineering projects (writing code, building tools, improving architecture). Otherwise, you’re plugging system holes with human labor, and the holes keep getting deeper. To sustain this ratio, a team needs at least 6-8 people for rotation — fewer than that, and engineering time gets eaten by shifts.

Google SRE’s on-call methods and tools provides a clear analysis: Google manages the entire alert lifecycle through Outalator, including aggregation, tagging, analysis, and handoff reports — making On-Call “managed” rather than just “woken up.” Outalator’s core features include:

• Alert aggregation: Merge related alerts into a single “incident” to reduce duplicate notifications
• Tagging: Label incidents for filtering and statistics by dimension
• Data analysis: Analyze alert trends by team, individual, service, data center
• One-click handoff reports: Select a batch of incidents and auto-generate an email-format handoff document

Most domestic teams can’t fully replicate Google’s approach — culture, team size, and compensation mechanisms all differ. But tools and processes can be adopted first.

Four Core Questions of On-Call

An effective On-Call system must answer four questions (referencing Flashcat’s On-Call practice guide):

1. Who is responsible — who’s the current on-call engineer, who’s the backup
2. When are they responsible — how are rotation periods and time windows divided
3. How are they notified — what channel delivers alerts to the on-call engineer
4. What if nobody responds — escalation policy and fallback mechanism

These four questions map to the 7 engineering decisions below.

2. Seven Engineering Decisions

Decision 1: Shift Cadence — Weekly or Daily?

The first question in scheduling: how often do you rotate?

This seems simple, but getting it wrong has outsized impact. Rotate too frequently, and handoff costs eat your time with lost context. Rotate too slowly, and fatigue accumulates — by week two, the on-call engineer is running on fumes.

I recommend weekly rotation. The reasoning is straightforward: daily rotation’s handoff cost is too high. You just figured out yesterday’s intermittent timeout cause, and today someone new takes over who has to start from scratch. Bi-weekly is too much pressure — two weeks of constant readiness means the second week is basically survival mode.

In an 8-person SRE team I led, we tried all three cadences. Daily rotation lasted two weeks before we abandoned it — 20 minutes of handoff every day, with information constantly dropping through the cracks. The on-call engineer would just figure out the context of a recurring alert, and the next day’s replacement had to relearn it. Weekly rotation was the final choice: Monday morning handoff, last weekend’s lingering issues become Monday’s priority items, and the transition is natural.

Decision framework for shift cadence:

Team size < 5 → Don't do formal On-Call. Use "you build it, you run it" model
Team size 5-8 → Weekly rotation, each person on-call 1-2 weeks/month
Team size 8-15 → Weekly or daily, depending on alert volume
Team size > 15 → Daily rotation, with multi-time-zone coverage

Pitfall warning: Shift cadence isn’t set in stone. During peak business periods (e.g., e-commerce Double Eleven, logistics peak season), temporarily switch to daily rotation to spread the burden. Switch back to weekly when the peak passes.

Another easily overlooked point: Friday through Sunday is the hardest stretch. A problem Friday night means weekend firefighting, and Monday morning you’re back at your desk. I recommend splitting weekend on-call from weekday on-call — separate people. The weekday person gets Friday evening off; the weekend person does project work Monday through Thursday. With an 8-person team, 2 people are on-call each week (1 weekday + 1 weekend), each rotating every two weeks. Pressure stays manageable.

Decision 2: Primary-Secondary Mechanism — Don’t Let One Person Carry It Alone

The “Primary” on-call engineer responds to alerts, but what if Primary doesn’t respond? There must be a “Secondary” on-call engineer.

This is like an aircraft’s pilot and co-pilot. The pilot flies, the co-pilot monitors and backs up. If the pilot becomes incapacitated, the co-pilot takes over. A single-pilot aircraft can fly, but when something goes wrong, there’s no fallback.

Core rules for primary-secondary design:

• Primary handles all P1/P2 alert first response
• Secondary automatically takes over if Primary doesn’t respond within 5 minutes
• Secondary does not handle P3/P4 alerts (those can wait until business hours)
• Primary and Secondary cannot be the same person (obvious, but teams have done it)
• Primary and Secondary ideally aren’t in the same physical location (prevents simultaneous network loss)

Configuring escalation policies in PagerDuty or Opsgenie looks something like this:

# PagerDuty escalation policy example (YAML format, conceptual config)
escalation_policy:
  name: "order-service-production"
  levels:
    - level: 1          # Level 1: Primary on-call
      targets:
        - user: "primary-oncall-user"
      delay_minutes: 0  # Notify immediately
    - level: 2          # Level 2: Secondary (escalate after 5 min no response)
      targets:
        - user: "secondary-oncall-user"
      delay_minutes: 5
    - level: 3          # Level 3: Tech lead (15 min still no response)
      targets:
        - user: "tech-lead"
      delay_minutes: 15
    - level: 4          # Level 4: CTO/Ops Director (30 min disaster level)
      targets:
        - user: "ops-director"
      delay_minutes: 30

The delay between levels is critical. 5 minutes for Primary response is reasonable — if you’re in the shower, 5 minutes is enough to dry your hands and answer. But if Primary is in deep sleep and doesn’t hear the phone, Secondary should be woken up after 5 minutes.

Measured data: After implementing the primary-secondary mechanism, our alert miss-response rate dropped from 3.2% to below 0.1%. Before Secondary existed, if Primary was in the bathroom, taking a shower, or had their phone on silent, alerts disappeared into the void. With Secondary, someone always responds within 5 minutes.

A counterintuitive finding: More Secondaries isn’t better. I saw a team configure 3 Secondaries. When a P1 alert fired, 4 people were woken simultaneously, and 3 of them waited for “someone else to handle it first.” The core of primary-secondary is “clear accountability,” not “strength in numbers.” One Primary + one Secondary is sufficient.

For more on escalation strategy design, see Alerting Strategy Design: From Noise to Signal, which covers alert routing and inhibition in detail.

Decision 3: Alert Tiering — Not Every Alert Deserves to Wake Someone Up

This is the most easily overlooked decision, and also the one with the biggest impact.

I saw a team with 400 Prometheus alert rules, all pushing to PagerDuty. The on-call engineer was woken up 8 times a night, 6 of which were “disk usage above 80%” — non-urgent alerts. A month later, that engineer resigned.

Alerts must be tiered. It’s like a hospital triage desk: cardiac arrest goes to the ER, a common cold waits in the lobby. If you send every patient to the ER, real emergencies don’t get timely treatment.

Four-tier alert system:

Only P1 and P2 trigger On-Call alerts. P3 and P4 go through tickets or email, not phone pushes.

The impact of this tiering was enormous. We previously had 300+ daily alerts, all pushed to the on-call phone. After tiering, phone-triggering P1/P2 alerts dropped to 15-20 per day. The on-call engineer’s nightly wake-ups dropped from an average of 4 per night to less than 1.

Alert tiering implementation steps:

1. Inventory existing alerts: Export all alert rules, tag each with P1-P4
2. Modify notification routing: P1/P2 go to phone push, P3/P4 go to email or IM group
3. Set transition period: Observe for 1 week, check for miscategorized alerts
4. Regular review: Monthly check — should any P3/P4 be promoted to P1/P2, or vice versa

A practical heuristic: Ask yourself — “If this alert fires at 3 AM, do I want to be woken up?” If the answer is no, it’s not P1/P2.

The logic: alert value depends on signal-to-noise ratio, not quantity. If 280 of 300 alerts are noise, the on-call engineer starts ignoring all alerts after the 10th one — including the 20 that actually matter. This is “alert fatigue,” covered extensively in PagerDuty’s Going On-Call guide.

3 practical methods for alert noise reduction:

1. Aggregate related alerts: If CPU, memory, and disk alerts for the same service fire simultaneously, merge them into one “service resource anomaly” notification. This is one of Google Outalator’s core features. Prometheus achieves this via Alertmanager’s group_by configuration.

2. Set alert inhibition rules: If a P1 alert has fired (e.g., “database down”), automatically inhibit related downstream alerts (e.g., “order service errors,” “payment service timeouts”). Downstream failures are inevitable consequences of upstream outages — no need for duplicate notifications.

3. Replace threshold-based alerts with SLO-based alerts: Instead of “CPU >80% alert,” configure “error budget consumption >50% alert.” SLO alerts directly correlate to user experience and naturally produce less noise.

Decision 4: Fatigue Quantification — Use Data to Decide When to Rest

“I feel like I’m about to break” — this has been said in countless retrospectives, but managers can’t make decisions based on feelings. You need data to quantify fatigue.

Why quantification matters:

A common managerial misconception is “on-call isn’t a big deal” — you’re just waiting for calls at home, right? But research shows that after sleep interruption, next-day cognitive performance is equivalent to a blood alcohol concentration of 0.1%. An engineer woken up 3+ times per night for a week shows significant decline in judgment and reaction speed. This isn’t “power through it” territory — it’s measurable physiological damage.

5 core measurement metrics (referencing Google SRE’s Outalator design):

How to use this data:

In the weekly retrospective, review these metrics’ trends. Don’t look at absolute values — look at trends. If alert volume suddenly triples compared to the previous week, even if the absolute value hasn’t crossed the threshold, it warrants attention.

If an on-call engineer’s night wake-ups exceed 4 for two consecutive weeks, it means either alert quality is poor (too much noise) or the system is unstable (frequent failures). Immediate intervention is needed:

• Too much alert noise → Pause that engineer’s On-Call, spend a week on alert governance
• System instability → Pull in the dev team to investigate root causes; SREs shouldn’t be perpetual firefighters

A real case: One engineer on our team had 6 and 7 night wake-ups for two consecutive weeks. Looking at the data, the problem wasn’t alerts — it was a memory leak in a microservice causing a daily OOM restart at 2 AM. After fixing that bug, wake-ups dropped to zero. Without data, we would have assumed “too many alerts configured” or “this person has bad luck.”

For more on MTTR optimization, see MTTR from 40 Minutes to 8, which covers the complete fault localization acceleration approach.

Decision 5: Handoffs — Information Can’t Die in a Dream

Handoff is the most easily overlooked part of On-Call.

Imagine this: Monday morning, last week’s on-call engineer is on vacation, you’re taking over. Last night at 3 AM they handled a database slow query alert and changed a parameter. You don’t know. This afternoon the system slows down again, and you spend 30 minutes discovering that last night’s parameter change was wrong — the temporarily increased connection count actually worsened database load.

With a handoff document, you’d know in 5 minutes what was changed, why, and what the risks were. Without one, you’re排查 from scratch.

Handoffs must accomplish three things:

1. Write a handoff document: What was handled, what config was changed, what’s still unresolved
2. Verbal walkthrough: 15-minute Monday standup to cover key items face-to-face
3. Update Runbook: If new troubleshooting steps were discovered, add them to the Runbook

Handoff document template:

# On-Call Handoff Document (2026-08-19 ~ 2026-08-25)

## This Week's Alert Summary
- P1 alerts: 3 (all recovered)
- P2 alerts: 12 (all recovered)
- Night wake-ups: 2

## Key Incident Records
### 2026-08-22 03:15 - Order service 5xx spike
- Cause: MySQL connection pool saturated (max_connections=500, actual 498)
- Action: Temporarily increased max_connections to 800, restarted order service
- Outstanding: Need to investigate why connections surged, ticket #2026-0822-001
- Note: max_connections=800 is temporary, dev team investigating root cause Tuesday

## Outstanding Issues
1. Redis cluster node 3 intermittent timeout (ticket opened, network team investigating)
2. Alert rule "order_service_latency_p99" threshold too low, recommend 200ms → 300ms
3. MySQL connection surge root cause pending (ticket #2026-0822-001)

Key principle: The handoff document isn’t written for others — it’s written for your future self. Two weeks later you might be on-call again, and you’ll need that document to recall “what happened with that connection pool issue last time.”

3 disciplines for handoff documents:

• Timeliness: Write 3 lines immediately after each P1/P2 incident. Don’t wait until the weekend — by then you’ll have forgotten the details
• Actionability: Outstanding issues must include ticket numbers or links, not just “something to look into”
• Conciseness: Each record should be 5 lines max. The handoff doc is a memo, not an incident report

Decision 6: New Hire Onboarding — From Shadow to Solo

Can a new engineer go directly on On-Call? No.

I saw a team put a new hire on the on-call schedule in their second week. On their first on-call day, they encountered a database master-slave failover and had no idea what to do — didn’t know the failover command, didn’t know what to check after failover, didn’t know who to ask for help. By the time the alert escalated to the tech lead, 20 minutes had passed. That P1 incident became a P0.

Three-phase new hire onboarding (referencing Rootly’s On-Call guide):

Shadow phase is like sitting in the passenger seat watching a driving instructor. The new hire doesn’t act but follows the on-call engineer through alerts, logs, and troubleshooting. The focus is understanding “what’s the first step when an alert comes in” — which Grafana dashboard to check first, which service’s logs to pull, who to ask for context.

Reverse shadow phase is where the new hire starts doing, with the veteran watching over their shoulder. The new hire handles alerts, changes configs, performs rollbacks — the veteran only intervenes if something’s about to break. Mistakes happen here — that’s fine. Making mistakes in a controlled environment is the best way to learn.

Key rules for reverse shadow:

• New hire handles P2 alerts; veteran handles P1 (P1 is too risky for practice)
• Every action (restart, config change, scale) must be verbally reported and confirmed before execution
• After each incident, 5-minute debrief: what went well, what was slow, what to improve

Before going solo, the new hire must pass an assessment: independently handle 3 common alert scenarios (e.g., service restart, database failover, traffic switching). The assessment uses simulated fault injection — in a test environment, inject a fault and watch the new hire complete the full Runbook workflow.

A painful lesson: We once let a new hire handle P1 alerts during reverse shadow. They executed kubectl delete pod instead of kubectl rollout restart during a service restart, causing a brief outage. The veteran was watching but the new hire was too stressed to hear the correction. After that, we changed the rule: reverse shadow only handles P2; P1 must be veteran-led.

Decision 7: Compensation and Measurement — Making On-Call Sustainable

The final decision, and the hardest to push through: On-Call compensation.

On-Call means your personal time can be interrupted at any moment. Middle-of-the-night wake-ups, weekends that can’t stray far from home, pulling out the laptop mid-movie. Without reasonable compensation, engineers vote with their feet — they leave.

Three compensation models:

I recommend the third model. In a ride-hailing project, we implemented “half-day comp time per P1/P2 night response + 2000 yuan monthly On-Call stipend.” It worked well: engineers didn’t feel exploited, and comp time ensured rest after response. The key is that comp time must be taken within 30 days — fatigue compensation loses its value if delayed.

3 principles for compensation design:

1. Cover nights only: Daytime On-Call is part of normal work, no extra compensation needed. Night (22:00-08:00) response is the extra付出
2. Per response, not per alert: One incident may trigger 10 alerts but the on-call engineer handles one event. Per-alert compensation would penalize alert aggregation
3. Differentiate P1 and P2: P1 response compensation should be higher than P2 — P1 typically means longer handling time and greater mental pressure

Team health measurement:

Beyond the 5 alert metrics, track team-level health signals:

PagerDuty’s research indicates that replacing an engineer who quit due to On-Call fatigue costs up to $300,000 (approximately 2 million RMB). This doesn’t include the hidden costs of knowledge loss and team morale decline. Skimping on compensation is the most short-sighted decision you can make.

3. Production Pitfall Log

A few real pitfalls, each a painful lesson.

Pitfall 1: No alert tiering, everyone on-call

At an e-commerce platform, all Prometheus alerts pushed to a DingTalk group. 20 people in the group, each alert popped up, everyone glanced and waited for someone else to handle it. Psychologists call this the “bystander effect” — the more people, the more each person assumes “someone else will deal with it.”

Result: P1 alert average response time was 12 minutes. Once, when the database master went down, 20 people in the group all waited — 7 minutes before the first engineer started investigating.

After switching to on-call-only notifications, response time dropped from 12 minutes to 2 minutes.

Lesson: On-Call’s core is “clear accountability.” An alert group isn’t On-Call — it’s a blame-shifting group.

Pitfall 2: No handoff document, same problem investigated twice

A database slow query alert fired. Monday’s on-call engineer spent 30 minutes finding that a SQL query wasn’t using an index. Added the index, recovered. But no handoff document. Wednesday’s on-call engineer hit the same problem — the execution plan had changed again after data volume shifted. Another 30-minute investigation.

After mandating 3-line handoff records for every P1/P2, our duplicate investigation rate dropped 70%.

Lesson: Handoff documents have the highest ROI of all On-Call practices. 3 lines of text saves 30 minutes of investigation. That’s a 1:600 return.

Pitfall 3: New hire on solo on-call, chaos during incident

Covered above. After that incident, we established the rule: no solo on-call within 4 weeks of joining. Must complete shadow → reverse shadow → solo.

Lesson: A new hire isn’t a resource — they’re an investment. The first 4 weeks of “no on-call” isn’t waste; it’s developing an engineer who can handle incidents independently.

Pitfall 4: Same person for weekday and weekend on-call

An engineer on 7 consecutive days of on-call was essentially non-functional by Sunday. One Sunday evening P1 alert, the on-call engineer responded but was noticeably slower than usual — retrospective revealed they’d been woken 3 consecutive nights, with impaired judgment.

We then split weekday and weekend on-call into separate people. The weekend person does project work Mon-Thu, takes over Friday evening, hands back Monday morning. The weekday person gets Friday evening off.

Lesson: Consecutive on-call beyond 5 days significantly degrades cognitive ability. Don’t test human nature — protect people with systems.

4. Tool Selection Guide

On-Call scheduling tools fall into three categories:

My recommendations:

• Team <5: Don’t bother. Feishu/DingTalk group + rotation spreadsheet is sufficient. Tool overhead exceeds benefit.
• Team 5-15: Get Flashduty or PagerDuty. Focus on scheduling and escalation policy features.
• Team >15: Professional tool is mandatory. Manual scheduling coordination eats a full-time person’s workload.

I once built a Go-based On-Call scheduling system with rotation calendar, alert routing, escalation policies, and auto-generated handoff reports. Development took 3 weeks. Then I discovered Flashduty covered 90% of the requirements. If your core business isn’t building On-Call tools, don’t self-build — spend that time on alert governance and Runbook writing instead. Higher ROI.

5. From Zero to One: Action Checklist

If your team doesn’t have a formal On-Call system yet, here’s the 0-to-1 action list:

Execution rhythm:

• Week 1: Complete P0 items. On-Call without alert tiering and a schedule is just negligence.
• Weeks 2-3: Complete P1 items. Primary-secondary and handoff documents are the foundation of stable operations.
• Week 4: Start P2 items. Metrics dashboard and compensation policy in parallel.
• Month 2+: P3 items quarterly. Satisfaction surveys and drills become routine.

On-Call drills are often overlooked. Each quarter, inject a simulated P1 fault: in a test environment, deliberately cause a failure (e.g., kill the database master node) and have the current on-call engineer run the full response process — receive alert, pull logs, diagnose, execute recovery, write handoff. The drill’s purpose isn’t to test individual ability but to validate the entire On-Call process. If the drill reveals “a Runbook step is outdated” or “a tool permission is misconfigured,” you fix it before a real incident exposes the gap.

After the drill, spend 30 minutes on retrospective. Record three things: was response time within target, was the Runbook executable, was the toolchain smooth. Log drill findings as improvement items, verify fixes in the next quarter’s drill.

Multi-Time-Zone Teams

If your team spans multiple time zones (e.g., Shenzhen + Beijing + overseas), you can implement “Follow the Sun” scheduling: each time zone’s engineers only cover their local working hours, handing off to the next zone. No one does overnight on-call.

But Follow the Sun has prerequisites: at least 3-4 engineers per time zone, and system complexity must allow cross-zone handoffs. If the team is too small, you still need overnight On-Call with fair compensation. Don’t force Follow the Sun — handoff costs may exceed overnight on-call costs.

Summary

On-Call rotation design isn’t just “make a schedule.” It involves 7 dimensions of engineering decisions: shift cadence, primary-secondary mechanism, alert tiering, fatigue quantification, handoffs, new hire development, and compensation.

Three core principles:

1. Only the right person gets woken up — alert tiering + precise routing
2. The person woken up can solve the problem — primary-secondary + Runbook + handoff documents
3. The person woken up won’t quit because of it — fatigue quantification + fair compensation + rotation equity

Google SRE’s standard (toil <50%, team of 6-8, Outalator full-lifecycle management) is the ideal. Most domestic teams have an SRE-to-dev ratio near 1:100, making full replication impractical. But you can start with tools and processes: alert tiering, automated scheduling, handoff documents, metrics dashboards — these don’t depend on team size and can be done today.

Finally, a practical truth: On-Call system quality ultimately comes down to people. Tools and processes are the skeleton; engineer morale and professionalism are the flesh and blood. If engineers feel “on-call is being exploited,” the best tools are wasted. If engineers feel “on-call is an important responsibility for system stability, and the team respects my contribution,” even a DingTalk group rotation can run at a professional level.

Don’t wait for an engineer to resign before fixing On-Call. Fatigue accumulates. When it erupts, you lose not just a person — you lose the troubleshooting knowledge that only existed in their head.

References & Acknowledgments

The following resources were referenced during the writing of this article. Credit to the original authors:

1. Google SRE’s on-call methods and tools — Flashcat, detailed analysis of Google SRE’s OnCall culture, mechanisms, tools, and metrics. Content on toil management, Outalator, and the 5 measurement metrics references this article.
2. Going On-Call: Best Practices for On Call Teams — PagerDuty, comprehensive On-Call practice guide covering technical preparation, team norms, culture building, and management advice. Content on alert fatigue and team health references this guide.
3. Building On-Call Schedules for Humans — Rootly, guide on human-centric On-Call scheduling design. The three-phase new hire onboarding (shadow → reverse shadow → solo) references this article’s Shadowing and Reverse Shadowing methods.
4. Overrides, the Most Human Feature in PagerDuty — PagerDuty Blog, article on On-Call engineer fatigue management and override mechanisms. The On-Call attrition cost ($300,000) data is sourced from this article.
5. Managing On-Call Rotations & Schedules — Squadcast, practical guide on On-Call rotation management. The categorization of rotation challenges (fatigue, false alarms, knowledge transfer) references this article.
6. Efficient OnCall: From Concept to Practice — Flashcat, framework on On-Call’s four core questions (who, when, how, what-if-no-response). The On-Call core questions section references this article.
7. Site Reliability Engineering: How Google Runs Production Systems — Google SRE Team. The On-Call chapters on toil management principles and team size recommendations are an important theoretical reference for this article.