Eliminating Toil: SRE's Approach to Managing Operational Work

Overview

The Google SRE Book contains a frequently cited principle: SRE teams should spend no more than 50% of their total work time on toil. This principle seems simple, but in practice, many SRE teams’ toil ratio far exceeds 50% — some even reach 80% or more.

Why should SRE take “toil” so seriously? Because toil is the invisible killer of reliability:

Toil consumes enormous amounts of time, leaving engineers no energy for work that genuinely improves reliability
Toil typically involves manual operations that are error-prone, actually introducing new incidents
Toil leads to burnout and attrition of talented engineers
Toil doesn’t scale — when the system grows 10x, toil grows 10x too

This article systematically covers how to manage operational work across toil definition and identification, source analysis, automation elimination paths, the 50% cap principle, measurement and tracking methods, and team practices.

For a systematic discussion of toil, see Google SRE Book - Eliminating Toil.

1. Defining and Identifying Toil

What Is Toil

Google SRE defines toil as:

Work that is directly tied to running a production service, that is manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly with service growth.

This definition contains six key characteristics, all of which must be present:

Characteristic	Meaning	Example
Manual	Requires human intervention rather than automatic execution	Manual scaling, manual log cleanup
Repetitive	Not one-time; recurs regularly	Manually modifying config for every release
Automatable	Has clear rules and steps; a machine could do it	Manually checking disk space and cleaning up
Tactical	Reactive rather than proactively planned	Firefighting alert after alert
No enduring value	Completing it produces no reusable output	Manually restarting a service (without improving self-healing)
Scales with growth	As the system grows, workload grows proportionally	Each new server requires manual configuration

What Is NOT Toil

Identifying toil also means identifying what is not toil, to avoid misclassifying valuable work as drudgery:

Work	Is Toil?	Reason
Manually handling a novel, complex incident	❌ No	Not repetitive; requires creative judgment
Writing an automation script to eliminate toil	❌ No	Has enduring value; is engineering work
Attending a Postmortem review meeting	❌ No	Has enduring value; drives system improvement
Daily manual server health inspections	✅ Yes	Manual, repetitive, automatable
Manually answering “how to configure X” questions	✅ Yes	Repetitive, automatable (via documentation)
Code review	❌ No	Requires human judgment, not automatable
Manually executing database migration scripts	✅ Yes	Automatable, scales with growth

Toil Identification Checklist

When you’re unsure whether a task is toil, use this checklist:

□ Is this task a manual operation?
□ Does this task recur (at least once a month)?
□ Does this task have clear steps and rules (could be written as a document for someone else to follow)?
□ After completion, does it produce no reusable output?
□ If the system scale doubled, would this workload also double?

→ 5 "yes": Definite toil, prioritize elimination
→ 3-4 "yes": Likely toil, needs assessment
→ 0-2 "yes": Not toil, normal engineering work

2. Sources of Toil

Seven Major Toil Sources

Toil in SRE teams typically comes from the following seven areas:

1. Manual Operations

The most common toil source. Every operation requires human intervention — both inefficient and error-prone.

Typical scenarios:
  - Manual scaling (kubectl scale)
  - Manual disk space cleanup
  - Manual service restarts
  - Manual config modification and reload
  - Manual database backups

Automation direction: HPA auto-scaling, CronJob periodic cleanup, Kubernetes health check auto-restart, config center hot reload, automated backup scripts.

2. Alert Handling

Not all alert handling is toil, but processing large volumes of low-quality alerts certainly is.

# Typical alert toil scenarios
alert_noise:
  - alert: DiskUsageHigh
    trigger: "Disk usage > 80%"
    frequency: "3-5 times per day"
    action: "SSH into server, clean up logs, restore to 60%"
    toil_assessment: "Fully automatable → auto-cleanup + threshold adjustment"

  - alert: PodRestarted
    trigger: "Pod restart"
    frequency: "10+ times per day"
    action: "Check logs, determine if OOM or Liveness probe failure"
    toil_assessment: "Partially automatable → auto-classify and create tickets"

Automation direction: Alert governance (eliminate noisy alerts), alert auto-remediation (auto-execute fix scripts), alert correlation (merge multiple alerts from the same incident).

3. Deployment and Release

Manual deployment is one of SRE’s biggest toil sources.

Manual deployment process (pure toil):
  1. Pull code from Git
  2. Build image
  3. Push to image registry
  4. Modify Kubernetes YAML
  5. kubectl apply
  6. Manually check service health
  7. Manually notify relevant teams
  → 30-60 minutes per deployment, 5-10 deployments per week

Automation direction: CI/CD pipelines (Jenkins/GitHub Actions/ArgoCD), GitOps auto-sync, post-deployment automated health checks.

4. Certificate and Secret Management

Certificate expiration is a classic toil trap — infrequent but always urgent when it happens.

Typical scenarios:
  - TLS certificate expiry causing service unavailability
  - Emergency certificate renewal across multiple services
  - API key rotation
  - Database password changes

Automation direction: cert-manager for automated certificate management, secret management services (HashiCorp Vault), certificate expiry alerts (30 days in advance).

5. Capacity Management

Manual capacity planning becomes massive toil as system scale grows.

Typical scenarios:
  - Weekly checks of resource utilization for each service
  - Manual adjustment of requests/limits
  - Manual server procurement requests
  - Manual load balancer configuration

Automation direction: HPA/VPA for automatic resource adjustment, Cluster Autoscaler for node provisioning, data-driven capacity forecasting.

6. Dependency Upgrades and Patch Management

Security patches and dependency upgrades are continuous toil.

Typical scenarios:
  - Monthly security patch updates
  - Base image CVE fixes
  - Dependency library version upgrades
  - Kubernetes version upgrades

Automation direction: Automated dependency scanning (Trivy/Grype), base image auto-updates (Renovate/Dependabot), automated rolling upgrades.

7. Documentation and Knowledge Transfer

Repeatedly answering the same questions is hidden toil.

Typical scenarios:
  - "How to view logs for service X?" (asked 10 times/month)
  - "How to configure alerting for X?" (asked 5 times/month)
  - "How to deploy service X?" (asked 8 times/month)
  - Onboarding new hires, repeatedly explaining the same processes

Automation direction: Runbook documentation, internal knowledge base, automated FAQ bots, onboarding documentation.

Toil Source Distribution

In a typical SRE team, toil is distributed roughly as follows:

Toil source distribution (example):
  ┌─────────────────────────────────────────┐
  │ Alert handling   35% ██████████████     │
  │ Manual ops       25% ██████████        │
  │ Deployment       15% ██████             │
  │ Certs/secrets    10% ████              │
  │ Capacity mgmt     8% ███               │
  │ Dependency upg    4% ██                │
  │ Docs/consulting   3% █                 │
  └─────────────────────────────────────────┘

Different teams will have different distributions, but alert handling and manual operations typically occupy the top two spots. These are also the areas with the highest ROI for automation investment.

3. Automation Elimination Paths

Levels of Automation

Eliminating toil is not a one-step process but a gradual, layered progression:

Level	Characteristic	Example	Human Role
L1 Manual execution	Fully human-operated	Manually SSH into server to clean logs	Executor
L2 Script-assisted	Has scripts but requires manual triggering	Run cleanup script, manually verify results	Trigger + Verifier
L3 Scheduled automatic	Timed auto-execution, human monitoring	CronJob periodic cleanup, alert monitoring	Monitor
L4 Condition-triggered	Auto-triggered based on conditions	Disk > 80% triggers auto-cleanup	Exception handler
L5 Self-healing closed loop	Auto-detect → decide → execute → verify	Auto-detect anomaly → diagnose → fix → verify	Improver

Each level up reduces human involvement by one degree. The goal is to move people from “executor” to “improver” — people no longer do repetitive operations but optimize the automation system itself.

Automation Decision Framework

Not all toil is worth automating. You need to evaluate ROI:

Automation ROI = (Time saved × Frequency × Expected lifetime) / Automation development cost

Example:
  Task: Manual disk cleanup
  Time saved: 15 minutes per occurrence
  Frequency: 3 times per week
  Expected lifetime: 12 months
  Automation development cost: 4 hours

  ROI = (15min × 3 × 52 weeks) / (4h × 60min) = 2340 / 240 = 9.75
  → ROI is nearly 10, strongly recommend automating

Automation priority matrix:

	High frequency	Low frequency
High effort	🔴 Prioritize automation (manual scaling, manual deployment)	🟡 Worth automating (quarterly capacity assessment)
Low effort	🟡 Worth automating (alert acknowledgment, log queries)	🟢 Don’t automate yet (annual architecture review)

Automation Implementation Steps

Using “manual disk cleanup” as an example, here’s the complete path from toil to automation:

Step 1: Document the manual procedure

# Current manual procedure
ssh prod-server-01
du -sh /var/log/*
rm -rf /var/log/app/*.log.2025*
find /tmp -type f -mtime +7 -delete
df -h | grep /var

Step 2: Encapsulate as a script

#!/bin/bash
# disk_cleanup.sh - Automated disk space cleanup
# Usage: ./disk_cleanup.sh [threshold]

THRESHOLD=${1:-80}
LOG_DIR="/var/log/app"
TMP_RETENTION_DAYS=7

current_usage=$(df /var | tail -1 | awk '{print $5}' | tr -d '%')

if [ "$current_usage" -lt "$THRESHOLD" ]; then
    echo "Disk usage ${current_usage}% below threshold ${THRESHOLD}%, no action needed."
    exit 0
fi

echo "Disk usage ${current_usage}% exceeds threshold ${THRESHOLD}%, starting cleanup..."

# Clean old logs
find "$LOG_DIR" -name "*.log" -mtime +7 -delete
echo "Cleaned logs older than 7 days in $LOG_DIR"

# Clean temporary files
find /tmp -type f -mtime +${TMP_RETENTION_DAYS} -delete
echo "Cleaned /tmp files older than ${TMP_RETENTION_DAYS} days"

# Post-cleanup check
new_usage=$(df /var | tail -1 | awk '{print $5}' | tr -d '%')
echo "Cleanup complete. Disk usage: ${current_usage}% → ${new_usage}%"

if [ "$new_usage" -gt "$THRESHOLD" ]; then
    echo "WARNING: Disk usage still above threshold after cleanup!"
    exit 1
fi

Step 3: Scheduled automatic execution

# Kubernetes CronJob
apiVersion: batch/v1
kind: CronJob
metadata:
  name: disk-cleanup
  namespace: production
spec:
  schedule: "0 */6 * * *"    # Run every 6 hours
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: cleanup
            image: busybox:latest
            command: ["/bin/bash", "/scripts/disk_cleanup.sh", "75"]
            volumeMounts:
            - name: log-volume
              mountPath: /var/log/app
            - name: script-volume
              mountPath: /scripts
          volumes:
          - name: log-volume
            persistentVolumeClaim:
              claimName: log-pvc
          - name: script-volume
            configMap:
              name: cleanup-scripts
          restartPolicy: OnFailure

Step 4: Condition-triggered (alert-driven)

# Prometheus AlertManager triggers automated cleanup
- alert: DiskSpaceHigh
  expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 80
  for: 5m
  labels:
    severity: warning
    auto_remediate: true
  annotations:
    description: "Disk usage on {{ $labels.instance }} is {{ $value }}%"
    remediation: "disk_cleanup"

# Auto-remediation webhook receives alert and triggers cleanup
@app.route('/webhook', methods=['POST'])
def handle_alert():
    alert = request.json['alerts'][0]
    if alert['labels'].get('auto_remediate') == 'true':
        remediation = alert['annotations'].get('remediation')
        instance = alert['labels']['instance']

        if remediation == 'disk_cleanup':
            # Trigger cleanup Job
            kubectl_run_job(f'disk-cleanup-{instance}', 'cleanup-image',
                          ['disk_cleanup.sh', '75'])
            log.info(f'Triggered disk cleanup on {instance}')

    return jsonify({'status': 'ok'})

Step 5: Self-healing closed loop

The final automation is not just “execute cleanup” but a complete closed loop:

Detect (disk > 80%)
  → Diagnose (which directory is using the most? logs or data?)
  → Decide (logs can be cleaned, data cannot)
  → Execute (clean logs)
  → Verify (did disk drop below threshold?)
  → Record (log cleanup event for audit)
  → Alert (if still high after cleanup, alert for human intervention)

Automation Risk Control

Automation is not a panacea. Bad automation is more dangerous than manual operation — because it produces errors faster and more consistently.

Automation safety checklist:

automation_safety_checklist:
  - idempotent: true           # Operations are idempotent; repeated execution has no side effects
  - bounded_blast_radius: true # Limit impact scope (clean one node at a time)
  - rollback_capable: true     # Has rollback mechanism
  - dry_run: true              # Supports dry-run mode
  - audit_logged: true         # All automated operations are audit-logged
  - rate_limited: true         # Has rate limiting to prevent automation runaway
  - human_override: true       # Humans can interrupt at any time

# Safe automation wrapper
class SafeAutomation:
    def __init__(self, service, max_blast_radius=1, dry_run=False):
        self.service = service
        self.max_blast_radius = max_blast_radius
        self.dry_run = dry_run

    def execute(self, action, targets):
        # 1. Limit blast radius
        if len(targets) > self.max_blast_radius:
            raise Exception(f"Targets ({len(targets)}) exceed max blast radius "
                          f"({self.max_blast_radius})")

        # 2. Record audit log
        audit_log(action, targets, self.dry_run)

        # 3. Dry-run mode
        if self.dry_run:
            log.info(f"[DRY RUN] Would execute {action} on {targets}")
            return {"status": "dry_run", "action": action, "targets": targets}

        # 4. Execute operation
        result = self._do_action(action, targets)

        # 5. Verify result
        if not self._verify(action, targets, result):
            log.error(f"Verification failed for {action}, initiating rollback")
            self._rollback(action, targets)
            raise Exception(f"Action {action} verification failed, rolled back")

        return result

4. The 50% Toil Cap Principle

What the Principle Means

Google SRE’s 50% principle:

SRE teams should spend no more than 50% of their total work time on toil. The other 50% should be spent on engineering work — automation, tooling, reliability improvements, capacity planning, etc.

This principle is not an arbitrary number — it’s a resource allocation constraint:

If toil exceeds 50%, it indicates fundamental problems in system design or operational processes
If toil exceeds 50%, SRE degenerates into traditional operations — only executing, not improving
50% engineering time is the key guarantee that distinguishes SRE from traditional ops

Why 50% and Not Lower

The ideal toil ratio is of course as low as possible, but 50% is a practical lower bound:

New toil always emerges: Systems continuously evolve, and new toil constantly appears
Automation itself needs maintenance: Automation systems aren’t one-and-done; they need continuous maintenance
Unpredictable incidents: There will always be some incidents requiring human intervention
Time for learning and research: Evaluating new technologies and designing architecture solutions takes time

How to Measure Toil Ratio

Measuring toil ratio is a prerequisite for management. Here are three measurement methods:

Method 1: Time Tracking

# Weekly time tracking
weekly_time_tracking:
  toil:
    alert_handling: 6h      # Alert handling
    manual_ops: 4h           # Manual operations
    deployment: 3h           # Deployment
    cert_management: 1h      # Certificate management
    user_support: 2h         # User support
    total: 16h

  engineering:
    automation_dev: 8h       # Automation development
    tooling: 4h              # Tool development
    capacity_planning: 3h    # Capacity planning
    documentation: 2h        # Documentation
    postmortem: 3h           # Postmortems
    total: 20h

  toil_ratio: 16 / (16 + 20) = 44%

Method 2: Ticket Classification

# Auto-calculate toil ratio from ticketing system
def calculate_toil_ratio(team, week):
    tickets = query_tickets(team=team, week=week)

    toil_categories = [
        "manual_ops", "alert_handling", "deployment",
        "cert_renewal", "user_support", "routine_maintenance"
    ]
    engineering_categories = [
        "automation", "tooling", "capacity_planning",
        "documentation", "postmortem", "research"
    ]

    toil_hours = sum(t.hours for t in tickets if t.category in toil_categories)
    eng_hours = sum(t.hours for t in tickets if t.category in engineering_categories)

    return toil_hours / (toil_hours + eng_hours)

Method 3: On-Call Reports

# On-Call Weekly Report Template - Toil Statistics

## This Week's On-Call Stats
- Total alerts: 47
- Required human intervention: 12 (26%)
- Average handling time: 18 minutes
- Total on-call time: 3.6 hours

## Toil Breakdown
- Manual service restarts: 3 times, 45 minutes
- Manual scaling: 2 times, 30 minutes
- Config changes: 4 times, 60 minutes
- Alert acknowledgment (no action needed): 3 times, 15 minutes

## Automatable Items
- [ ] Manual restarts → Kubernetes health check self-healing
- [ ] Manual scaling → HPA auto-scaling
- [ ] Config changes → Config center hot reload

Action Plan When Exceeding 50%

When toil ratio consistently exceeds 50%, systemic action is needed:

toil_reduction_plan:
  trigger: "toil_ratio > 50% for 2 consecutive weeks"

  actions:
    1_freeze:
      description: "Freeze non-urgent engineering work, focus on eliminating toil"
      duration: "1-2 weeks"

    2_audit:
      description: "Comprehensive audit of toil sources, sorted by time spent"
      output: "toil heat map"

    3_prioritize:
      description: "Select highest-ROI toil items for priority automation"
      criteria: "Time spent × Frequency / Automation cost"

    4_escalate:
      description: "If toil consistently exceeds threshold, escalate to management"
      message: "Toil ratio consistently above 50%, indicating structural issues
               in system design or processes that require management resource investment"

    5_root_cause:
      description: "Root cause analysis for toil overrun"
      common_causes:
        - "Unreasonable system architecture, high operational complexity"
        - "Poor monitoring quality, too many noisy alerts"
        - "Insufficient automation coverage, lots of manual operations"
        - "Understaffed team"
        - "New systems launched without considering operational costs"

5. Measurement and Tracking Methods

Toil Metrics

Metric	Definition	Target
Toil ratio	Toil time / Total work time	< 50%
Absolute toil hours	Total toil hours per week	Declining month-over-month
Alert human intervention rate	Alerts requiring human action / Total alerts	< 30%
Automation coverage	Automated operations / Total operations	> 80%
Repeat ticket rate	Repeat-type tickets / Total tickets	Declining month-over-month
MTTR (manual portion)	Time spent on manual operations in incident resolution	Declining month-over-month

Toil Tracking Dashboard

# Monthly Toil Tracking Dashboard
toil_dashboard:
  period: "2026-07"
  team: "SRE Platform"

  summary:
    toil_ratio: 38%           # ↓ from 45% last month
    toil_hours: 60h           # ↓ from 72h
    engineering_hours: 98h
    target: "<50%"
    status: "🟢 on track"

  toil_breakdown:
    alert_handling: 20h       # 33% of toil
    manual_ops: 15h           # 25%
    deployment: 12h           # 20%
    cert_management: 5h       # 8%
    user_support: 5h          # 8%
    other: 3h                 # 6%

  automation_progress:
    - item: "Automated disk cleanup"
      status: "completed"
      toil_eliminated: "4h/week"
    - item: "Automated alert classification"
      status: "in_progress"
      estimated_savings: "3h/week"
    - item: "Automated certificate renewal"
      status: "planned"
      estimated_savings: "2h/week"

  trend:
    - month: "2026-04"
      toil_ratio: 58%
    - month: "2026-05"
      toil_ratio: 52%
    - month: "2026-06"
      toil_ratio: 45%
    - month: "2026-07"
      toil_ratio: 38%

Periodic Toil Audit

A comprehensive toil audit is recommended quarterly:

# Quarterly Toil Audit Template

## Audit Scope
- Time range: 2026 Q2
- Participants: All SRE team members

## Toil Inventory
| # | Toil Description | Frequency | Time per Occurrence | Monthly Total | Automatable | Priority |
|---|---------|------|---------|-----------|---------|--------|
| 1 | Manual disk cleanup | 3x/week | 15min | 180min | ✅ Yes | P0 |
| 2 | Manual alert acknowledgment | 10x/day | 3min | 600min | ✅ Yes | P0 |
| 3 | Manual deployment | 3x/week | 45min | 540min | ✅ Yes | P0 |
| 4 | Certificate renewal | 1x/quarter | 120min | 40min | ✅ Yes | P1 |
| 5 | Manual inspection reports | 1x/day | 30min | 900min | ✅ Yes | P1 |
| 6 | New hire onboarding | 1x/quarter | 240min | 80min | Partial | P2 |

## Elimination Plan
- Q3 target: Eliminate #1, #2, #3, estimated 22h/month toil reduction
- Q4 target: Eliminate #4, #5, estimated 12h/month toil reduction

6. Team Practices

Building a “Zero Tolerance for Toil” Culture

This doesn’t mean eliminating all toil immediately, but establishing an attitude: every time you encounter toil, record it and create an elimination plan.

Daily practices:
  - When encountering toil, first record it in the ticketing system
  - If it can be automated in 5 minutes, do it immediately
  - If it needs more time, add to backlog
  - Review new toil items weekly
  - Conduct toil audits quarterly

“Toil Tuesday” Practice

Some teams dedicate fixed time for toil elimination:

toil_tuesday:
  schedule: "Every Tuesday afternoon, 2 hours"
  rules:
    - "No daily toil during this time"
    - "Focus on eliminating one toil item"
    - "Can work solo or in pairs"
    - "Update tracking dashboard when done"
  examples:
    - "Convert manual alert acknowledgment to auto-classification script"
    - "Convert manual deployment to semi-automated pipeline"
    - "Write FAQ documentation"

New Hire Onboarding and Toil

New hires often inherit toil work. The right approach:

Wrong approach:
  New hire joins → Assign daily toil → New hire becomes toil executor → New hire burnout

Right approach:
  New hire joins → Assign daily toil (to learn the system)
               → Require new hire to automate one toil item within 1 month
               → New hire learns the system AND contributes automation improvements

Preventing “Automation-Generated Toil”

Automation systems themselves can become new toil sources:

Original toil: Manual disk cleanup (3x/week, 15min each)
After automation: CronJob auto-cleanup
New toil: CronJob occasionally fails, requires manual investigation and fix

Counter-strategies:

Automation systems need self-monitoring: Alert when automated tasks fail
Keep automation systems simple: Overly complex automation may cost more to maintain than it saves
Periodically review automation systems: Quarterly checks of whether automation is still effective and whether it has created new toil

7. Advanced Thinking on Toil Elimination

From Eliminating Toil to Eliminating the Root Cause of Toil

The highest form of toil elimination is not automating it, but eliminating the root cause that produces the toil:

Toil: Manually clean disk space weekly
  → Automation: CronJob periodic cleanup (treats the symptom)
  → Root fix: Why does disk fill up?
    → Unreasonable log retention policy → Fix log rotation policy
    → Log level set too high → Adjust log level
    → Insufficient disk capacity → Expand or use object storage

Architectural-Level Toil Elimination

Many toil sources are rooted in architecture design problems. Eliminating toil at the architectural level is the most thorough approach:

Toil Source	Architecture-Level Solution
Manual scaling	Stateless design + HPA auto-scaling
Manual failover	Multi-active architecture + automatic failover
Manual config management	Declarative configuration + GitOps
Manual certificate management	cert-manager auto-issuance
Manual log cleanup	Centralized logging + auto-rotation policies

From Toil to Engineering

SRE’s value is not in how many tickets were handled, but in how many future tickets were eliminated. Every toil elimination should produce engineering value:

Handle one toil → Write automation script → Consolidate into tool/platform → Team-wide sharing → Continuous improvement
(Pure execution)    (Individual efficiency)    (Team efficiency)     (Organizational efficiency)  (Long-term value)

Each step in this chain elevates the work from “individual time savings” to “organizational capability improvement.”

Summary

Eliminating toil is not a one-time project but a continuous engineering practice. Core principles:

Identification is the prerequisite: Use the six-characteristic definition to identify toil, use time tracking to quantify toil ratio
50% is the floor: Toil above 50% is consuming SRE’s engineering capacity; action must be taken
Automation is the means: Progress from manual → script → scheduled → condition-triggered → self-healing closed loop
Root cause elimination is the goal: Don’t just automate toil — eliminate the root causes that produce toil (architecture, processes, design)
Measurement is the safeguard: Continuously track toil ratio and trends, use data to drive toil governance

A healthy SRE team should have these characteristics: the system is growing, team size isn’t growing linearly, toil ratio is continuously declining, and engineers spend most of their time on design and building, not firefighting.

Remember: If you’re doing the same thing every week, you’re doing toil. Delegate repetitive work to machines and reserve creative work for people — that’s the value of SRE.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

Google SRE Book - Eliminating Toil — Google SRE Team, referenced for Google SRE Book - Eliminating Toil

Overview#

1. Defining and Identifying Toil#

What Is Toil#

What Is NOT Toil#

Toil Identification Checklist#

2. Sources of Toil#

Seven Major Toil Sources#

1. Manual Operations#

2. Alert Handling#

3. Deployment and Release#

4. Certificate and Secret Management#

5. Capacity Management#

6. Dependency Upgrades and Patch Management#

7. Documentation and Knowledge Transfer#

Toil Source Distribution#

3. Automation Elimination Paths#

Levels of Automation#

Automation Decision Framework#

Automation Implementation Steps#

Automation Risk Control#

4. The 50% Toil Cap Principle#

What the Principle Means#

Why 50% and Not Lower#

How to Measure Toil Ratio#

Action Plan When Exceeding 50%#

5. Measurement and Tracking Methods#

Toil Metrics#

Toil Tracking Dashboard#

Periodic Toil Audit#

6. Team Practices#

Building a “Zero Tolerance for Toil” Culture#

“Toil Tuesday” Practice#

New Hire Onboarding and Toil#

Preventing “Automation-Generated Toil”#

7. Advanced Thinking on Toil Elimination#

From Eliminating Toil to Eliminating the Root Cause of Toil#

Architectural-Level Toil Elimination#

From Toil to Engineering#

Summary#

References & Acknowledgments#