Building an SRE Team: From Hiring to Organizational Capability Model

Overview

3 AM. The core trading system is down. The on-call engineer frantically flips through the Runbook. The DBA says it’s not a database issue. The network team says the links are fine. The developers say nothing changed. Three teams point fingers at each other. Incident recovery drags on for 47 minutes.

This is the reality of operations in many companies. The problem isn’t that people aren’t trying hard enough. The problem is there’s no engineering-driven reliability team to decompose the problem.

SRE (Site Reliability Engineering) was a concept Google introduced in 2003. Ben Treynor Sloss brought together a group of software engineers to solve operations problems with code, not by throwing more bodies at them. (The Google SRE Book tells this story clearly.)

But “building an SRE team” is far more complex than “hiring a few SRE engineers.” This note records practical experience from building SRE teams from scratch — how to design roles, hire people, assess capabilities, and lead the team. No theoretical frameworks here, just real pitfalls and proven approaches.

Why Build an SRE Team Instead of Sticking with Traditional Operations

Let’s address a fundamental question first: what’s the real difference between SRE and traditional operations?

Traditional operations teams operate on a “manual ops” model — troubleshooting by experience, executing daily tasks by hand, passing knowledge through apprenticeship. The more people you add, the more systems you can cover, but efficiency doesn’t improve. The core problem is that operational workload scales linearly with system size, and you can’t hire people indefinitely.

Google’s approach was to replace repetitive operations with software engineering. In an SRE team, each person spends no more than 50% of their time on pure operational tasks. The rest must go to engineering improvements — writing automation tools, designing monitoring systems, optimizing deployment pipelines. Google explicitly requires SRE teams to keep toil below 50% in the “50% Rule.” (Google SRE Book - Eliminating Toil)

Here’s the comparison:

One sentence: traditional ops “carries the system with manpower,” SRE “raises the system with code.”

SRE Team Organizational Design

Three Common Models

How you organize depends on company size and business complexity. I’ve seen three mainstream models, each with its own sweet spot:

Model 1: Centralized SRE Team

A standalone SRE department serving all business lines. Suitable for small-to-mid companies (50-500 person tech team).

CTO
 └── SRE Department
      ├── Infrastructure Group (datacenter, network, storage)
      ├── Platform Tools Group (CI/CD, monitoring, logging)
      ├── Reliability Group (SLO, chaos engineering, capacity planning)
      └── On-Call Rotation Group

The upside: unified tech stack, reusable toolchain, consistent standards. The downside: detached from the business, sometimes不理解 why the business needs things done a certain way.

Model 2: Embedded SRE

SRE engineers are distributed across business lines, reporting to the business line with a dotted-line to the SRE department. Suitable for large companies (500+ tech team, multiple business lines).

CTO
 ├── Trading Business Line
 │    ├── Development Team
 │    └── SRE (belongs to business line, dotted-line to SRE dept)
 ├── Risk Control Business Line
 │    ├── Development Team
 │    └── SRE
 └── SRE Platform Department (provides shared tools and standards)

The upside: SRE deeply understands the business, fast response. The downside: hard to unify standards, each business line does its own thing.

Model 3: Platform + Embedded Hybrid

This is the model used by Google and other big tech companies. A central SRE platform team builds infrastructure and tools, while embedded SREs in each business line focus on deep business reliability. The platform team “builds the weapons,” the embedded SREs “fight the battles.”

CTO
 ├── SRE Platform Department
 │    ├── Monitoring & Alerting Platform Group
 │    ├── Container & Orchestration Group
 │    ├── Change Management Group
 │    └── Reliability Engineering Group
 ├── Business Line A
 │    └── Embedded SRE (uses platform tools, deep business focus)
 └── Business Line B
      └── Embedded SRE

My Recommendation

For tech teams under 100 people, go centralized. For 100-500, centralized with sub-groups by business line. For 500+, consider platform + embedded.

Don’t copy Google’s model on day one. Google has thousands of SREs and can pull off embedded because their platform infrastructure is mature enough. If you don’t even have monitoring figured out and go embedded, every business line will build its own thing — worse than traditional ops.

Role Design and Responsibility Division

Core SRE Roles

An SRE team needs the following role types. Not necessarily one person per role — small teams can have one person wearing multiple hats:

1. SRE Engineer (Core Role)

The backbone of the SRE team. Requires both systems operations skills and coding skills. Daily work includes:

• Participating in architecture reviews, providing reliability recommendations
• Writing automation tools (deployment, inspection, self-healing)
• Designing and implementing monitoring and alerting
• Participating in on-call and incident response
• Writing and maintaining Runbooks

The most critical requirement: can write code. Not “knows shell scripting” — can write Go/Python services and maintain infrastructure code (Terraform/Ansible).

2. Platform Engineer

Responsible for building internal developer platforms — enabling development teams to self-serve deployment, monitoring, autoscaling, and other operations. This role leans toward architecture design and requires understanding developer team needs and pain points.

3. Reliability Engineer

Focused on SLO design, error budget management, chaos engineering, and capacity planning. This role doesn’t necessarily need strong coding skills but must have deep understanding of reliability theory and the ability to drive business teams toward reliability decisions.

4. On-Call Engineer

The shift role responsible for first-line incident response. In small teams, this is a rotating duty shared by SRE engineers. Large teams can have dedicated roles.

Staffing Ratio Recommendations

The Dev:SRE ratio is an empirical value. Google runs approximately 1:7 to 1:10 (depending on business complexity), but Google’s SRE bar is extremely high. For most domestic companies, 1:10 to 1:15 is a reasonable starting ratio.

SRE Capability Model

Capability Matrix

The following capability matrix is distilled from actual team management experience, organized into four dimensions and three levels.

Four Dimensions:

1. Systems Engineering — Linux kernel, networking, storage, databases
2. Coding & Automation — At least one programming language, IaC tools, CI/CD
3. Observability — Monitoring, logging, distributed tracing, alerting design
4. Reliability Engineering — SLO, error budgets, failure analysis, capacity planning

Three Levels:

Full capability matrix:

Assessing Candidates During Hiring

The biggest headache in hiring SRE: how do you tell a real SRE from a repackaged traditional ops engineer?

My method: four interview rounds, each with a different focus.

Round 1: Online Coding (30 minutes)

Not a LeetCode algorithm problem. Give a real ops scenario and ask the candidate to write code. For example:

“Write a Python script that checks all Pods’ resource usage in a Kubernetes cluster, finds Pods with CPU usage above 80% and insufficient Requests, and outputs to CSV.”

This tests: Python coding ability, Kubernetes API usage, understanding of Request/Limit. A traditional ops engineer most likely can’t do this.

Round 2: System Troubleshooting (45 minutes)

Give a real incident scenario and ask the candidate to describe their debugging approach. For example:

“Online service latency went from 50ms to 500ms. CPU usage is normal, memory at 60%, network bandwidth normal. How do you troubleshoot?”

Good answer: starts at the application layer — check GC logs, database slow queries, lock contention. Bad answer: try restarting.

Round 3: System Design (45 minutes)

Ask the candidate to design a monitoring system. For example:

“The company has 50 microservices and 300 servers. Design a monitoring solution covering infrastructure, middleware, and application layers, with minute-level fault detection.”

Tests: monitoring layered thinking, tool selection, alerting strategy design.

Round 4: Culture & Values (30 minutes)

Discuss views on on-call, postmortems, and automation. Assess engineering culture awareness.

“What do you think SRE’s most important responsibility is?” “During a postmortem, if you find the incident was caused by someone’s operational mistake, how do you handle it?”

The second question’s answer matters a lot. If the candidate says “punish that person,” they don’t understand SRE culture yet. The right direction: analyze why this operation went wrong (was there a missing guardrail? was automation insufficient?), and drive systemic improvement.

Team Pipeline Development

The 0-to-1 Building Path

Phase 1: Core 1-2 People (0-3 months)

Find 1-2 L2/L3 SREs and set up basic monitoring and alerting. Don’t hire L1 at this stage — without L2/L3 mentors, L1 engineers will flounder.

Goal: make the team know what the system is doing. Monitoring coverage from 0 to 60%, core services have alerts.

Phase 2: Expand to 3-5 People (3-6 months)

Add 1-2 L1s and 1 L2. L1 handles daily on-call and operations; L2 builds platform tools.

Goal: monitoring coverage 80%, basic CI/CD pipeline running, core services have Runbooks.

Phase 3: Forming Team 5-10 People (6-12 months)

Add platform engineers and reliability engineers. Start doing SLO design, chaos engineering, capacity planning.

Goal: core services have SLOs, complete first chaos engineering exercise, establish on-call rotation.

Phase 4: Mature Team 10+ People (12 months+)

Split into platform team and business SRE teams. Platform team builds tools; business SREs focus on business lines.

Goal: automation coverage 70%+, Toil below 50%, complete reliability measurement system.

Talent Development Mechanisms

Mentorship

Pair each L1 with an L2/L3 mentor. Mentor responsibilities: code review, technical design review, career advice. This is far more effective than throwing documents at new hires.

Rotation

Rotate SRE engineers across business lines. Benefit: prevents knowledge silos. Drawback: short-term efficiency dip. Recommend rotating every 6-12 months.

Postmortem-Driven Learning

Every postmortem is a team learning opportunity. Postmortem meetings require full attendance (make-up if the on-call person is unavailable). Improvement items from postmortems are assigned to individuals, and results are reviewed at the next postmortem.

Google SRE teams have a principle called “Blameless Postmortem.” The purpose of a postmortem is to “understand what happened,” not to “find who to blame.” (Google SRE Book - Postmortem Culture) This principle must be enforced consistently. Otherwise, the team will start hiding problems — that’s the real danger.

Performance and Incentives

SRE OKR Design

SRE team performance can’t be measured by “how many tickets were processed” — that encourages ticket creation. Use reliability metrics and engineering metrics instead.

O1: Improve System Reliability

O2: Improve Automation Coverage

O3: Knowledge Accumulation

On-Call Incentives

On-call is the hardest work in an SRE team. You can’t let on-call engineers suffer without reward — otherwise no one will want to take shifts.

My approach:

1. On-call compensation — 200 CNY/day on weekdays, 500 CNY/day on weekends, 800 CNY/day on holidays
2. Post-on-call compensatory leave — woken up at night for incident handling, half-day off the next day
3. Quarterly On-Call Star — voted by the team, additional bonus

Recommended rotation period: 1-2 weeks per shift. Too short means high handoff costs; too long leads to fatigue buildup.

Pitfall Guide

Pitfall 1: Hiring Traditional Ops with SRE Label

The most common pitfall. Hiring a bunch of “ops who can write shell scripts,” slapping on the SRE title, but actually doing traditional ops work.

How to avoid: interviews must test coding. Not algorithms — real engineering problems. If the candidate can’t even call Kubernetes APIs, they’re not SRE material yet.

Pitfall 2: SRE Becomes the Scapegoat

“The system is down — SRE’s fault.” This perception needs correcting early. SRE’s responsibility is to “ensure reliability,” but reliability is designed in, not just operated. If development teams deploy without testing, without canary releases, without rate limiting, SRE can’t cover for everything.

Solution: establish a change management process. Any deployment must pass SRE review (at least for core services). CI/CD pipelines should enforce quality gates. SRE has the right to reject deployments that don’t meet standards.

Pitfall 3: More Tools, More Chaos

SRE teams easily fall into the “reinventing wheels” trap. Monitoring with Prometheus but also running Zabbix. CI/CD with Jenkins but also setting up GitLab CI. Alerting with Alertmanager but also building a custom alerting platform.

More tools isn’t better. Each additional tool means more maintenance overhead. My principle: use open-source instead of building in-house; use one set instead of two. If existing tools genuinely can’t meet requirements, evaluate alternatives first. Only build in-house as a last resort.

Pitfall 4: Over-Pursuing SLO Precision

Some teams want to design SLOs with extreme precision from day one — four nines, five nines, Six Sigma. Higher precision means higher engineering cost.

Google’s experience: 99.9% is enough for most services. Users can’t tell the difference between 99.9% and 99.95%. Instead of chasing that 0.05% improvement, cut MTTR from 30 minutes to 10 minutes. (Google SRE Book - Service Level Objectives)

Pitfall 5: Team Culture Isn’t a Slogan

“Blameless postmortem” and “automate everything” aren’t posters on the wall. They need the team leader to embody them in every postmortem and every review. If the leader starts assigning blame during postmortems, the team’s trust foundation crumbles fast.

Pitfall 6: Treating AI as a Silver Bullet

AIOps is hot in 2026. Many companies think deploying AI will solve all ops problems. In reality, if your monitoring data quality is poor and alerting strategy is flawed, AI just makes you receive more junk alerts faster. Build the foundation first — standardize monitoring data, optimize alerting strategy, accumulate high-quality historical data — then talk about AI.

SRE and Development Team Collaboration Boundaries

If this isn’t resolved, the team will be built but consumed by internal friction.

SRE owns: infrastructure, platform tools, monitoring and alerting, on-call response, reliability strategy

Development owns: business code, unit tests, performance optimization, business monitoring instrumentation

Shared responsibility: SLO definition, capacity planning, postmortems

Key principle: SRE isn’t the development team’s babysitter, and development isn’t SRE’s client. They collaborate. SRE provides the platform and standards; development self-serves on the platform. If development teams need SRE to hand-hold deployments, the platform isn’t good enough yet — that’s SRE’s problem, not development’s.

Summary

Building an SRE team comes down to three things:

Hire the right people. SRE’s core skills are coding + system understanding + reliability thinking. Interview with real engineering problems — don’t get fooled by certifications on a resume.

Structure it well. Organizational structure should match company size. Centralized for small teams, hybrid for large. Don’t copy big tech models — solve your own problems first.

Nurture the culture. Blameless postmortems, the 50% rule, automation-first. These aren’t slogans — they need to be lived daily.

Team building is a long-term effort. Don’t expect results in six months. The fastest team I’ve seen took 12 months to become effective. But once it clicks, the SRE team’s returns far exceed its costs — more stable systems, faster recovery, higher dev efficiency, and controlled tech debt.

One last thing: the SRE team’s value isn’t “preventing all failures” — it’s “recovering fast when failures happen, and never making the same mistake twice.” That’s the essence of reliability engineering.

References & Acknowledgments

This article referenced the following materials during writing. Thanks to the original authors for their contributions:

1. Google SRE Book - Introduction — Google SRE Team, origin and core philosophy of SRE
2. Google SRE Book - Eliminating Toil — Google SRE Team, the 50% rule and toil management
3. Google SRE Book - Postmortem Culture — Google SRE Team, blameless postmortem culture
4. Google SRE Book - Service Level Objectives — Google SRE Team, SLO design methods and practices
5. What Does a Site Reliability Engineer Do? — Coursera, SRE role responsibilities and skill requirements
6. Building your digital transformation team: 4 essential roles — Red Hat, team role design and responsibility division