故障复盘改进项跟踪：从清单到闭环的工程化实践

示例：
- 修复 order-service 的 N+1 查询问题（P0，2 天内）
- 为 payment-gateway 增加限流中间件（P1，1 周内）
- 将 Redis 连接池从 lettuce 换成 jedis 并配置合理超时（P2，2 周内）

示例：
- 修改发布流程，所有 P0 级变更必须经过灰度环境验证（P1，2 周内）
- 更新 oncall 值班手册，增加 Redis 集群切换的操作步骤（P2，1 周内）
- 制定配置变更审批制度，所有线上配置变更需双人复核（P1，3 周内）

示例：
- 将订单服务的同步库存扣减改为异步消息驱动（P1，拆分为 5 个子任务，3 个月内完成）
- 核心交易链路多机房容灾建设（P0，拆分为 12 个子任务，6 个月内完成）

示例：
- 编写 Redis 集群脑裂故障的 Runbook（P2，1 周内）
- 将本次故障的排查过程整理为培训材料，在团队内分享（P2，2 周内）

待开始（Open）
   │
   ├──→ 进行中（In Progress）
   │         │
   │         ├──→ 待验收（Pending Verification）
   │         │         │
   │         │         ├──→ 已完成（Done）──→ 已归档（Archived）
   │         │         │
   │         │         └──→ 打回（Rejected）──→ 进行中（In Progress）
   │         │
   │         └──→ 已取消（Cancelled）  ← 发现改进项不再需要
   │
   └──→ 已逾期（Overdue）──→ 升级通知

Issue Type: Postmortem Action Item
自定义字段：
  - postmortem-id: 关联的复盘文档 ID
  - action-type: 技术修复/流程改进/架构优化/知识沉淀
  - verification-method: 自动验收/人工验收
  - verifier: 验收人
  - actual-completion-date: 实际完成日期

# Jira Workflow Definition（简化版）
transitions:
  - from: Open
    to: In Progress
    condition: assignee != null
  - from: In Progress
    to: Pending Verification
    condition: true
  - from: Pending Verification
    to: Done
    condition: verifier_approval == true
  - from: Pending Verification
    to: In Progress
    condition: verifier_approval == false
  - from: Open
    to: Cancelled
    condition: role == "SRE Lead"

# 逾期前 7 天提醒
rule: pre-deadline-reminder
trigger:
  - schedule: "0 9 * * *"
  - condition: duedate <= now() + 7d AND status != Done
action:
  - send_email:
      to: assignee
      subject: "改进项即将到期提醒"
      body: "改进项 {{issue.key}} 将在 {{issue.duedate}} 到期，请尽快处理"

# 逾期后升级通知
rule: overdue-escalation
trigger:
  - schedule: "0 10 * * *"
  - condition: duedate < now() AND status != Done
action:
  - send_email:
      to: [assignee, assignee_manager, sre_lead]
      subject: "改进项已逾期"
      body: "改进项 {{issue.key}} 已逾期 {{days_overdue}} 天"
  - transition_issue:
      to: Overdue

---
postmortem: INC-2026-0721-001
type: technical-fix
priority: P0
assignee: @zhangsan
due-date: 2026-08-04
verifier: @lisi
---

## 改进项描述

为订单服务的下游调用增加超时熔断，超时阈值设为 3 秒。

## 验收标准

- [ ] Prometheus 中出现 order_downstream_timeout_total 指标
- [ ] 压测环境验证熔断生效
- [ ] 代码已合并到 main 分支

## 关联复盘

- 复盘文档：[INC-2026-0721-001 复盘报告](link-to-postmortem)
- 故障时间：2026-07-21 02:30 - 03:15
- 影响：订单服务超时，全量用户 45 分钟

status/open          — 待开始
status/in-progress   — 进行中
status/verification  — 待验收
status/done          — 已完成
status/overdue       — 已逾期
status/cancelled     — 已取消

type/technical       — 技术修复
type/process         — 流程改进
type/architecture    — 架构优化
type/knowledge       — 知识沉淀

priority/p0          — P0 紧急
priority/p1          — P1 重要
priority/p2          — P2 一般

# .github/workflows/action-item-reminder.yml
name: Action Item Reminder
on:
  schedule:
    - cron: "0 1 * * *"  # 每天北京时间 9 点

jobs:
  check-overdue:
    runs-on: ubuntu-latest
    steps:
      - name: Check overdue action items
        uses: actions/github-script@v7
        with:
          script: |
            const issues = await github.rest.issues.listForRepo({
              owner: context.repo.owner,
              repo: context.repo.repo,
              labels: ['postmortem-action-item', 'status/open'],
              state: 'open'
            });
            
            const now = new Date();
            for (const issue of issues.data) {
              const body = issue.body || '';
              const dueMatch = body.match(/due-date:\s*(.+)/);
              if (dueMatch) {
                const dueDate = new Date(dueMatch[1].trim());
                const daysLeft = Math.ceil((dueDate - now) / (1000 * 60 * 60 * 24));
                
                if (daysLeft <= 0) {
                  await github.rest.issues.addLabels({
                    ...context.repo,
                    issue_number: issue.number,
                    labels: ['status/overdue']
                  });
                  await github.rest.issues.createComment({
                    ...context.repo,
                    issue_number: issue.number,
                    body: `⚠️ 此改进项已逾期 ${Math.abs(daysLeft)} 天，请尽快处理。`
                  });
                } else if (daysLeft <= 7) {
                  await github.rest.issues.createComment({
                    ...context.repo,
                    issue_number: issue.number,
                    body: `提醒：此改进项将在 ${daysLeft} 天后到期。`
                  });
                }
              }
            }            

#!/usr/bin/env python3
"""
Postmortem Action Item 自动验收框架
"""

import json
import subprocess
import requests
from datetime import datetime, timedelta
from dataclasses import dataclass, field
from typing import List, Optional, Callable
from enum import Enum

class VerificationResult(Enum):
    PASSED = "passed"
    FAILED = "failed"
    ERROR = "error"

@dataclass
class ActionItem:
    """改进项数据模型"""
    item_id: str
    title: str
    assignee: str
    due_date: str
    priority: str  # P0 / P1 / P2
    action_type: str  # technical-fix / process / architecture / knowledge
    verification_method: str  # auto / manual
    verification_fn: Optional[Callable] = None
    status: str = "open"
    postmortem_id: str = ""
    
@dataclass 
class VerificationReport:
    """验收报告"""
    item_id: str
    result: VerificationResult
    message: str
    checked_at: str
    details: dict = field(default_factory=dict)

class ActionItemVerifier:
    """改进项自动验收器"""
    
    def __init__(self, prometheus_url: str):
        self.prometheus_url = prometheus_url.rstrip("/")
        self.reports: List[VerificationReport] = []
    
    def check_prometheus_metric(self, metric_name: str) -> bool:
        """检查 Prometheus 中是否存在指定 metric"""
        try:
            resp = requests.get(
                f"{self.prometheus_url}/api/v1/query",
                params={"query": metric_name},
                timeout=10
            )
            data = resp.json()
            return data.get("status") == "success" and len(data.get("data", {}).get("result", [])) > 0
        except Exception as e:
            print(f"Prometheus 检查失败: {e}")
            return False
    
    def check_jenkins_job_exists(self, job_name: str) -> bool:
        """检查 Jenkins 是否存在指定 Job"""
        try:
            result = subprocess.run(
                ["jenkins-cli", "list-jobs"],
                capture_output=True, text=True, timeout=10
            )
            return job_name in result.stdout.splitlines()
        except Exception as e:
            print(f"Jenkins 检查失败: {e}")
            return False
    
    def check_grafana_dashboard(self, dashboard_uid: str) -> bool:
        """检查 Grafana 仪表盘是否存在"""
        try:
            resp = requests.get(
                f"http://grafana.internal/api/dashboards/uid/{dashboard_uid}",
                timeout=10
            )
            return resp.status_code == 200
        except Exception as e:
            print(f"Grafana 检查失败: {e}")
            return False
    
    def check_github_pr_merged(self, repo: str, pr_number: int) -> bool:
        """检查 GitHub PR 是否已合并"""
        try:
            resp = requests.get(
                f"https://api.github.com/repos/{repo}/pulls/{pr_number}",
                timeout=10
            )
            data = resp.json()
            return data.get("merged", False)
        except Exception as e:
            print(f"GitHub PR 检查失败: {e}")
            return False
    
    def verify(self, item: ActionItem) -> VerificationReport:
        """执行验收"""
        if item.verification_method != "auto" or item.verification_fn is None:
            return VerificationReport(
                item_id=item.item_id,
                result=VerificationResult.ERROR,
                message="该改进项不支持自动验收",
                checked_at=datetime.now().isoformat()
            )
        
        try:
            passed = item.verification_fn()
            return VerificationReport(
                item_id=item.item_id,
                result=VerificationResult.PASSED if passed else VerificationResult.FAILED,
                message="验收通过" if passed else "验收未通过",
                checked_at=datetime.now().isoformat()
            )
        except Exception as e:
            return VerificationReport(
                item_id=item.item_id,
                result=VerificationResult.ERROR,
                message=f"验收脚本执行出错: {e}",
                checked_at=datetime.now().isoformat()
            )
    
    def batch_verify(self, items: List[ActionItem]) -> List[VerificationReport]:
        """批量验收"""
        reports = []
        for item in items:
            if item.status == "pending-verification":
                report = self.verify(item)
                reports.append(report)
                print(f"[{item.item_id}] {report.result.value}: {report.message}")
        return reports
    
    def generate_summary(self, reports: List[VerificationReport]) -> str:
        """生成验收摘要"""
        total = len(reports)
        passed = sum(1 for r in reports if r.result == VerificationResult.PASSED)
        failed = sum(1 for r in reports if r.result == VerificationResult.FAILED)
        errors = sum(1 for r in reports if r.result == VerificationResult.ERROR)
        
        summary = f"""
=====================================
  改进项自动验收报告
  生成时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}
=====================================
  总计: {total} 条
  通过: {passed} 条
  未通过: {failed} 条
  错误: {errors} 条
  通过率: {(passed/total*100):.1f}%（仅统计可自动验收的改进项）
=====================================
"""
        for r in reports:
            status_icon = "✅" if r.result == VerificationResult.PASSED else "❌"
            print(f"  {status_icon} [{r.item_id}] {r.message}")
        
        return summary


# ============ 使用示例 ============

if __name__ == "__main__":
    verifier = ActionItemVerifier(prometheus_url="http://prometheus.internal:9090")
    
    # 定义改进项（实际应从 Jira 或 GitHub Issues 拉取）
    items = [
        ActionItem(
            item_id="AI-2026-0721-001",
            title="为订单服务增加超时熔断监控指标",
            assignee="zhangsan",
            due_date="2026-08-04",
            priority="P0",
            action_type="technical-fix",
            verification_method="auto",
            verification_fn=lambda: verifier.check_prometheus_metric("order_downstream_timeout_total"),
            status="pending-verification",
            postmortem_id="INC-2026-0721-001"
        ),
        ActionItem(
            item_id="AI-2026-0721-002",
            title="创建 Grafana 订单服务仪表盘",
            assignee="lisi",
            due_date="2026-08-11",
            priority="P1",
            action_type="technical-fix",
            verification_method="auto",
            verification_fn=lambda: verifier.check_grafana_dashboard("order-service-dashboard"),
            status="pending-verification",
            postmortem_id="INC-2026-0721-001"
        ),
    ]
    
    # 批量验收
    reports = verifier.batch_verify(items)
    
    # 生成报告
    print(verifier.generate_summary(reports))

改进项完成率 = 已完成的改进项数 / 总改进项数 × 100%

MTTC = 所有已闭环改进项的（闭环日期 - 创建日期）之和 / 已闭环改进项数

逾期率 = 逾期改进项数 / 总改进项数 × 100%

同类故障复发率 = 复盘后 90 天内再次发生同类故障的次数 / 总复盘次数 × 100%

验收通过率 = 验收通过的改进项数 / 参与验收的改进项数 × 100%

{
  "dashboard": {
    "title": "改进项跟踪度量面板",
    "panels": [
      {
        "title": "改进项完成率（按优先级）",
        "type": "gauge",
        "targets": [
          {
            "expr": "action_items_completed{priority=\"P0\"} / action_items_total{priority=\"P0\"} * 100"
          }
        ]
      },
      {
        "title": "改进项 MTTC 趋势（按类型）",
        "type": "graph",
        "targets": [
          {
            "expr": "avg by (type) (action_item_mttc_seconds{status=\"done\"} / 86400)"
          }
        ]
      },
      {
        "title": "逾期改进项数量",
        "type": "stat",
        "targets": [
          {
            "expr": "count(action_items{status=\"overdue\"})"
          }
        ]
      },
      {
        "title": "同类故障复发率（90天窗口）",
        "type": "graph",
        "targets": [
          {
            "expr": "incident_recurrence_rate_90d * 100"
          }
        ]
      }
    ]
  }
}

#!/usr/bin/env python3
"""
从 Jira 拉取改进项状态并推送 Prometheus 指标
"""

import requests
from prometheus_client import CollectorRegistry, Gauge, push_to_gateway

JIRA_URL = "https://jira.internal"
JIRA_USER = "metrics-reader"
JIRA_TOKEN = "your-token-here"  # 建议从环境变量读取

def fetch_action_items():
    """从 Jira 拉取所有改进项"""
    jql = 'project = SRE AND issuetype = "Postmortem Action Item"'
    headers = {"Authorization": f"Bearer {JIRA_TOKEN}"}
    
    items = []
    start_at = 0
    while True:
        resp = requests.get(
            f"{JIRA_URL}/rest/api/2/search",
            params={"jql": jql, "startAt": start_at, "maxResults": 100,
                    "fields": "status,priority,assignee,created,duedate,resolutiondate"},
            headers=headers,
            timeout=30
        )
        data = resp.json()
        items.extend(data.get("issues", []))
        if start_at + 100 >= data.get("total", 0):
            break
        start_at += 100
    
    return items

def push_metrics(items):
    """推送 Prometheus 指标"""
    registry = CollectorRegistry()
    
    # 按状态统计
    status_gauge = Gauge(
        "action_items_by_status", 
        "Action items count by status",
        ["status", "priority"],
        registry=registry
    )
    
    # 按类型统计
    type_gauge = Gauge(
        "action_items_by_type",
        "Action items count by type", 
        ["type"],
        registry=registry
    )
    
    # MTTC
    from datetime import datetime
    mttc_gauge = Gauge(
        "action_item_mttc_seconds",
        "Mean time to close in seconds",
        ["type"],
        registry=registry
    )
    
    for item in items:
        fields = item["fields"]
        status = fields["status"]["name"].lower().replace(" ", "_")
        priority = fields.get("priority", {}).get("name", "P2")
        
        status_gauge.labels(status=status, priority=priority).inc()
    
    push_to_gateway(
        "pushgateway.internal:9091",
        job="action-item-metrics",
        registry=registry
    )
    print(f"已推送 {len(items)} 条改进项指标到 Prometheus")

if __name__ == "__main__":
    items = fetch_action_items()
    push_metrics(items)