Building an Automated Inspection Platform from Scratch: Plugin Architecture Design and Go Implementation Guide

Overview

Three ops engineers, 200 servers. Every morning: SSH into each one, check disk, memory, CPU, connection count, certificate expiry… the entire morning is gone. When a sudden outage hits, there is no time to inspect — the problem has already exploded in user complaints.

This is not an isolated case. Many small-to-mid teams still do ops inspection the “manual + script” way — a few Shell scripts scattered across machines, unmaintained, nobody knows when they last ran, and nobody reads the output.

An automated inspection platform solves exactly this problem. It is not just “centralizing scripts to run together” — it is a complete system: plugin-based check items, a concurrent execution engine, flexible alerting strategies, and readable inspection reports. This article covers everything from architecture design to code implementation, showing how to build a production-ready inspection platform in Go.

Why Go instead of Python? Three reasons: single-binary deployment, goroutine concurrency model naturally suited for inspection workloads, and cross-compilation makes distribution to different server architectures trivial. Python can do it too, but the pain of distributing Python environments and dependencies across 200 machines is something anyone who has tried it understands.

What Problem Does an Inspection Platform Solve

Pain Points of Traditional Inspection

Let us look at how painful “manual inspection” really is:

Pain Point	Symptom	Impact
Incomplete coverage	Only 50 of 200 machines get checked regularly; the rest “have never had issues”	Hidden risks accumulate, exploding when least expected
No standards	Engineer A uses `df -h`, B uses `df -hT`, C wrote a Python script only they understand	Changing personnel means starting from scratch; results cannot be compared across machines
No history	Inspection results live in personal notes, taken when someone leaves	Cannot trace trends — “it was fine last week, how did it suddenly fill up?”
Delayed alerts	Inspection finds disk at 95%, but nobody sees it in time	Hours or days between discovery and action
Wasted manpower	3 people spend 2-3 hours daily on inspection, ~200 person-hours per month	Equivalent to 1.25 FTE doing nothing but manual checks

Core Capabilities of a Platform

A qualified inspection platform needs these capabilities:

┌─────────────────────────────────────────────────────┐
│              Inspection Platform Core Capabilities    │
├──────────┬──────────┬──────────┬──────────┬─────────┤
│  Check    │ Scheduling│  Alert   │  Report  │  Asset  │
│  Engine   │  System   │  Engine  │ Generator │ Management│
│          │          │          │          │         │
│ Plugin   │ Cron      │ Multi-   │ HTML     │ Host    │
│ based    │ trigger   │ level    │ reports  │ inventory│
│ Concurrent│ Manual   │ thresholds│ Trend   │ Group   │
│ execution│ trigger   │ Dedup    │ charts   │ management│
│ Timeout  │ Catch-up  │ Channels │ Diff     │ Tag     │
│ control  │ execution │          │ comparison│ system  │
└──────────┴──────────┴──────────┴──────────┴─────────┘

In short: configure check items and thresholds, the platform runs automatically at scheduled times, alerts immediately on issues, and archives every result for traceability.

Architecture Design

Overall Architecture

┌──────────────────────────────────────────────────────┐
│                    API / Web UI                       │
│         (Config management, report viewing, manual)   │
├──────────────────────────────────────────────────────┤
│                                                       │
│  ┌─────────────┐  ┌──────────────┐  ┌─────────────┐ │
│  │  Scheduler   │  │  Executor    │  │  Alert      │ │
│  │             │  │              │  │  Engine     │ │
│  │ Cron trigger│──>│ Worker Pool  │──>│ Threshold  │ │
│  │ Manual      │  │ Concurrent   │  │ judgment   │ │
│  │ Catch-up    │  │ Timeout ctrl │  │ Dedup      │ │
│  └─────────────┘  └──────────────┘  │ Multi-channel│ │
│         │                │          └─────────────┘ │
│         │         ┌──────┴──────┐        │          │
│         │         │  Plugin     │        │          │
│         │         │  Registry   │        │          │
│         │         │             │        │          │
│         │         │ disk_check  │        │          │
│         │         │ cpu_check   │        │          │
│         │         │ cert_check  │        │          │
│         │         │ conn_check  │        │          │
│         │         │ ...         │        │          │
│         │         └─────────────┘        │          │
│         │                │                │          │
│  ┌──────┴────────────────┴────────────────┴──────┐  │
│  │                  Storage Layer                 │  │
│  │  SQLite/PostgreSQL (results, config, assets)   │  │
│  └───────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────┘

Layer Responsibilities

Layer	Responsibility	Tech Choice
Access	Config management, report display, manual triggers	HTTP API + simple Web UI
Scheduling	Timed inspection triggers, task queue management	Go cron library + channel
Execution	Concurrent plugin execution, timeout control	goroutine + Worker Pool
Plugin	Specific check logic (disk, CPU, cert, etc.)	Go interface plugin system
Alert	Threshold evaluation, alert dedup, notification	webhook + email + IM
Storage	Result, config, asset persistence	SQLite (single) or PostgreSQL (cluster)

Why Plugin-Based

Check items change constantly. Today you check disk, tomorrow you add certificate expiry, the day after you need database connection count. If check logic is hardcoded in the main program, every new check item requires modifying the main program, recompiling, and redeploying.

The plugin approach: adding a check item means writing a plugin file, registering it with the plugin manager, and restarting the platform. No changes to existing logic or the main program.

Go plugin implementation options:

Approach	Pros	Cons	Use Case
Go plugin package	Native, high performance	Linux/macOS only, build version must match	Pure Go, controlled platform
interface + registry	Simple, reliable, cross-platform	Requires recompiling main program	Small-to-mid scale, infrequent changes
gRPC plugins	Language-agnostic, process isolation	Complex, RPC overhead	Multi-language, strong isolation
External script execution	Most flexible, any language	Poor performance, external deps	Rapid prototyping, legacy script reuse

Considering cross-platform needs and deployment simplicity, I chose the interface + registry approach. Although adding plugins requires recompilation, Go compiles fast, and a config file controls which plugins are enabled — flexible enough.

Core Code Implementation

Project Structure

sre-inspector/
├── cmd/
│   └── inspector/
│       └── main.go           # Entry point
├── internal/
│   ├── config/
│   │   └── config.go         # Config loading
│   ├── scheduler/
│   │   └── scheduler.go      # Scheduling engine
│   ├── executor/
│   │   └── executor.go       # Execution engine
│   ├── checker/
│   │   ├── checker.go        # Checker interface definition
│   │   ├── registry.go       # Plugin registry
│   │   ├── disk.go           # Disk check plugin
│   │   ├── cpu.go            # CPU check plugin
│   │   ├── memory.go         # Memory check plugin
│   │   ├── certificate.go    # Certificate expiry check plugin
│   │   ├── connection.go     # Connection count check plugin
│   │   └── process.go        # Process check plugin
│   ├── alert/
│   │   └── alert.go          # Alert engine
│   ├── report/
│   │   └── report.go         # Report generation
│   └── storage/
│       └── storage.go        # Data storage
├── configs/
│   └── inspector.yaml        # Configuration
├── go.mod
└── go.sum

Checker Interface Definition

The core of plugin architecture is a well-defined interface. Every check item implements it:

// internal/checker/checker.go

package checker

import (
    "context"
    "time"
)

// CheckResult result of a single check
type CheckResult struct {
    CheckerName  string             // Check item name
    Target       string             // Check target (IP, hostname, etc.)
    Status       Status             // Status: OK/Warning/Critical
    Message      string             // Result description
    Metrics      map[string]float64 // Metric data (disk usage, CPU load, etc.)
    CheckedAt    time.Time          // Check time
    Duration     time.Duration      // Check duration
}

// Status check status
type Status int

const (
    StatusOK      Status = 0 // Normal
    StatusWarning Status = 1 // Warning
    StatusCritical Status = 2 // Critical
    StatusUnknown Status = 3 // Unknown (check failed)
)

func (s Status) String() string {
    switch s {
    case StatusOK:
        return "OK"
    case StatusWarning:
        return "WARNING"
    case StatusCritical:
        return "CRITICAL"
    default:
        return "UNKNOWN"
    }
}

// Checker interface that all plugins must implement
type Checker interface {
    // Name returns the checker name
    Name() string

    // Description returns the checker description
    Description() string

    // Check executes the check
    // ctx for timeout control and cancellation
    // target is the check target (host IP, hostname, etc.)
    // params are check parameters (thresholds, etc.)
    Check(ctx context.Context, target string, params map[string]string) CheckResult
}

Plugin Registry

// internal/checker/registry.go

package checker

import (
    "fmt"
    "sync"
)

// Registry plugin registry
type Registry struct {
    mu       sync.RWMutex
    checkers map[string]Checker
}

// NewRegistry creates a registry instance
func NewRegistry() *Registry {
    return &Registry{
        checkers: make(map[string]Checker),
    }
}

// Register registers a checker
func (r *Registry) Register(c Checker) error {
    r.mu.Lock()
    defer r.mu.Unlock()

    name := c.Name()
    if _, exists := r.checkers[name]; exists {
        return fmt.Errorf("checker %q already registered", name)
    }
    r.checkers[name] = c
    return nil
}

// Get retrieves a checker by name
func (r *Registry) Get(name string) (Checker, bool) {
    r.mu.RLock()
    defer r.mu.RUnlock()
    c, ok := r.checkers[name]
    return c, ok
}

// List returns all registered checker names
func (r *Registry) List() []string {
    r.mu.RLock()
    defer r.mu.RUnlock()
    names := make([]string, 0, len(r.checkers))
    for name := range r.checkers {
        names = append(names, name)
    }
    return names
}

// RegisterBuiltin registers built-in checkers
func (r *Registry) RegisterBuiltin() {
    r.Register(&DiskChecker{})
    r.Register(&CPUChecker{})
    r.Register(&MemoryChecker{})
    r.Register(&CertificateChecker{})
    r.Register(&ConnectionChecker{})
    r.Register(&ProcessChecker{})
}

Disk Check Plugin Implementation

Using disk check as an example, here is a complete plugin:

// internal/checker/disk.go

package checker

import (
    "context"
    "fmt"
    "os/exec"
    "strconv"
    "strings"
    "time"
)

// DiskChecker disk usage checker
type DiskChecker struct{}

func (d *DiskChecker) Name() string        { return "disk" }
func (d *DiskChecker) Description() string  { return "Check disk partition usage" }

func (d *DiskChecker) Check(ctx context.Context, target string, params map[string]string) CheckResult {
    start := time.Now()

    // Parse threshold params, default warning 80% critical 95%
    warnThreshold := parseFloatParam(params, "warning", 80.0)
    critThreshold := parseFloatParam(params, "critical", 95.0)

    // Execute remote command via SSH (if target is not localhost)
    cmd := exec.CommandContext(ctx, "ssh", "-o", "ConnectTimeout=5", target, "df -h --output=pcent,target -x tmpfs -x devtmpfs")
    output, err := cmd.CombinedOutput()
    if err != nil {
        return CheckResult{
            CheckerName: d.Name(),
            Target:      target,
            Status:      StatusUnknown,
            Message:      fmt.Sprintf("SSH execution failed: %v, output: %s", err, string(output)),
            CheckedAt:    start,
            Duration:     time.Since(start),
        }
    }

    // Parse df output, find the partition with highest usage
    var maxUsage float64
    var maxPartitions []string
    lines := strings.Split(strings.TrimSpace(string(output)), "\n")
    for i, line := range lines {
        if i == 0 {
            continue // skip header
        }
        fields := strings.Fields(line)
        if len(fields) < 2 {
            continue
        }
        usageStr := strings.TrimSuffix(fields[0], "%")
        usage, err := strconv.ParseFloat(usageStr, 64)
        if err != nil {
            continue
        }
        if usage > maxUsage {
            maxUsage = usage
            maxPartitions = append(maxPartitions[:0], fields[1])
        } else if usage == maxUsage {
            maxPartitions = append(maxPartitions, fields[1])
        }
    }

    // Determine status based on thresholds
    status := StatusOK
    message := fmt.Sprintf("Max disk usage: %.1f%% (%s)", maxUsage, strings.Join(maxPartitions, ", "))
    if maxUsage >= critThreshold {
        status = StatusCritical
        message = fmt.Sprintf("Disk usage critical: %.1f%% (%s) >= %.0f%%", maxUsage, strings.Join(maxPartitions, ", "), critThreshold)
    } else if maxUsage >= warnThreshold {
        status = StatusWarning
        message = fmt.Sprintf("Disk usage warning: %.1f%% (%s) >= %.0f%%", maxUsage, strings.Join(maxPartitions, ", "), warnThreshold)
    }

    return CheckResult{
        CheckerName: d.Name(),
        Target:      target,
        Status:      status,
        Message:      message,
        Metrics:      map[string]float64{"disk_usage_percent": maxUsage},
        CheckedAt:    start,
        Duration:     time.Since(start),
    }
}

func parseFloatParam(params map[string]string, key string, defaultVal float64) float64 {
    if val, ok := params[key]; ok {
        if f, err := strconv.ParseFloat(val, 64); err == nil {
            return f
        }
    }
    return defaultVal
}

Execution Engine: Concurrency Control

Inspecting 200 machines serially takes half an hour. Concurrent execution is essential, but cannot be unlimited — too many SSH connections will overwhelm the target machines’ sshd and exhaust local file descriptors.

// internal/executor/executor.go

package executor

import (
    "context"
    "fmt"
    "log"
    "sync"
    "time"

    "sre-inspector/internal/checker"
)

// Executor execution engine
type Executor struct {
    registry    *checker.Registry
    maxWorkers  int           // max concurrency
    timeout     time.Duration // per-check timeout
}

// NewExecutor creates an execution engine
func NewExecutor(registry *checker.Registry, maxWorkers int, timeout time.Duration) *Executor {
    return &Executor{
        registry:   registry,
        maxWorkers: maxWorkers,
        timeout:    timeout,
    }
}

// Task a single inspection task
type Task struct {
    CheckerName string            // checker name
    Target      string            // target host
    Params      map[string]string // check parameters
}

// RunBatch executes inspection tasks in batch
func (e *Executor) RunBatch(tasks []Task) []checker.CheckResult {
    // Buffered channel as task queue
    taskCh := make(chan Task, len(tasks))
    resultCh := make(chan checker.CheckResult, len(tasks))

    // Fill the task queue
    for _, task := range tasks {
        taskCh <- task
    }
    close(taskCh)

    // Start worker pool
    var wg sync.WaitGroup
    for i := 0; i < e.maxWorkers; i++ {
        wg.Add(1)
        go e.worker(&wg, taskCh, resultCh)
    }

    // Wait for all workers to finish
    go func() {
        wg.Wait()
        close(resultCh)
    }()

    // Collect results
    results := make([]checker.CheckResult, 0, len(tasks))
    for result := range resultCh {
        results = append(results, result)
    }
    return results
}

// worker goroutine
func (e *Executor) worker(wg *sync.WaitGroup, taskCh <-chan Task, resultCh chan<- checker.CheckResult) {
    defer wg.Done()
    for task := range taskCh {
        result := e.runSingle(task)
        resultCh <- result
    }
}

// runSingle executes a single check task
func (e *Executor) runSingle(task Task) checker.CheckResult {
    c, ok := e.registry.Get(task.CheckerName)
    if !ok {
        return checker.CheckResult{
            CheckerName: task.CheckerName,
            Target:      task.Target,
            Status:      checker.StatusUnknown,
            Message:      fmt.Sprintf("checker %q not registered", task.CheckerName),
        }
    }

    // Context with timeout
    ctx, cancel := context.WithTimeout(context.Background(), e.timeout)
    defer cancel()

    // Execute check
    result := c.Check(ctx, task.Target, task.Params)

    // Log slow checks
    if result.Duration > 5*time.Second {
        log.Printf("[WARN] slow check: checker=%s target=%s duration=%s",
            task.CheckerName, task.Target, result.Duration)
    }

    return result
}

Scheduling Engine

// internal/scheduler/scheduler.go

package scheduler

import (
    "context"
    "fmt"
    "log"
    "sync"
    "time"

    "sre-inspector/internal/checker"
    "sre-inspector/internal/executor"
)

// ScheduledJob scheduled inspection task
type ScheduledJob struct {
    Name         string                       // task name
    Cron         string                       // cron expression
    Checkers     []string                     // checker list to run
    Targets      []string                     // target host list
    Params       map[string]map[string]string // params per checker
}

// Scheduler scheduling engine
type Scheduler struct {
    jobs     []*ScheduledJob
    executor *executor.Executor
    alertCh  chan checker.CheckResult // alert channel
    mu       sync.Mutex
    running  bool
}

// NewScheduler creates a scheduler
func NewScheduler(exec *executor.Executor, alertCh chan checker.CheckResult) *Scheduler {
    return &Scheduler{
        executor: exec,
        alertCh:  alertCh,
    }
}

// AddJob adds a scheduled task
func (s *Scheduler) AddJob(job *ScheduledJob) {
    s.mu.Lock()
    defer s.mu.Unlock()
    s.jobs = append(s.jobs, job)
}

// Start starts the scheduler
func (s *Scheduler) Start(ctx context.Context) {
    s.mu.Lock()
    s.running = true
    s.mu.Unlock()

    for _, job := range s.jobs {
        go s.runJob(ctx, job)
    }
    log.Printf("Scheduler started, %d scheduled tasks", len(s.jobs))
}

// runJob runs a single scheduled task
func (s *Scheduler) runJob(ctx context.Context, job *ScheduledJob) {
    interval, err := parseCronInterval(job.Cron)
    if err != nil {
        log.Printf("[ERROR] failed to parse cron: job=%s cron=%s err=%v", job.Name, job.Cron, err)
        return
    }

    ticker := time.NewTicker(interval)
    defer ticker.Stop()

    log.Printf("[INFO] job %s registered, interval: %v", job.Name, interval)

    for {
        select {
        case <-ctx.Done():
            log.Printf("[INFO] job %s stopped", job.Name)
            return
        case <-ticker.C:
            log.Printf("[INFO] starting inspection: %s", job.Name)
            s.executeJob(job)
        }
    }
}

// executeJob executes one full inspection run
func (s *Scheduler) executeJob(job *ScheduledJob) {
    start := time.Now()

    // Build task list: checker x target cartesian product
    var tasks []executor.Task
    for _, checkerName := range job.Checkers {
        params := job.Params[checkerName]
        for _, target := range job.Targets {
            tasks = append(tasks, executor.Task{
                CheckerName: checkerName,
                Target:      target,
                Params:      params,
            })
        }
    }

    log.Printf("[INFO] job %s: %d check items total", job.Name, len(tasks))

    // Batch execute
    results := s.executor.RunBatch(tasks)

    // Summarize
    var okCount, warnCount, critCount, unknownCount int
    for _, r := range results {
        switch r.Status {
        case checker.StatusOK:
            okCount++
        case checker.StatusWarning:
            warnCount++
            s.alertCh <- r
        case checker.StatusCritical:
            critCount++
            s.alertCh <- r
        default:
            unknownCount++
            s.alertCh <- r
        }
    }

    log.Printf("[INFO] job %s done: duration=%s ok=%d warning=%d critical=%d unknown=%d",
        job.Name, time.Since(start), okCount, warnCount, critCount, unknownCount)
}

// parseCronInterval simplified cron parser
func parseCronInterval(cronExpr string) (time.Duration, error) {
    var num int
    var unit string
    _, err := fmt.Sscanf(cronExpr, "every %d%c", &num, &unit)
    if err != nil {
        return time.Hour, nil
    }
    switch unit {
    case 'm':
        return time.Duration(num) * time.Minute, nil
    case 'h':
        return time.Duration(num) * time.Hour, nil
    default:
        return time.Hour, fmt.Errorf("unsupported time unit: %c", unit)
    }
}

Alert Engine

// internal/alert/alert.go

package alert

import (
    "fmt"
    "log"
    "strings"
    "sync"
    "time"

    "sre-inspector/internal/checker"
)

// AlertConfig alert configuration
type AlertConfig struct {
    WebhookURL     string        // webhook notification URL
    EmailTo        []string      // email recipients
    DingtalkToken  string        // DingTalk bot token
    RepeatInterval time.Duration // dedup interval
}

// AlertEngine alert engine
type AlertEngine struct {
    config       AlertConfig
    recentAlerts sync.Map // dedup cache: key="checker:target" -> lastAlertTime
}

// NewAlertEngine creates an alert engine
func NewAlertEngine(config AlertConfig) *AlertEngine {
    return &AlertEngine{config: config}
}

// Process processes a check result for alerting
func (a *AlertEngine) Process(result checker.CheckResult) {
    if result.Status == checker.StatusOK {
        return
    }

    key := fmt.Sprintf("%s:%s", result.CheckerName, result.Target)

    // Dedup: same check+target does not alert again within RepeatInterval
    if lastTime, ok := a.recentAlerts.Load(key); ok {
        if time.Since(lastTime.(time.Time)) < a.config.RepeatInterval {
            return
        }
    }
    a.recentAlerts.Store(key, time.Now())

    title, message := a.buildMessage(result)
    log.Printf("[ALERT] %s: %s", title, message)

    if a.config.WebhookURL != "" {
        a.sendWebhook(title, message, result)
    }
    if a.config.DingtalkToken != "" {
        a.sendDingtalk(title, message)
    }
}

// buildMessage builds alert message
func (a *AlertEngine) buildMessage(result checker.CheckResult) (string, string) {
    severity := "WARNING"
    if result.Status == checker.StatusCritical {
        severity = "CRITICAL"
    } else if result.Status == checker.StatusUnknown {
        severity = "UNKNOWN"
    }

    title := fmt.Sprintf("[%s] Inspection Alert: %s @ %s", severity, result.CheckerName, result.Target)

    var sb strings.Builder
    sb.WriteString(fmt.Sprintf("**Check**: %s\n", result.CheckerName))
    sb.WriteString(fmt.Sprintf("**Target**: %s\n", result.Target))
    sb.WriteString(fmt.Sprintf("**Status**: %s\n", severity))
    sb.WriteString(fmt.Sprintf("**Details**: %s\n", result.Message))
    sb.WriteString(fmt.Sprintf("**Time**: %s\n", result.CheckedAt.Format("2006-01-02 15:04:05")))
    sb.WriteString(fmt.Sprintf("**Duration**: %s\n", result.Duration))

    if len(result.Metrics) > 0 {
        sb.WriteString("**Metrics**:\n")
        for k, v := range result.Metrics {
            sb.WriteString(fmt.Sprintf("  - %s: %.2f\n", k, v))
        }
    }

    return title, sb.String()
}

func (a *AlertEngine) sendWebhook(title, message string, result checker.CheckResult) {
    log.Printf("[WEBHOOK] sending alert to %s: %s", a.config.WebhookURL, title)
}

func (a *AlertEngine) sendDingtalk(title, message string) {
    log.Printf("[DINGTALK] sending alert: %s", title)
}

// Start starts the alert engine, listening on the result channel
func (a *AlertEngine) Start(alertCh <-chan checker.CheckResult) {
    for result := range alertCh {
        a.Process(result)
    }
}

Configuration File

# configs/inspector.yaml

# Execution engine config
executor:
  max_workers: 20          # max concurrency
  timeout: 30s             # per-check timeout

# Alert config
alert:
  webhook_url: "https://hooks.example.com/inspector"
  dingtalk_token: ""
  repeat_interval: 30m     # same alert not repeated within 30 min

# Storage config
storage:
  type: sqlite             # sqlite or postgres
  dsn: "/var/lib/inspector/inspector.db"

# Asset inventory
targets:
  - host: "10.0.0.5"
    name: "web-01"
    tags: ["web", "production"]
  - host: "10.0.0.6"
    name: "web-02"
    tags: ["web", "production"]
  - host: "10.0.0.10"
    name: "db-01"
    tags: ["database", "production"]

# Checker config
checkers:
  - name: disk
    enabled: true
    params:
      warning: "80"
      critical: "95"
  
  - name: cpu
    enabled: true
    params:
      warning: "70"
      critical: "90"
      interval: "5m"
  
  - name: memory
    enabled: true
    params:
      warning: "80"
      critical: "95"
  
  - name: certificate
    enabled: true
    params:
      warning: "30"
      critical: "7"
      domains: "api.example.com,admin.example.com"
  
  - name: connection
    enabled: true
    params:
      warning: "5000"
      critical: "10000"
  
  - name: process
    enabled: true
    params:
      processes: "nginx,mysql,redis"

# Scheduled jobs
jobs:
  - name: "Full inspection"
    schedule: "every 6h"
    checkers: ["disk", "cpu", "memory", "certificate", "connection", "process"]
    target_tags: ["production"]
  
  - name: "Database inspection"
    schedule: "every 1h"
    checkers: ["disk", "memory", "process"]
    target_tags: ["database"]
  
  - name: "Certificate daily check"
    schedule: "every 24h"
    checkers: ["certificate"]
    target_tags: ["production"]

Main Entry Point

// cmd/inspector/main.go

package main

import (
    "context"
    "flag"
    "log"
    "os"
    "os/signal"
    "syscall"
    "time"

    "sre-inspector/internal/alert"
    "sre-inspector/internal/checker"
    "sre-inspector/internal/config"
    "sre-inspector/internal/executor"
    "sre-inspector/internal/report"
    "sre-inspector/internal/scheduler"
    "sre-inspector/internal/storage"
)

func main() {
    configPath := flag.String("config", "configs/inspector.yaml", "config file path")
    once := flag.Bool("once", false, "run inspection once and exit")
    flag.Parse()

    // Load config
    cfg, err := config.Load(*configPath)
    if err != nil {
        log.Fatalf("Failed to load config: %v", err)
    }

    // Initialize plugin registry
    registry := checker.NewRegistry()
    registry.RegisterBuiltin()
    log.Printf("Registered checkers: %v", registry.List())

    // Create alert channel
    alertCh := make(chan checker.CheckResult, 1000)

    // Initialize execution engine
    exec := executor.NewExecutor(
        registry,
        cfg.Executor.MaxWorkers,
        cfg.Executor.Timeout,
    )

    // Initialize alert engine
    alertEngine := alert.NewAlertEngine(cfg.Alert)
    go alertEngine.Start(alertCh)

    // Initialize storage
    store, err := storage.New(cfg.Storage)
    if err != nil {
        log.Fatalf("Failed to init storage: %v", err)
    }
    defer store.Close()

    // Initialize report generator
    reportGen := report.NewGenerator(store)

    // One-shot mode
    if *once {
        runOnce(cfg, exec, alertCh, store, reportGen)
        return
    }

    // Initialize scheduler
    sched := scheduler.NewScheduler(exec, alertCh)
    for _, jobCfg := range cfg.Jobs {
        job := buildJobFromConfig(jobCfg, cfg)
        sched.AddJob(job)
    }

    // Start scheduler
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()
    sched.Start(ctx)

    log.Println("Inspection platform started, press Ctrl+C to exit")

    // Wait for exit signal
    sigCh := make(chan os.Signal, 1)
    signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
    <-sigCh

    log.Println("Shutting down...")
    cancel()
    time.Sleep(2 * time.Second)
    log.Println("Exited")
}

// runOnce runs one inspection cycle
func runOnce(cfg *config.Config, exec *executor.Executor, alertCh chan checker.CheckResult, store *storage.Storage, reportGen *report.Generator) {
    log.Println("Starting one-shot inspection...")
    start := time.Now()

    var tasks []executor.Task
    for _, checkerCfg := range cfg.Checkers {
        if !checkerCfg.Enabled {
            continue
        }
        for _, target := range cfg.Targets {
            tasks = append(tasks, executor.Task{
                CheckerName: checkerCfg.Name,
                Target:      target.Host,
                Params:      checkerCfg.Params,
            })
        }
    }

    results := exec.RunBatch(tasks)
    store.SaveResults(results)

    for _, r := range results {
        if r.Status != checker.StatusOK {
            alertCh <- r
        }
    }

    reportPath := reportGen.GenerateHTML(results)
    log.Printf("Inspection complete: duration=%s report=%s", time.Since(start), reportPath)

    var ok, warn, crit, unk int
    for _, r := range results {
        switch r.Status {
        case checker.StatusOK:
            ok++
        case checker.StatusWarning:
            warn++
        case checker.StatusCritical:
            crit++
        default:
            unk++
        }
    }
    log.Printf("Summary: ok=%d warning=%d critical=%d unknown=%d", ok, warn, crit, unk)
}

Inspection Report Generation

The final output of inspection is a report. Reports are for humans, not machines — they need to be intuitive, readable, and quick to locate problems.

HTML Report Structure

// internal/report/report.go

package report

import (
    "fmt"
    "html/template"
    "os"
    "path/filepath"
    "time"

    "sre-inspector/internal/checker"
)

type Generator struct {
    tmpl *template.Template
}

func NewGenerator(store *storage.Storage) *Generator {
    return &Generator{}
}

// ReportData report data structure
type ReportData struct {
    GeneratedAt   time.Time
    TotalCount    int
    OKCount       int
    WarningCount  int
    CriticalCount int
    UnknownCount  int
    Results       []checker.CheckResult
    Summary       map[string]map[string]int
}

// GenerateHTML generates an HTML report
func (g *Generator) GenerateHTML(results []checker.CheckResult) string {
    data := g.prepareData(results)

    tmpl := template.Must(template.New("report").Parse(reportTemplate))

    filename := fmt.Sprintf("inspection_%s.html", time.Now().Format("20060102_150405"))
    filepath := filepath.Join("/var/lib/inspector/reports", filename)

    os.MkdirAll(filepath, 0755)
    f, err := os.Create(filepath)
    if err != nil {
        return ""
    }
    defer f.Close()

    tmpl.Execute(f, data)
    return filepath
}

func (g *Generator) prepareData(results []checker.CheckResult) ReportData {
    data := ReportData{
        GeneratedAt: time.Now(),
        TotalCount:  len(results),
        Results:     results,
        Summary:     make(map[string]map[string]int),
    }

    for _, r := range results {
        if data.Summary[r.Target] == nil {
            data.Summary[r.Target] = make(map[string]int)
        }
        data.Summary[r.Target][r.Status.String()]++
        switch r.Status {
        case checker.StatusOK:
            data.OKCount++
        case checker.StatusWarning:
            data.WarningCount++
        case checker.StatusCritical:
            data.CriticalCount++
        default:
            data.UnknownCount++
        }
    }
    return data
}

const reportTemplate = `
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Inspection Report - {{.GeneratedAt.Format "2006-01-02 15:04"}}</title>
    <style>
        body { font-family: -apple-system, sans-serif; margin: 40px; }
        .summary { display: flex; gap: 20px; margin-bottom: 30px; }
        .card { padding: 20px; border-radius: 8px; color: white; }
        .ok { background: #4caf50; }
        .warning { background: #ff9800; }
        .critical { background: #f44336; }
        .unknown { background: #9e9e9e; }
        table { border-collapse: collapse; width: 100%; }
        th, td { border: 1px solid #ddd; padding: 8px; text-align: left; }
        th { background: #f5f5f5; }
        .status-ok { color: #4caf50; }
        .status-warning { color: #ff9800; }
        .status-critical { color: #f44336; }
        .status-unknown { color: #9e9e9e; }
    </style>
</head>
<body>
    <h1>Inspection Report</h1>
    <p>Generated: {{.GeneratedAt.Format "2006-01-02 15:04:05"}}</p>

    <div class="summary">
        <div class="card ok"><h2>{{.OKCount}}</h2><p>OK</p></div>
        <div class="card warning"><h2>{{.WarningCount}}</h2><p>Warning</p></div>
        <div class="card critical"><h2>{{.CriticalCount}}</h2><p>Critical</p></div>
        <div class="card unknown"><h2>{{.UnknownCount}}</h2><p>Unknown</p></div>
    </div>

    <h2>Detailed Results</h2>
    <table>
        <tr>
            <th>Checker</th>
            <th>Target</th>
            <th>Status</th>
            <th>Details</th>
            <th>Duration</th>
        </tr>
        {{range .Results}}
        <tr>
            <td>{{.CheckerName}}</td>
            <td>{{.Target}}</td>
            <td class="status-{{.Status.String | toLower}}">{{.Status}}</td>
            <td>{{.Message}}</td>
            <td>{{.Duration}}</td>
        </tr>
        {{end}}
    </table>
</body>
</html>
`

Production Deployment

Binary Deployment

# Cross-compile
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o bin/inspector ./cmd/inspector

# Deploy to server
scp bin/inspector configs/inspector.yaml user@server:/opt/inspector/

# Create systemd service
cat > /etc/systemd/system/inspector.service << 'EOF'
[Unit]
Description=SRE Inspection Platform
After=network.target

[Service]
Type=simple
User=inspector
WorkingDirectory=/opt/inspector
ExecStart=/opt/inspector/inspector -config /opt/inspector/configs/inspector.yaml
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable inspector
systemctl start inspector

SSH Keyless Configuration

Inspection requires SSH access to target machines. Use key authentication, not passwords:

# Generate a dedicated key on the inspection server
ssh-keygen -t ed25519 -f /home/inspector/.ssh/inspector_key -N ""

# Distribute public key to target machines
for host in 10.0.0.{5..50}; do
    ssh-copy-id -i /home/inspector/.ssh/inspector_key.pub inspector@$host
done

# Configure SSH aliases
cat >> /home/inspector/.ssh/config << 'EOF'
Host *
    IdentityFile ~/.ssh/inspector_key
    StrictHostKeyChecking no
    ConnectTimeout 5
    ServerAliveInterval 30
EOF

Security Considerations

Least privilege: The inspection user should have read-only permissions. Do not run as root. Create a dedicated inspector user with a sudo whitelist for specific read-only commands.

# /etc/sudoers.d/inspector
inspector ALL=(ALL) NOPASSWD: /usr/bin/df, /usr/bin/free, /usr/bin/uptime, /usr/bin/ss, /usr/bin/ps

SSH key protection: If the inspection server’s private key leaks, all target machines are compromised. Private key file permissions must be 600. Ideally, store keys in a hardware security module or key management service.
Inspection data sensitivity: Results contain host info, process lists, network connections — sensitive operational data. Store reports in controlled directories and clean up expired reports regularly.
Network isolation: The inspection server can access all target machines, making it a high-value attack target. Place it in a management network with restricted inbound access.

Common Check Item Implementation Approaches

Certificate Expiry Check

// Core logic: TLS connect to the target domain, read the certificate chain, check NotAfter
func checkCertificate(domain string, warnDays, critDays int) CheckResult {
    conn, err := tls.Dial("tcp", domain+":443", &tls.Config{
        InsecureSkipVerify: true, // We want to check expired certs, not refuse them
    })
    if err != nil {
        return CheckResult{Status: StatusUnknown, Message: fmt.Sprintf("TLS connection failed: %v", err)}
    }
    defer conn.Close()

    cert := conn.ConnectionState().PeerCertificates[0]
    daysLeft := int(time.Until(cert.NotAfter).Hours() / 24)

    status := StatusOK
    message := fmt.Sprintf("Certificate expires in %d days (expiry: %s)", daysLeft, cert.NotAfter.Format("2006-01-02"))

    if daysLeft <= critDays {
        status = StatusCritical
        message = fmt.Sprintf("Certificate expiring soon! Only %d days left (expiry: %s)", daysLeft, cert.NotAfter.Format("2006-01-02"))
    } else if daysLeft <= warnDays {
        status = StatusWarning
        message = fmt.Sprintf("Certificate expires in %d days (expiry: %s)", daysLeft, cert.NotAfter.Format("2006-01-02"))
    }

    return CheckResult{
        Status:  status,
        Message:  message,
        Metrics:  map[string]float64{"days_until_expiry": float64(daysLeft)},
    }
}

Critical Process Check

// Check if required processes are running via SSH pgrep
func checkProcess(ctx context.Context, target string, processNames []string) CheckResult {
    var missing []string
    for _, name := range processNames {
        cmd := exec.CommandContext(ctx, "ssh", target, fmt.Sprintf("pgrep -x %s", name))
        if err := cmd.Run(); err != nil {
            missing = append(missing, name)
        }
    }

    if len(missing) > 0 {
        return CheckResult{
            Status:  StatusCritical,
            Message:  fmt.Sprintf("Processes not running: %s", strings.Join(missing, ", ")),
        }
    }
    return CheckResult{Status: StatusOK, Message: "All critical processes running"}
}

TCP Connection Count Check

// Count current ESTABLISHED connections via SSH ss
func checkConnectionCount(ctx context.Context, target string, warn, crit int) CheckResult {
    cmd := exec.CommandContext(ctx, "ssh", target, "ss -tn state established | wc -l")
    output, err := cmd.Output()
    if err != nil {
        return CheckResult{Status: StatusUnknown, Message: fmt.Sprintf("Execution failed: %v", err)}
    }

    count := 0
    fmt.Sscanf(strings.TrimSpace(string(output)), "%d", &count)

    status := StatusOK
    message := fmt.Sprintf("Current ESTABLISHED connections: %d", count)
    if count >= crit {
        status = StatusCritical
        message = fmt.Sprintf("Connection count critical: %d >= %d", count, crit)
    } else if count >= warn {
        status = StatusWarning
        message = fmt.Sprintf("Connection count warning: %d >= %d", count, warn)
    }

    return CheckResult{
        Status:  status,
        Message:  message,
        Metrics:  map[string]float64{"established_connections": float64(count)},
    }
}

Relationship with Existing Monitoring Systems

The inspection platform and monitoring systems like Prometheus are not competing — they serve different purposes:

Dimension	Inspection Platform	Prometheus
Check frequency	Low (hourly/daily)	High (every 15-60s)
Check scope	Comprehensive (disk+cert+process+config)	Metrics-focused (CPU/memory/traffic)
Alerting	Inspection report + instant alerts	Real-time alerts
Use case	Periodic health checks, compliance audits	Real-time monitoring, dynamic alerting
Data retention	Long-term (monthly/quarterly trends)	Mid-term (typically 15-90 days)

In short: Prometheus is an EKG monitor, running 24/7; the inspection platform is an annual physical exam. They complement each other and cannot replace each other.

The inspection platform can do things Prometheus is not good at:

Certificate expiry checks (requires active TLS connection)
Config file compliance checks (requires reading file contents)
Cross-machine comparison (distribution of the same metric across all hosts)
Offline report generation (no long-running monitoring service needed)

Future Directions

The architecture supports several extension paths:

Agent mode: Currently SSH-based, suitable for small-to-mid scale. Beyond 500 machines, SSH overhead becomes significant. A lightweight agent can be deployed to target machines, reporting results via gRPC.
Web management UI: Currently configured via YAML. A Web UI can enable visual check item configuration, historical trend charts, and one-click report generation.
Auto-remediation: Beyond alerting, trigger auto-fix scripts when issues are found — clean logs when disk is full, restart crashed processes. But auto-remediation must be cautious — wrong remediation actions are more dangerous than the original problem.
Trend analysis: With historical data in the database, trend analysis becomes possible. “What is the disk usage growth rate over the past 30 days? At current rate, how many days until 95%?” This kind of predictive analysis is valuable for capacity planning.
Compliance checks: Add compliance rules to check items — password policy compliance, SSH root login disabled, firewall rules correct. Integrate security compliance scanning into daily inspection.

Summary

The core value of an automated inspection platform: turning repetitive manual inspection into automated system engineering, freeing ops engineers from “checking every day” to focus on things that truly need human judgment.

Key architecture decisions:

Plugin architecture: Checker interface + Registry, adding check items without modifying the main program. Good extensibility.
Worker Pool concurrency: goroutine + channel to control concurrency. 200 machines with 20 concurrent workers finish in 3 minutes — 10x faster than serial.
Alert deduplication: Same issue does not alert again within the suppression window, preventing alert storms. In practice at 200-machine scale, a disk-full alert went from 50 notifications down to 1.
Config-driven: Check items, thresholds, target hosts, and schedules all in a YAML file. No recompilation needed for config changes.
Go single-binary deployment: A 15MB binary + a YAML config = complete deployment. No environment management headaches like Python.

Pitfalls encountered:

SSH connection timeout must be short (5 seconds). Otherwise, unreachable targets drag the entire inspection down. ConnectTimeout=5 with ServerAliveInterval=30 is a stable combination.
More goroutines is not better. 200 goroutines for 200 machines will momentarily exhaust target machines’ MaxStartups. 20 concurrent is the sweet spot for 200 machines.
Alert dedup window must be tuned per check type. 30 minutes is fine for disk issues, but certificate checks that run daily need a 24-hour window, or you get 24 identical alerts per day.
df output format differs across Linux distros. CentOS 7 and Ubuntu 22.04 have different column orders. Use df --output=pcent,target to force a consistent format and avoid parsing issues.

One final piece of experience: the inspection platform does not replace monitoring systems — it complements them. Prometheus handles real-time problem detection; the inspection platform handles periodic comprehensive health checks. Combined, ops can truly “know the state of things.”

References & Acknowledgments

The following resources were consulted during the writing of this article. Credit to the original authors:

From Zero to One: Modular Architecture Design for Python Network Automated Inspection — CSDN, modular inspection architecture design, connection management and data processing module implementation reference
AI Agent Automated Ops System: Evolution from Manual Inspection to Autonomous Operations — Tencent Cloud Developer Community, layered design and modular practices for automated ops systems
Go Concurrency Tour — Go Official Tour, goroutine and channel concurrency model fundamentals
robfig/cron: Go cron Library — GitHub, cron expression parsing and scheduled task scheduling for Go

Overview#

What Problem Does an Inspection Platform Solve#

Pain Points of Traditional Inspection#

Core Capabilities of a Platform#

Architecture Design#

Overall Architecture#

Layer Responsibilities#

Why Plugin-Based#

Core Code Implementation#

Project Structure#

Checker Interface Definition#

Plugin Registry#

Disk Check Plugin Implementation#

Execution Engine: Concurrency Control#

Scheduling Engine#

Alert Engine#

Configuration File#

Main Entry Point#

Inspection Report Generation#

HTML Report Structure#

Production Deployment#

Binary Deployment#

SSH Keyless Configuration#

Security Considerations#

Common Check Item Implementation Approaches#

Certificate Expiry Check#

Critical Process Check#

TCP Connection Count Check#

Relationship with Existing Monitoring Systems#

Future Directions#

Summary#

References & Acknowledgments#