Don't Let Jenkins Dictate Your Release Cadence: Architecture Decisions and Lessons from Building a Go DAG Scheduler Engine

Overview

2 AM. A ride-hailing platform’s release window just opened. Jenkins Master suddenly hits CPU 100%. The build queue backs up with 200+ tasks. The entire deployment pipeline is frozen. The ops team spent 40 minutes finding the root cause: a legacy project’s Git repo had 15GB of binary files stuffed in it. A single clone took 20 minutes, exhausting all of Master’s thread pool.

This isn’t an isolated case. Jenkins is a fine CI tool, but when you have 120+ microservices with tangled deployment dependencies and need fine-grained control over concurrency and rollback ordering, Jenkins’s Stage model starts to crack. Stage is linear. parallel enables same-level concurrency but can’t express complex DAG dependencies — “services A and B can deploy concurrently, but C must wait for both A and B to complete, while D only depends on B” — this kind of topology written in Jenkinsfile becomes a mess of nested parallel and stage blocks. A maintenance nightmare.

When building an ops platform for a new-energy logistics company, I built a DAG scheduler engine in Go that compressed 120+ microservices’ deployment time from 1.5 hours to 5 minutes. This article isn’t about reinventing the wheel (unless your deployment scenario truly demands it). It’s a complete record of architecture decisions and pitfalls, giving you a reference for tool selection and a guide for avoiding the traps I fell into.

Who this is for: Ops engineers with CI/CD experience who are considering building their own deployment platform or need deep customization of task orchestration. I’ll assume you’re familiar with Jenkins/GitLab CI basics and jump straight into production-grade architecture design.

Why Not Use Existing Tools

Bottom line: if your deployment scenario is “one repo, one pipeline,” Jenkins/GitLab CI/ArgoCD is perfectly adequate. Don’t build your own. But if you encounter the following scenarios, a custom DAG engine is worth considering.

Where Jenkins Hits Its Ceiling

Jenkins’s Stage model is fundamentally a linear pipeline. The parallel block enables same-level concurrency but can’t express cross-level dependencies. Consider this deployment topology:

build-A ──┬── deploy-A ──┬── smoke-test-A ──┬── rollout-A
          │              │                   │
build-B ──┤── deploy-B ──┤── smoke-test-B ──┤
          │              │                   │
build-C ─────────────────── deploy-C ────────┘

Service C’s deployment doesn’t depend on A/B’s deploy, but the final rollout needs all three services’ smoke tests to pass. Writing this in Jenkinsfile looks roughly like:

// Jenkinsfile — nested parallel nightmare
stage('Deploy') {
    parallel {
        stage('Deploy A') { steps { sh 'deploy A' } }
        stage('Deploy B') { steps { sh 'deploy B' } }
    }
}
// What about C? It doesn't depend on A/B's deploy, but can't go in the parallel above
// because parallel means "all complete before continuing"
stage('Deploy C') { steps { sh 'deploy C' } }
// The semantics are already wrong: C waits for A/B to finish deploying

Jenkins’s parallel semantics mean “all branches complete before entering the next stage.” It can’t express “C doesn’t wait for A/B, starts independently.” You’d need to put deploy-C in a separate parallel block, but that means A/B/C deploy completely independently — what if B’s deploy depends on some output from A? Jenkins’s model can’t express this.

ArgoCD’s Applicability Boundary

ArgoCD is an excellent GitOps tool, but its core is “declarative sync” — syncing Git repo state to a K8s cluster. It doesn’t care about “deploy A before B” ordering; it relies on K8s controllers to handle that. If your release needs strict ordering control (e.g., database migration must precede application deployment), you need Argo Rollouts or similar orchestration outside ArgoCD, and the complexity isn’t lower.

When to Build Your Own

My criteria are simple — if two of three conditions are met, it’s worth considering:

The logistics platform hit all three: 120+ microservices with layered deployment dependencies (infrastructure → middleware → business → gateway), DB migration must be serial but app layer can be 10-concurrent, and rollback needs to reverse from gateway layer. Managing this with Jenkins meant an 800-line Jenkinsfile where every release process change felt like walking on thin ice.

Core Architecture of the DAG Scheduler Engine

Overall Design

The engine is divided into four layers with clearly isolated responsibilities:

┌─────────────────────────────────────────┐
│         API Layer (Gin + gRPC)          │  ← Pipeline definition, triggering, status queries
├─────────────────────────────────────────┤
│      Scheduler Layer (DAG Engine)       │  ← Topological sort, layered scheduling, dependency resolution
├─────────────────────────────────────────┤
│      Executor Layer (Worker Pool)       │  ← Concurrent execution, timeout control, retry
├─────────────────────────────────────────┤
│     State Layer (Redis + MySQL)         │  ← Task state persistence, checkpoint recovery
└─────────────────────────────────────────┘

Why four layers instead of one big module? Because each layer’s concerns are completely different. Scheduler only cares “who should run now.” Executor only cares “how to run.” State only cares “where did it get to.” The biggest benefit of layering: when you find a scheduling bug, you don’t need to dig through 2000 lines of executor code — just look at the Scheduler’s 300 lines.

Core Data Model

// Pipeline definition — a complete deployment pipeline
type Pipeline struct {
    ID          string
    Name        string
    Tasks       []*Task
    Concurrency int           // Global concurrency limit
    Timeout     time.Duration // Whole pipeline timeout
    RetryPolicy RetryPolicy   // Global retry strategy
}

// Task definition — a single execution unit in the pipeline
type Task struct {
    ID          string
    Name        string
    Type        TaskType     // SHELL, HTTP, K8S_APPLY, GRPC_CALL
    DependsOn   []string     // List of dependency Task IDs
    Command     string       // Command to execute or endpoint to call
    Timeout     time.Duration
    RetryCount  int
    Conditions  []Condition  // Conditional execution (e.g., "only if previous task succeeded")
    OnFailure   FailureAction // ABORT, SKIP, RETRY, MANUAL
    Idempotent  bool         // Whether idempotent (used in checkpoint recovery)
}

// Task state machine
type TaskState string
const (
    StatePending  TaskState = "PENDING"   // Waiting for dependencies
    StateRunning  TaskState = "RUNNING"   // Currently executing
    StateSuccess  TaskState = "SUCCESS"   // Completed successfully
    StateFailed   TaskState = "FAILED"    // Execution failed
    StateSkipped  TaskState = "SKIPPED"   // Condition not met, skipped
    StateAborted  TaskState = "ABORTED"   // Manually terminated
)

Several key design decisions in this data model:

DependsOn uses []string instead of []*Task. Initially I used pointer references, which caused circular reference issues during JSON deserialization — encoding/json stack-overflowed. Switching to ID references made serialization/deserialization clean, with dependency resolution handled uniformly in the Scheduler layer.

OnFailure field was added later. The initial version only had “abort on failure” and “skip on failure” behaviors. After two months of usage, I discovered the need for “manual intervention on failure” — e.g., when database migration fails, you can’t auto-skip or auto-retry; someone needs to confirm.

Idempotent field designed for checkpoint recovery. K8s apply is naturally idempotent (declarative), but Shell commands and HTTP calls need explicit marking. Non-idempotent tasks aren’t auto-retried during checkpoint recovery — they’re suspended pending manual confirmation. More on this later.

Topological Sort: Engineering Implementation of Kahn’s Algorithm

Why Kahn Over DFS

Two mainstream topological sort algorithms exist: Kahn’s algorithm (BFS) and DFS reverse post-order. Core differences:

The core reason for choosing Kahn is layered parallelism. The essence of CI/CD scheduling is “tasks at the same layer can run in parallel; cross-layer tasks must wait for dependencies.” Kahn’s algorithm naturally forms a layer by extracting all in-degree-0 nodes each round, which can be thrown directly to the Worker Pool for concurrent execution.

DAG Construction and Cycle Detection

type DAG struct {
    nodes    map[string]*Task
    edges    map[string][]string   // Adjacency list: from -> [to...]
    inDegree map[string]int
    mu       sync.RWMutex
}

func NewDAG(pipeline *Pipeline) (*DAG, error) {
    dag := &DAG{
        nodes:    make(map[string]*Task),
        edges:    make(map[string][]string),
        inDegree: make(map[string]int),
    }
    // Register all nodes
    for _, task := range pipeline.Tasks {
        dag.nodes[task.ID] = task
        dag.inDegree[task.ID] = 0
    }
    // Build edges and in-degrees
    for _, task := range pipeline.Tasks {
        for _, dep := range task.DependsOn {
            if _, ok := dag.nodes[dep]; !ok {
                return nil, fmt.Errorf("task %s depends on unknown task %s", task.ID, dep)
            }
            dag.edges[dep] = append(dag.edges[dep], task.ID)
            dag.inDegree[task.ID]++
        }
    }
    // Cycle detection (must complete at load time)
    if cycle := dag.detectCycle(); cycle != nil {
        return nil, fmt.Errorf("cycle detected: %s", formatCycle(cycle))
    }
    return dag, nil
}

Cycle detection uses DFS three-color marking (white/gray/black). Encountering a gray node means a cycle. The full path is recorded for error reporting:

// detectCycle — returns cycle path instead of simple true/false
func (d *DAG) detectCycle() []string {
    color := make(map[string]string) // white/gray/black
    for id := range d.nodes { color[id] = "white" }
    var cyclePath []string
    var dfs func(nodeID string, path []string) bool
    dfs = func(nodeID string, path []string) bool {
        color[nodeID] = "gray"
        path = append(path, nodeID)
        for _, neighbor := range d.edges[nodeID] {
            if color[neighbor] == "gray" {
                cyclePath = append(path, neighbor) // Found cycle, record path
                return true
            }
            if color[neighbor] == "white" {
                if dfs(neighbor, path) { return true }
            }
        }
        color[nodeID] = "black"
        return false
    }
    for id := range d.nodes {
        if color[id] == "white" { if dfs(id, []string{}) { return cyclePath } }
    }
    return nil
}

Cycle Detection Pitfalls

Pitfall 1: Error only says “cycle exists” without specifying which edge. The initial version returned only true/false with error message cycle detected in pipeline. When the pipeline had 50+ tasks, developers had to manually trace each dependency edge to find the cycle. After changing to return the full path cycle detected: task-A -> task-B -> task-C -> task-A, localization time dropped from 30 minutes to 10 seconds.

Pitfall 2: Cross-layer dependency cycles. A -> B, B -> C, C -> A — this triangle cycle isn’t visible from adjacent edges. Checking only direct parent-child relationships misses it. Full DFS traversal is required. Three-color marking covers this 100%: when DFS reaches C from A and finds that C’s downstream A is gray (currently being visited), it immediately reports the cycle.

Pitfall 3: Concurrent addEdge race conditions. The DFS used in detectCycle has issues during concurrent addEdge — one goroutine traversing while another adds an edge may cause missed detection. Solution: complete all edge additions and cycle detection at NewDAG time; at runtime, read-only. If dynamic task addition is needed, you must acquire a write lock and re-detect — but I strongly recommend against supporting runtime dynamic edge addition. Fix the graph structure at load time.

Core Methods for Layered Scheduling

// GetReadyTasks — get currently executable tasks (in-degree 0 and PENDING state)
func (d *DAG) GetReadyTasks() []*Task {
    d.mu.RLock()
    defer d.mu.RUnlock()
    var ready []*Task
    for id, degree := range d.inDegree {
        if degree == 0 {
            if task := d.nodes[id]; task != nil {
                ready = append(ready, task)
            }
        }
    }
    return ready
}

// CompleteTask — mark task complete, decrement downstream in-degrees
func (d *DAG) CompleteTask(taskID string) {
    d.mu.Lock()
    defer d.mu.Unlock()
    for _, downstream := range d.edges[taskID] {
        d.inDegree[downstream]--
    }
}

Worker Pool Design

Why Not Bare Goroutines

Go’s goroutines are lightweight. Why not just go task.Execute()? Three reasons:

1. Uncontrolled concurrency: 50 ready tasks in one layer would spawn 50 goroutines simultaneously, potentially exhausting target servers’ connection pools
2. Difficult error handling: Panics inside goroutines aren’t caught by outer recover
3. Troublesome timeout control: Bare goroutines have no unified timeout management; stuck tasks hold resources forever

Worker Pool Core Implementation

type WorkerPool struct {
    maxWorkers int
    taskQueue  chan *ExecutableTask
    wg         sync.WaitGroup
    ctx        context.Context
    cancel     context.CancelFunc
}

type TaskResult struct {
    TaskID   string
    State    TaskState
    ExitCode int
    Output   string
    Error    error
    Duration time.Duration
    Retries  int
}

func NewWorkerPool(maxWorkers int, parentCtx context.Context) *WorkerPool {
    ctx, cancel := context.WithCancel(parentCtx)
    return &WorkerPool{
        maxWorkers: maxWorkers,
        taskQueue:  make(chan *ExecutableTask, maxWorkers*2),
        ctx:        ctx,
        cancel:     cancel,
    }
}

func (wp *WorkerPool) Start() {
    for i := 0; i < wp.maxWorkers; i++ {
        wp.wg.Add(1)
        go wp.worker()
    }
}

func (wp *WorkerPool) worker() {
    defer wp.wg.Done()
    for {
        select {
        case <-wp.ctx.Done():
            return
        case execTask := <-wp.taskQueue:
            // recover prevents single task panic from crashing entire pool
            result := wp.safeExecute(execTask)
            execTask.Result <- result
        }
    }
}

safeExecute with panic recovery and exponential backoff retry:

func (wp *WorkerPool) safeExecute(execTask *ExecutableTask) TaskResult {
    defer func() {
        if r := recover(); r != nil {
            log.Printf("[worker] task %s panicked: %v", execTask.Task.ID, r)
        }
    }()
    return wp.executeWithRetry(execTask)
}

func (wp *WorkerPool) executeWithRetry(execTask *ExecutableTask) TaskResult {
    var lastResult TaskResult
    for attempt := 0; attempt <= execTask.Task.RetryCount; attempt++ {
        ctx, cancel := context.WithTimeout(wp.ctx, execTask.Task.Timeout)
        result := wp.executeSingle(ctx, execTask.Task)
        result.Retries = attempt
        cancel()
        if result.State == StateSuccess { return result }
        lastResult = result
        if attempt < execTask.Task.RetryCount {
            backoff := execTask.Task.RetryDelay * time.Duration(1<<uint(attempt))
            if backoff > 60*time.Second { backoff = 60 * time.Second }
            time.Sleep(backoff)
        }
    }
    return lastResult
}

Empirical Concurrency Values

Worker Pool size isn’t “bigger is better.” I’ve tested across different scales:

Measured data: at the logistics platform’s 120-microservice scale, increasing concurrency from 1 to 10 dropped deployment time from 45 minutes to 5 minutes. From 10 to 20, only 5 minutes to 4 minutes. From 20 to 50, it actually went from 4 minutes to 6 minutes — massive concurrent K8s API requests caused API Server rate limiting, with requests queued.

Conclusion: concurrency 10 is the sweet spot for cost-effectiveness. Beyond 15, you need to tune K8s API Server’s --max-requests-inflight, otherwise you’re creating your own bottleneck.

Scheduler Main Loop

type Scheduler struct {
    dag        *DAG
    workerPool *WorkerPool
    stateStore StateStore
    eventBus   chan TaskResult
    pipelineID string
}

func (s *Scheduler) Run(ctx context.Context) error {
    // Checkpoint recovery: restore completed task states
    if err := s.restoreState(); err != nil {
        return fmt.Errorf("restore state failed: %w", err)
    }
    s.workerPool.Start()
    defer s.workerPool.Stop()

    ticker := time.NewTicker(500 * time.Millisecond)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case result := <-s.eventBus:
            s.handleResult(result)
        case <-ticker.C:
            s.dispatchReadyTasks()
            if !s.dag.HasRemaining() { return nil }
        }
    }
}

dispatch and handleResult

func (s *Scheduler) dispatchReadyTasks() {
    for _, task := range s.dag.GetReadyTasks() {
        s.stateStore.UpdateTaskState(s.pipelineID, task.ID, StateRunning)
        s.dag.markDispatched(task.ID)
        if err := s.workerPool.Submit(&ExecutableTask{Task: task, Result: s.eventBus}); err != nil {
            // Queue full, rollback state, retry on next tick
            s.stateStore.UpdateTaskState(s.pipelineID, task.ID, StatePending)
            s.dag.unmarkDispatched(task.ID)
        }
    }
}

func (s *Scheduler) handleResult(result TaskResult) {
    s.stateStore.SaveTaskResult(s.pipelineID, result)
    if result.State == StateSuccess || result.State == StateSkipped {
        s.dag.CompleteTask(result.TaskID)
    } else if result.State == StateFailed {
        task := s.dag.nodes[result.TaskID]
        switch task.OnFailure {
        case FailureActionAbort:
            s.abortPipeline(result.TaskID)
        case FailureActionSkip:
            s.dag.CompleteTask(result.TaskID) // Skip failed task, continue downstream
        case FailureActionManual:
            s.suspendPipeline(result.TaskID, result) // Suspend for manual intervention
        }
    }
}

Why 500ms Ticker Instead of Pure Event-Driven

Pure event-driven is theoretically more efficient — trigger next dispatch immediately when a task completes. But in practice there’s a hidden risk: race conditions between handleResult and dispatchReadyTasks could cause “task completed but downstream never scheduled.”

500ms ticker polling is simpler and more reliable. The 500ms delay is completely acceptable in CI/CD scenarios — task execution times are typically seconds to minutes, so 500ms scheduling latency is imperceptible. Plus, the ticker model naturally supports checkpoint recovery: after restart, the first tick resumes scheduling.

Checkpoint Recovery: State Persistence Design

Why Checkpoint Recovery Matters

During one release, the scheduler process got OOM-killed. 80 of 120 microservices were already deployed; 40 remained. Without checkpoint recovery, the only option was to redeploy everything — but what about the 80 already-deployed services? Roll them all back and redo? That would double the release window.

The core of checkpoint recovery: the scheduler can resume from where it left off after restart — no re-running successful tasks, no skipping unexecuted tasks.

Redis + MySQL Dual-Write

• Redis: Hot data, real-time task state updates, scheduler reads state from Redis (millisecond response)
• MySQL: Cold data, task execution results and history persistence, for audit and tracing

Dual-write order is Redis first, then MySQL. Redis write succeeds but MySQL write fails? Acceptable — worst case is missing audit records, but the scheduler keeps running. The reverse — MySQL first, then Redis — if MySQL succeeds but Redis fails, the scheduler reads stale state and re-runs tasks. This is catastrophic in database migration scenarios.

func (s *RedisStateStore) UpdateTaskState(pipelineID, taskID string, state TaskState) error {
    // 1. Write Redis first (hot path)
    key := fmt.Sprintf("pipeline:%s:task:%s:state", pipelineID, taskID)
    if err := s.redis.Set(ctx, key, string(state), 24*time.Hour).Err(); err != nil {
        return fmt.Errorf("redis write failed: %w", err)
    }
    // 2. Async write MySQL (cold path, failure only logs, doesn't affect scheduling)
    go func() {
        if _, err := s.mysql.Exec(
            "UPDATE task_executions SET state=?, updated_at=NOW() WHERE pipeline_id=? AND task_id=?",
            string(state), pipelineID, taskID); err != nil {
            log.Printf("[state] mysql write failed for task %s: %v", taskID, err)
        }
    }()
    return nil
}

Recovery Key: Idempotency Check

func (s *Scheduler) restoreState() error {
    completed, _ := s.stateStore.GetCompletedTasks(s.pipelineID)
    for _, result := range completed {
        if result.State == StateSuccess || result.State == StateSkipped {
            s.dag.CompleteTask(result.TaskID)
            s.dag.markDispatched(result.TaskID)
        } else if result.State == StateRunning {
            task := s.dag.nodes[result.TaskID]
            if s.isIdempotent(task) {
                // Idempotent task: rollback to PENDING for rescheduling
                s.stateStore.UpdateTaskState(s.pipelineID, result.TaskID, StatePending)
            } else {
                // Non-idempotent task: suspend for manual confirmation
                s.suspendPipeline(result.TaskID, result)
            }
        }
    }
    return nil
}

Key decision: non-idempotent tasks must not auto-retry. ALTER TABLE ADD COLUMN is idempotent (column already exists errors but doesn’t affect data). But UPDATE users SET status = 'active' WHERE status = 'pending' is not idempotent — re-running affects new data that became pending between the two executions. For non-idempotent tasks, suspending for manual confirmation during checkpoint recovery is far safer than auto-retry.

Rollback Strategy: Reverse DAG Execution

Why Reverse Rollback

Forward deployment is infrastructure → middleware → business → gateway. Rollback must reverse: gateway → business → middleware → infrastructure. If you roll back infrastructure first, upper-layer business services are still running and immediately hit connection errors.

Implementation: reverse all edges of the original DAG to get a reverse DAG, then execute it with the same scheduler.

func (d *DAG) Reverse() *DAG {
    reversed := &DAG{
        nodes:    make(map[string]*Task),
        edges:    make(map[string][]string),
        inDegree: make(map[string]int),
    }
    for id, task := range d.nodes {
        reversed.nodes[id] = &Task{
            ID: task.ID, Name: "rollback-" + task.Name,
            Type: task.Type, Command: task.RollbackCommand,
        }
        reversed.inDegree[id] = 0
    }
    // Reverse all edges: original from->to becomes to->from
    for from, tos := range d.edges {
        for _, to := range tos {
            reversed.edges[to] = append(reversed.edges[to], from)
            reversed.inDegree[from]++
        }
    }
    return reversed
}

Rollback Is Not Auto-Triggered

Important architecture decision: rollback provides capability but is not auto-triggered.

Auto-rollback sounds great — detect release failure and automatically roll back. But in production, the risks far outweigh the benefits:

1. False positives cause unnecessary rollbacks: Health check temporarily fails (service still starting), auto-rollback undoes a perfectly good deployment
2. Rollback itself can fail: Rolling back to old version reveals old version has a bug too — now you can neither move forward nor backward
3. Data changes are irreversible: Database migration executed ALTER TABLE; rolling back application code but table structure already changed, old code incompatible

My approach: the scheduler provides Reverse() to build the reverse DAG, but rollback triggering is a human decision. The deployment platform UI has a “one-click rollback” button. Clicking it executes the reverse DAG with a preview — which tasks will roll back, in what order, estimated duration — for human confirmation before execution.

Production Pitfalls

Pitfall 1: Topological Sort Only Sorts, Doesn’t Execute

The classic trap. Many people write a topological sort function, call it, and assume tasks will execute in parallel. But topological sort only returns a linear sequence — it doesn’t handle execution.

In the initial version, I stored the topological sort result as []string and looped through with a for loop. All “parallelizable” tasks became serial, and deployment time didn’t improve at all.

// Wrong approach — linear execution after topological sort
sorted := topologicalSort(dag)
for _, taskID := range sorted {
    execute(taskID) // Serial! Parallelism lost!
}

// Correct approach — layered parallel execution
for {
    ready := dag.GetReadyTasks() // Tasks with in-degree 0
    if len(ready) == 0 { break }
    for _, task := range ready {
        workerPool.Submit(task) // Submit in parallel
    }
    waitBatchComplete(ready) // Wait for this batch before next
}

Kahn’s layered property is the key to parallelism — each round’s in-degree-0 nodes form a layer of parallelizable tasks, not the sorted linear sequence.

Pitfall 2: Map Concurrent Read/Write Panic

Go’s map is not concurrency-safe. In the scheduler main loop, GetReadyTasks() reads the inDegree map while CompleteTask() writes to it. If these happen in different goroutines simultaneously, instant panic.

Initial fix: wrapped with sync.Mutex. But GetReadyTasks is called frequently (every 500ms), and lock contention degraded performance.

Final solution: read-write separation. GetReadyTasks uses sync.RWMutex’s read lock (RLock), CompleteTask uses write lock (Lock). In read-heavy, write-light scenarios, RWMutex performs much better. Measured at 10 reads/sec, 1 write/sec: RWMutex throughput was 3x that of Mutex.

For under 50 tasks, atomic.Value copy-on-write is also viable — copy the entire map on each write, atomically replace. But with many tasks, copy overhead is significant, not worth it.

Pitfall 3: Incomplete Context Propagation

Worker Pool creates a cancellable context with context.WithCancel(parentCtx). But if the task execution function doesn’t properly use this context, the cancel signal never reaches the actual command.

// Wrong — context not passed to exec.Command
func (wp *WorkerPool) executeShell(ctx context.Context, cmd string) (TaskState, int, string, error) {
    c := exec.Command("bash", "-c", cmd)
    output, err := c.CombinedOutput() // Doesn't respond to ctx cancel!
    // ...
}

// Correct — use CommandContext
func (wp *WorkerPool) executeShell(ctx context.Context, cmd string) (TaskState, int, string, error) {
    c := exec.CommandContext(ctx, "bash", "-c", cmd)
    // Automatically sends SIGKILL when ctx is cancelled
    output, err := c.CombinedOutput()
    // ...
}

But CommandContext defaulting to SIGKILL is too aggressive — the child process has no cleanup opportunity. Better approach: send SIGTERM first, wait 5 seconds, then SIGKILL. In production, I implemented a gracefulCommandContext that listens for context cancellation, sends SIGTERM first, and escalates to SIGKILL on timeout, giving child processes time to clean up temp files and close database connections.

Pitfall 4: Worker Pool Queue Full — Silent Task Loss

The Submit method initially used select + default, silently returning on full queue without error. The scheduler thought submission succeeded, but the task was actually dropped — this silent failure is more dangerous than blocking.

Fix: return an error, let the scheduler decide retry or degrade. The scheduler rolls back the task state to PENDING on error, re-scheduling on the next ticker. This is better than blocking — blocking would stall the scheduler main loop, affecting other tasks’ state updates.

YAML Definition: Making Release Flows Readable and Maintainable

The DAG engine’s input is a YAML-defined pipeline. Developers and ops only write YAML, no Go code needed. This is the key design for lowering the barrier to use.

# pipeline.yaml — typical deployment pipeline for a logistics platform
name: "Logistics Platform Full Release"
concurrency: 10              # Global concurrency
timeout: 3600s               # Whole pipeline timeout: 1 hour

tasks:
  # Infrastructure layer
  - id: db-migration
    name: "Database Migration"
    type: SHELL
    command: "flyway migrate -configFiles=/etc/flyway.conf"
    timeout: 300s
    retry_count: 0            # No retry for DB migration
    on_failure: MANUAL        # Manual intervention on failure
    idempotent: true

  # Middleware layer — depends on db-migration
  - id: deploy-redis-cluster
    name: "Deploy Redis Cluster"
    type: K8S_APPLY
    command: "kubectl apply -f /manifests/redis/"
    depends_on: [db-migration]
    timeout: 120s
    retry_count: 2
    on_failure: ABORT

  - id: deploy-kafka
    name: "Deploy Kafka"
    type: K8S_APPLY
    command: "kubectl apply -f /manifests/kafka/"
    depends_on: [db-migration]
    timeout: 120s
    retry_count: 2
    on_failure: ABORT
    # redis and kafka can run in parallel (both only depend on db-migration)

  # Business layer — depends on middleware
  - id: deploy-order-service
    name: "Deploy Order Service"
    type: K8S_APPLY
    command: "kubectl apply -f /manifests/order-service/"
    depends_on: [deploy-redis-cluster, deploy-kafka]
    timeout: 90s

  - id: deploy-delivery-service
    name: "Deploy Delivery Service"
    type: K8S_APPLY
    command: "kubectl apply -f /manifests/delivery-service/"
    depends_on: [deploy-redis-cluster, deploy-kafka]
    timeout: 90s
    # order and delivery can run in parallel (both only depend on middleware)

  # Gateway layer — depends on all business services
  - id: deploy-gateway
    name: "Deploy Gateway"
    type: K8S_APPLY
    command: "kubectl apply -f /manifests/gateway/"
    depends_on: [deploy-order-service, deploy-delivery-service]
    timeout: 60s

  # Verification layer
  - id: smoke-test
    name: "Smoke Test"
    type: HTTP
    command: "POST http://smoke-tester.internal/run"
    depends_on: [deploy-gateway]
    timeout: 180s
    conditions:
      - "${deploy-gateway.state} == SUCCESS"

The DAG structure of this YAML:

db-migration ──┬── deploy-redis-cluster ──┬── deploy-order-service ──┬── deploy-gateway ── smoke-test
               │                          │                          │
               └── deploy-kafka ──────────┴── deploy-delivery-service ┘

deploy-redis-cluster and deploy-kafka can run in parallel (both depend only on db-migration). deploy-order-service and deploy-delivery-service can run in parallel (both depend only on the middleware layer). The scheduler automatically identifies these parallel opportunities — no need for developers to manually declare parallel blocks.

YAML design principle: use ID references for dependencies, not nested structures. Nested structures (like Jenkins’s stage nesting parallel) become unreadable at 50+ tasks with deep indentation. Flat ID references make each YAML line independent — adding or removing tasks only changes the corresponding lines, no need to adjust indentation levels.

Performance Comparison and Alternatives

Custom Engine vs Jenkins Pipeline

Measured comparison at the logistics platform’s 120-microservice scale:

Custom Engine vs Argo Workflows

If you’re in a K8s environment, Argo Workflows is a worthy alternative:

My recommendation: If all your deployment targets are on K8s and you don’t need non-K8s orchestration, use Argo Workflows. If you have a mixed environment (K8s + traditional VMs + bare metal), or need deep customization of release logic (custom rollback strategies, integration with external approval systems), building your own is more appropriate.

When NOT to Build Your Own

Fair warning: custom isn’t a silver bullet. These scenarios are better served by existing tools:

• Single-repo CI: One repo’s build → test → deploy flow — Jenkins/GitLab CI is sufficient
• Pure K8s deployment: Everything on K8s — use ArgoCD + Argo Rollouts (Related: GitOps Workflow with ArgoCD)
• No complex dependencies: Linear release order, no cross-dependencies
• Small team: Ops team of 3 or fewer — maintenance cost exceeds benefits

The right scenario for custom: 50+ services with complex deployment topology, need for fine-grained concurrency and rollback control, mixed environment deployment needs, and team with Go development capability. Meeting these conditions makes the ROI of a custom DAG engine positive (Related: CI/CD Deployment Speed Optimization).

Deployment Architecture and Operations

Engine Deployment

The scheduler engine is a Go binary. In production, deploy 2 instances for active-standby failover. The active instance handles scheduling; the standby stands by. Redis distributed lock implements failover — the active instance renews the lock every 5 seconds; if not renewed within 15 seconds, the standby takes over.

func (s *Scheduler) acquireLeadership(ctx context.Context) bool {
    lockKey := "scheduler:leader"
    ok, err := s.redis.SetNX(ctx, lockKey, s.instanceID, 15*time.Second).Result()
    if err != nil || !ok { return false }
    // Start renewal goroutine
    go func() {
        ticker := time.NewTicker(5 * time.Second)
        defer ticker.Stop()
        for {
            select {
            case <-ctx.Done(): return
            case <-ticker.C: s.redis.Expire(ctx, lockKey, 15*time.Second)
            }
        }
    }()
    return true
}

Monitoring Metrics

The scheduler engine should expose these Prometheus metrics:

worker_pool_queue_size growing continuously means insufficient concurrency — increase Worker Pool size. task_state_total{state="failed"} spiking means a task type is failing frequently — likely a target environment issue. dag_ready_tasks persistently > 0 means the scheduler can’t keep up with task completion (Related: Alerting Strategy Design: From Noise to Signal).

Key Design Decision Review

Looking back at the entire engine design, several key decisions are worth reviewing.

Decision 1: Kahn’s algorithm or DFS topological sort?

Kahn. The core reason is natural support for layered parallelism. CI/CD scheduling isn’t “sort then run one by one” — it’s “run one layer in parallel, then the next layer.” Kahn’s in-degree table + queue model naturally fits this: each round’s in-degree-0 nodes form a layer of parallelizable tasks.

Decision 2: Redis or MySQL for state storage?

Both. Redis for hot data (real-time task state reads/writes), MySQL for cold data (history audit). Redis-first-then-MySQL dual-write order ensures the scheduler always reads the latest state; MySQL write failure doesn’t affect scheduling. Using only MySQL, the per-update latency (10-50ms) accumulates into significant scheduling overhead at 120 concurrent tasks.

Decision 3: Auto or manual rollback?

Manual. Auto-rollback has too high a false positive rate in production. Health check failure might mean the service is still starting; auto-rollback undoes a normal deployment. Manual rollback with reverse DAG — humans make decisions, machines execute — is a safer division of labor.

Decision 4: Support runtime dynamic edge addition?

No. The DAG structure is fixed at load time; no edge addition at runtime. Dynamic edges require re-detecting cycles, recalculating in-degrees, handling in-flight task states — exponential complexity increase with minimal benefit. If dynamic tasks are needed (runtime decisions on which tasks to execute), use conditional execution (Conditions field) — graph structure stays fixed, task execution determined by condition evaluation.

Decision 5: How to handle non-idempotent tasks during checkpoint recovery?

Suspend for manual confirmation. Non-idempotent task re-runs may cause data inconsistency. Rather than auto-retry and investigate later, suspend for human confirmation: “where did this task get to last time, is it safe to re-run?” While this reduces automation, in production, safety takes priority over automation rate.

Summary

Building a custom CI/CD DAG scheduler engine isn’t “reinventing Jenkins.” It’s targeted deep customization for specific scenarios. When your deployment topology is too complex for general tools to express clearly, and concurrency and rollback ordering need fine-grained control, a lightweight scheduler engine based on Kahn’s algorithm + Worker Pool + state persistence can deliver order-of-magnitude efficiency gains.

The core architecture is four layers: API (receive pipeline definitions), Scheduler (DAG topological sort + layered scheduling), Executor (Worker Pool concurrent execution), State (Redis + MySQL persistent checkpoint recovery). Each layer has a single responsibility, independently testable and replaceable.

Hard-won lessons:

1. Topological sort only sorts, doesn’t execute — Kahn’s layered property is the key to parallelism, not the sorted linear sequence
2. Map concurrent read/write will panic — RWMutex or atomic.Value, no third option
3. Context must propagate to the end — exec.CommandContext not exec.Command, or cancel signals never reach child processes
4. Non-idempotent tasks must not auto-retry — suspend for manual confirmation during checkpoint recovery, safer than investigating data inconsistency later
5. Rollback is not auto-triggered — reverse DAG provides capability, humans make decisions

This engine ran for 18 months at the logistics platform, supporting 2000+ deployments across 120+ microservices. Average deployment time dropped from 1.5 hours to 5 minutes, with zero deployment-caused incidents. Not because it’s particularly ingenious, but because it solved exactly the core pain point of that scenario — complex dependency relationships need more flexible expression than the Stage model.

If you’re considering building your own deployment platform, ask yourself three questions: Is the deployment topology too complex for YAML? Does concurrency need fine-grained control? Does rollback have ordering constraints? If two of three answers are “yes,” this approach is worth referencing.

References & Acknowledgments

The following resources were referenced during the writing of this article. Thanks to the original authors for their contributions:

1. Harness Workflow Engine Kernel Analysis — Harness platform team, referenced for Pipeline/Stage/Step data model design and state machine transition rules
2. DAG Task Scheduling Pitfall Guide: Why Your Parallel Tasks Always Become Serial? — weixin_29281915, referenced for topological sort limitations and thread management misconceptions
3. Key to Building Efficient CI/CD Pipelines (Dependency Graph Deep Analysis) — CompiLume, referenced for dependency graph anti-patterns and parallelization identification
4. Go Task Orchestration and Scheduling: Golang DAG Directed Acyclic Graph Execution — php.cn technical community, referenced for Kahn vs DFS comparison and cycle detection engineering practices
5. State Machine Management in Multi-Agent Collaboration: Building a Lightweight DAG Task Flow Engine in Go — baronbool, referenced for DAG + FSM collaborative architecture design
6. 2 Years of Pitfalls: Enterprise Jenkins CI/CD Setup Guide — Tencent Cloud Developer Community, referenced for Jenkins Master/Agent architecture bottleneck analysis