Starting from Docker Machine Deprecation: 7 Production Decisions for GitLab CI Runner Elastic Architecture and Cache Governance

Overview

1 AM. You get an alert: GitLab CI pipeline queue has 47 pending jobs. The dev chat explodes—“code pushed 40 minutes ago, still not running"“is the Runner down?““just add more machines!” You check the console: all 3 Runners are at capacity, each running 10 concurrent jobs, 30 slots fully occupied. Add machines? Docker Machine executor needs 3 minutes to spin up a new EC2, then another 2 minutes to become Ready. Five minutes pass, the queue grows by 20 more.

This isn’t hypothetical. In 2025, while rebuilding a CI/CD platform for a mobility project, I hit this exact scenario. The Runner architecture was classic Docker Machine + AWS EC2 autoscaling, with peak queue times of 30-40 minutes being normal. After migrating to Kubernetes Executor + HPA autoscaling, queue times dropped to under 30 seconds. Along the way, we hit plenty of pitfalls and made plenty of architecture decisions.

This article skips .gitlab-ci.yml basics (there’s enough of that on the internet) and focuses on 7 engineering decisions that actually impact production efficiency: executor selection, resource planning, elastic scaling, cache design, pipeline orchestration, node segregation, and monitoring. Each decision comes with real test data and pitfall details.

If you’re evaluating CI/CD platform options, check out my earlier article Don’t Let Jenkins Dictate Your Release Cadence: Architecture Decisions and Lessons from Building a Go DAG Scheduler Engine for a comparison of Jenkins, GitLab CI, and a custom scheduler engine.

Decision 1: Executor Selection — What to Do After Docker Machine Deprecation

Executor Landscape Comparison

GitLab Runner supports multiple executors, each corresponding to a different runtime environment and isolation method. Choosing the wrong executor undermines everything downstream—elastic scaling and cache design become meaningless.

Docker Machine Deprecation Timeline

GitLab 17.5 (released October 2024) officially deprecated the Docker Machine executor. Full removal is planned for GitLab 20.0 (May 2027). This means:

• No new feature development
• Only critical bugs affecting CI/CD execution or cost are fixed
• Existing users must migrate to Docker Autoscaler or Kubernetes Executor before this date

The Docker Machine executor depends on a GitLab-maintained fork of Docker Machine (Docker officially stopped maintaining it long ago). This fork requires ongoing adaptation to cloud provider API changes, has high maintenance costs, and cannot support newer features like the fleeting scheduling framework.

My Recommendation: Kubernetes Executor

Given the 2026 landscape, if you have a Kubernetes cluster, choose Kubernetes Executor without hesitation. Here’s why:

1. Native elastic scaling: Combined with HPA + Cluster Autoscaler, Pod-level scaling is 10x faster than VM-level (seconds vs minutes)
2. High resource utilization: Job Pods are destroyed after execution, freeing resources immediately. With Docker Machine, EC2 instances incur charges even when idle until reclaimed
3. Unified operations: Both the Runner management plane and Job execution plane live within the K8s ecosystem—troubleshoot with kubectl, no need for separate Docker Machine tooling
4. Scheduling capabilities: Supports nodeSelector, tolerations, affinity, and other K8s scheduling features for fine-grained resource isolation

The one scenario where Kubernetes Executor is not recommended: Your CI/CD tasks need direct host access (kernel module testing, hardware driver compilation). In this case, Shell or Instance executors are more appropriate.

Migration Path from Docker Machine to Kubernetes Executor

Migration isn’t a flip of a switch. I recommend a “dual-track parallel + gradual cutover” strategy:

# Step 1: Deploy new K8s Runner, registered to the same GitLab instance
# Use different tags to distinguish old and new Runners
# Old Runner tag: docker-machine
# New Runner tag: kubernetes

# Step 2: In .gitlab-ci.yml, gradually switch job tags from docker-machine to kubernetes
# Start with non-critical jobs (lint, doc generation), then build jobs, finally deploy jobs

# Step 3: Observe for 1-2 weeks, then decommission Docker Machine Runner after confirming no compatibility issues

The most common migration pitfall: Under Docker Machine executor, jobs run on standalone EC2 instances with flat networking. Under K8s Executor, job Pods run in the Pod network. If your CI/CD scripts contain hardcoded IP addresses or rely on host networking, they will break after migration. Run grep -r '[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+' scripts/ before migrating.

Decision 2: Runner Manager Pod Resource Planning — Data-Driven

Manager Pod Responsibilities

Under the Kubernetes Executor architecture, the Runner Manager Pod is the scheduling hub for the entire CI/CD system. It doesn’t execute build/test/deploy tasks, but handles:

• Log processing: Collects log streams from Job Pods and forwards them to GitLab
• Cache management: Coordinates local cache and S3 distributed cache upload/download
• K8s API interaction: Creates, monitors, and deletes Job Pods
• GitLab API communication: Polls for jobs and reports execution status
• Pod lifecycle management: Manages Job Pod provisioning and cleanup

Each responsibility consumes different resource types. GitLab’s official performance testing provides a clear resource consumption model.

Official Test Data

The GitLab team ran systematic tests using a 4MB log-producing stress test job. The method was parallel: 100 concurrent execution, each job generating 4MB of random data and reading it chunk by chunk. Results:

From the data, we can derive the resource consumption formula:

• CPU: Base 10m + ~6m per concurrent job
• Memory: Base 50MB + ~2.5MB per concurrent job (based on 4MB log volume)

My Resource Planning Formula

Based on the official data, here’s the resource planning formula I use in production:

def calculate_manager_resources(concurrent_jobs, avg_log_mb=4):
    """GitLab Runner Manager Pod resource planning"""
    # CPU: ~6m per concurrent job + 10m base
    base_cpu = 0.01
    cpu_per_job = 0.006
    total_cpu = base_cpu + (concurrent_jobs * cpu_per_job)

    # Memory: ~2.5MB per job + 50MB base
    base_memory = 50
    memory_per_job = 2.5 * (avg_log_mb / 4)
    total_memory = base_memory + (concurrent_jobs * memory_per_job)

    return {
        'cpu_request': f"{int(total_cpu * 1000)}m",
        'cpu_limit': f"{int(total_cpu * 1.5 * 1000)}m",      # 50% headroom
        'memory_request': f"{int(total_memory)}Mi",
        'memory_limit': f"{int(total_memory * 2.0)}Mi"        # 100% headroom
    }

In practice, the configuration for 50 concurrent jobs:

# Manager Pod resource config for 50 concurrent jobs
resources:
  requests:
    cpu: "310m"    # 10m + (50 x 6m) = 310m
    memory: "175Mi" # 50 + (50 x 2.5) = 175MB
  limits:
    cpu: "465m"    # 50% headroom
    memory: "350Mi" # 100% headroom

Pitfall: Insufficient Memory Causes Silent Log Truncation

One pitfall is particularly insidious: when the Manager Pod runs low on memory, job logs get silently truncated. No error is thrown—the GitLab UI shows the log ending normally, but the last few lines are missing. If your CI/CD script relies on the final log output for build results (e.g., echo "BUILD_RESULT=success" followed by parsing), log truncation causes parse failures.

How to detect: Compare the last line of the job log in GitLab UI with the Job Pod’s actual stdout. If they don’t match, it’s log truncation.

Solution: Manager Pod memory limit should have at least 100% headroom. The x 2.0 in the formula isn’t over-engineering—it’s hard-won experience.

Decision 3: Elastic Scaling — From 30-Minute Queues to 30 Seconds

Three-Layer Elastic Scaling Architecture

GitLab CI Runner elastic scaling on K8s isn’t a single mechanism—it’s three layers working together:

┌─────────────────────────────────────────────────────────┐
│ Layer 1: Runner Manager Concurrency Control              │
│ config.toml: concurrent = 50                            │
│ Role: How many jobs a single Manager Pod can handle     │
├─────────────────────────────────────────────────────────┤
│ Layer 2: HPA Horizontal Scaling of Manager Pods          │
│ Role: Auto-scale Manager Pod count based on CPU/memory   │
│ minReplicas: 2  maxReplicas: 5                          │
├─────────────────────────────────────────────────────────┤
│ Layer 3: Cluster Autoscaler Expands K8s Nodes            │
│ Role: Auto-add nodes when Job Pods are pending           │
└─────────────────────────────────────────────────────────┘

HPA Configuration

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: gitlab-runner-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: gitlab-runner
  minReplicas: 2        # At least 2 for HA
  maxReplicas: 5        # Max 5, each with 50 concurrent = 250 total
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70   # Scale up when CPU exceeds 70%
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80   # Scale up when memory exceeds 80%

How to Determine Concurrency

The concurrent parameter (max concurrent jobs per Manager Pod) is the core tuning knob. Too high, and the Manager Pod runs out of resources causing OOM; too low, and you need more Manager Pods with increased management overhead.

My recommended values:

Key parameter request_concurrency: Controls how many concurrent requests the Manager Pod makes to the GitLab API for new jobs. Default is 1, meaning even with concurrent=50, the Manager only pulls one job at a time. If you notice job startup delays (jobs queued but Manager idle), increase request_concurrency to 5-10.

# config.toml key settings
concurrent = 50

[[runners]]
  limit = 50
  request_concurrency = 10   # Default 1, higher values speed up job pickup

  [runners.kubernetes]
    namespace = "gitlab-runner"
    poll_interval = "5s"       # Default 3s, increase to 5s to reduce K8s API pressure
    poll_timeout = "180s"

Measured Comparison: Docker Machine vs Kubernetes Executor

During the mobility project migration, I recorded comparison data:

The core reason for the cold start difference: Docker Machine needs to call cloud API to create EC2 → wait for Running → SSH connect → pull Runner image → start container, taking 3-4 minutes total. K8s Executor only needs K8s API to create Pod → schedule to existing node → pull image (seconds with local cache) → start container, completing in under 30 seconds.

Decision 4: Distributed Cache Design — 3x Build Speedup

Cache vs Artifacts: 90% of People Confuse Them

These are the two most misunderstood concepts in GitLab CI:

• Cache: Stores dependencies (e.g., node_modules/, .m2/), reused across pipelines. Stored on Runner local or S3. Not guaranteed available—Runner can clear cache at any time.
• Artifacts: Stores build outputs (e.g., target/app.jar, dist/), passed between stages within the same pipeline. Stored on the GitLab instance, with expiration.

Simply put: Cache is “dependency cache,” Artifacts is “stage output.” Many teams put compiled results in Cache and then can’t retrieve them across stages—because Cache is cross-pipeline, not cross-stage. Use Artifacts for inter-stage file passing.

S3 Distributed Cache Configuration

Single-machine Runner can use local cache, but multiple Runners require distributed cache. Otherwise, cache generated by Runner A can’t be accessed by Runner B, making caching pointless.

# config.toml distributed cache config
[runners.cache]
  Type = "s3"
  Shared = true    # Critical! When enabled, all Runners share the same cache bucket

  [runners.cache.s3]
    ServerAddress = "minio.internal.example.com"
    BucketName = "gitlab-runner-cache"
    Insecure = false
    AuthenticationType = "access-key"
    # AccessKey and SecretKey injected via environment variables, not in config file

Cache Key Design Strategies

Cache Key determines cache hit rate. Poor design means either cache never hits (re-downloading dependencies every time) or cache version conflicts (different branches’ dependencies mixed together).

# .gitlab-ci.yml cache config

# Strategy 1: Hash by dependency files (recommended)
# Cache is reused when dependencies unchanged, refreshed when changed
cache:
  key:
    files:
      - go.sum          # Go dependency lock file
      - package-lock.json # Node dependency lock file
  paths:
    - .cache/go-build/
    - node_modules/

# Strategy 2: Cache by branch (for projects with large dependency differences between branches)
cache:
  key: "$CI_COMMIT_REF_SLUG"
  paths:
    - .m2/repository/

# Strategy 3: Combined branch + dependency file (most granular)
cache:
  key:
    key: "$CI_COMMIT_REF_SLUG"
    files:
      - go.sum
  paths:
    - .cache/

I recommend Strategy 1 (hash by dependency files) because:

1. Most projects share the same dependencies across branches. Caching by branch causes each branch to re-download everything, wasting resources
2. Cache automatically invalidates when dependency files change, no manual version management needed
3. With Shared = true, all branches share the same cache, maximizing utilization

Measured Cache Impact

Using a Go project as example (200+ dependencies, go.sum ~50KB):

Cache hit reduces build time from 82 seconds to 23 seconds, a 3.5x speedup. For a team with 200 daily builds, this saves about 3.3 hours of build time per day.

Pitfall: Cache Version Conflict

In an e-commerce platform project, I hit this pitfall: the development branch used v1.2 of a dependency while the main branch used v1.1. The Cache Key only used the project name (key: "myproject"), causing the development branch’s cache to be picked up by main branch jobs, resulting in compilation errors.

It took 2 hours to track down the cache version conflict. The fix was to change the Cache Key to use dependency file hash:

# Before (wrong)
cache:
  key: "myproject"
  paths:
    - vendor/

# After (correct)
cache:
  key:
    files:
      - go.mod
  paths:
    - vendor/

Decision 5: .gitlab-ci.yml Pipeline Orchestration — include and Dynamic Pipelines

Template Reuse with include

When you have 10+ microservices with 90% identical .gitlab-ci.yml logic, maintaining 10 config files is a disaster. GitLab CI’s include keyword enables template reuse:

# .gitlab-ci.yml (individual microservice project)
include:
  - project: 'devops/ci-templates'
    file: '/go-service.yml'
    ref: 'main'

# Project-specific variables
variables:
  SERVICE_NAME: "user-service"
  REGISTRY: "registry.example.com"

# Only override project-specific config
# Common build, test, deploy logic is in the go-service.yml template

Template file go-service.yml:

# go-service.yml (shared template)
stages:
  - lint
  - test
  - build
  - deploy

variables:
  # Default values, overridable per project
  GO_VERSION: "1.22"

lint:
  stage: lint
  image: golang:${GO_VERSION}
  script:
    - go fmt ./...
    - go vet ./...
    - golangci-lint run

test:
  stage: test
  image: golang:${GO_VERSION}
  cache:
    key:
      files: [go.sum]
    paths: [.cache/]
  script:
    - go test -race -coverprofile=coverage.out ./...

build:
  stage: build
  image: docker:24.0
  services: [docker:24.0-dind]
  script:
    - docker build -t $REGISTRY/$SERVICE_NAME:$CI_COMMIT_SHORT_SHA .
    - docker push $REGISTRY/$SERVICE_NAME:$CI_COMMIT_SHORT_SHA

Dynamic Pipelines (Parent-Child Pipeline)

When pipeline logic becomes complex enough to require conditional generation, use Parent-Child Pipelines. For example: only run build when src/ has changes, only run docs deployment when docs/ has changes.

# .gitlab-ci.yml (parent pipeline)
stages:
  - generate
  - trigger

generate-config:
  stage: generate
  script:
    - |
      cat > generated-config.yml << 'EOF'
      include:
        - project: 'devops/ci-templates'
          file: '/go-service.yml'
      EOF
      # If docs/ has changes, append docs deployment job
      if git diff --name-only HEAD~1 | grep -q "^docs/"; then
        cat >> generated-config.yml << 'EOF'
        deploy-docs:
          stage: deploy
          script:
            - make docs-deploy
        EOF
      fi      
  artifacts:
    paths: [generated-config.yml]

trigger-child:
  stage: trigger
  trigger:
    include:
      - artifact: generated-config.yml
        job: generate-config
  strategy: depend    # Parent pipeline waits for child pipeline

The benefit of this pattern is dynamically generated config files with maximum flexibility. The downside is debugging complexity—when errors occur, you first need to check the parent pipeline’s generate-config job output to verify the generated config is correct.

rules vs only/except

GitLab 13.12+ recommends rules over only/except. rules is more flexible, supporting if conditions and changes matching:

# Recommended: rules
build:
  stage: build
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: never       # Skip build on MR pipelines
    - if: '$CI_COMMIT_BRANCH == "main"'
      changes:
        - src/**/*
      when: on_success   # Build only when src/ changes on main
    - when: manual        # Manual trigger otherwise
  script:
    - make build

Why only/except is not recommended: The only/except syntax has known limitations (no complex condition composition, and GitLab no longer adds new features to it). Use rules for new projects and gradually migrate old projects.

Decision 6: Node Segregation — Manager and Jobs Must Not Mix

Why Segregation Matters

By default, Manager Pods and Job Pods may be scheduled on the same K8s node. This causes two problems:

1. Resource contention: Job Pods (build tasks) consume significant CPU/memory, affecting Manager Pod scheduling stability
2. Log processing latency: The Manager Pod needs to process Job Pod log streams. If the Manager itself is slowed down by high-load Job Pods, log delays cause the GitLab UI to show no real-time output

Node Segregation Configuration

# Taint and label Manager nodes
kubectl taint nodes node-pool-manager runner.gitlab.com/manager=:NoExecute
kubectl label nodes node-pool-manager runner.gitlab.com/workload-type=manager

# Taint and label Worker nodes
kubectl taint nodes node-pool-worker runner.gitlab.com/job=:NoExecute
kubectl label nodes node-pool-worker runner.gitlab.com/workload-type=job

Manager Pod scheduling configuration:

# Manager Pod with nodeSelector and tolerations
spec:
  nodeSelector:
    runner.gitlab.com/workload-type: manager
  tolerations:
    - key: runner.gitlab.com/manager
      operator: Exists
      effect: NoExecute

Job Pod scheduling configuration (in config.toml):

[runners.kubernetes.node_selector]
"runner.gitlab.com/workload-type" = "job"

[runners.kubernetes.node_tolerations]
"runner.gitlab.com/job=" = "NoExecute"

Before vs After Segregation

In a 200-concurrency scenario, stability comparison before and after segregation:

The improvement in Job Pod creation delay after segregation was unexpected: the Manager Pod is no longer starved of CPU by Job Pods, making K8s API calls respond faster.

The cost of node segregation is additional Manager nodes. For small teams (<20 concurrency), this isn’t cost-effective. My recommendation: implement segregation when concurrency exceeds 50. For resource management fundamentals, see Kubernetes Resource Management: Requests and Limits.

Decision 7: Monitoring and Alerting — Replace User Complaints with Prometheus

Key Metrics

GitLab Runner exposes a Prometheus metrics endpoint (:9252/metrics). These are the metrics I monitor in production:

Prometheus Alert Rules

# Manager Pod sustained high CPU utilization
groups:
  - name: gitlab-runner
    rules:
      - alert: RunnerManagerHighCPU
        expr: |
          rate(container_cpu_usage_seconds_total{pod=~"gitlab-runner.*"}[5m]) * 1000
          / kube_pod_container_resource_limits{pod=~"gitlab-runner.*", resource="cpu"} * 1000
          > 70          
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Runner Manager CPU utilization sustained above 70%"

      - alert: RunnerManagerOOM
        expr: |
          container_memory_working_set_bytes{pod=~"gitlab-runner.*"}
          / kube_pod_container_resource_limits{pod=~"gitlab-runner.*", resource="memory"}
          > 0.8          
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Runner Manager memory usage above 80%, OOM imminent"

      - alert: RunnerJobQueueSaturation
        expr: |
          gitlab_runner_jobs / gitlab_runner_limit > 0.9          
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Runner Job queue saturation above 90%"

      - alert: RunnerErrors
        expr: increase(gitlab_runner_errors_total[5m]) > 0
        labels:
          severity: critical
        annotations:
          summary: "Runner errors in the last 5 minutes"

Performance Thresholds

GitLab’s official performance threshold reference:

I recommend more aggressive alert thresholds for job queue time: warning at 15 seconds, critical at 60 seconds. A 30-second queue already feels broken to developers—pushing code and waiting 30 seconds for the pipeline to start is perceived as “the system is down.”

Diagnostic Commands

Common commands for troubleshooting Runner performance issues:

# View current resource usage of Manager Pod
kubectl top pods --containers -l app=gitlab-runner

# Check error logs from the last 2 hours
kubectl logs -l app=gitlab-runner --since=2h | grep -E "(error|timeout|failed)"

# View pending Job Pods (persistent pending indicates insufficient resources)
kubectl get pods -n gitlab-runner --field-selector=status.phase=Pending

# Check if Manager Pod was OOMKilled
kubectl get pods -l app=gitlab-runner -o jsonpath='{.items[].status.containerStatuses[].lastState.terminated.reason}'

Summary

The core of GitLab CI Runner elastic architecture isn’t “add more machines”—it’s “choose the right executor + plan resources well + design cache properly.” Priority of the 7 decisions:

1. Executor selection is the foundation of everything. Docker Machine is deprecated, full removal in May 2027. Choose Kubernetes Executor if you have a K8s cluster, Docker Autoscaler otherwise.
2. Resource planning should be data-driven. CPU = 6m × concurrency + 10m, Memory = 2.5MB × concurrency + 50MB. Keep 100% memory headroom—don’t skimp.
3. Elastic scaling requires three layers working together. Runner concurrent → HPA → Cluster Autoscaler, all indispensable.
4. Distributed cache is the key to build speedup. S3 shared cache + dependency file hash as Cache Key gives the highest hit rate.
5. Pipeline orchestration uses include for template reuse and dynamic pipelines for conditional orchestration. Replace only/except with rules.
6. Node segregation prevents Manager and Job resource contention. Implement when concurrency exceeds 50.
7. Monitoring and alerting replaces user complaints with Prometheus. Alert when job queue exceeds 15 seconds.

One final thought: these decisions weren’t made correctly on the first try. In the actual mobility project implementation, we spent 3 months iterating through 4 versions before stabilizing. V1 only migrated executors without configuring cache; V2 added S3 cache but Key design was wrong, hit rate below 30%; V3 fixed the Key strategy but skipped node segregation, causing Manager Pod OOM during peak hours. V4 finally connected all decisions, reducing queue time from 30 minutes to 30 seconds. CI/CD platform building is a continuous optimization process—don’t expect to nail it in one shot.

References & Acknowledgments

1. GitLab Runner Executors — GitLab official documentation, executor type comparison and selection guide
2. Install and register GitLab Runner for autoscaling with Docker Machine — GitLab official documentation, Docker Machine deprecation timeline
3. Optimize GitLab Runner manager pod performance — GitLab official documentation, Manager Pod resource planning formulas and performance test data
4. Runner fleet configuration and best practices — GitLab official documentation, Runner fleet design best practices
5. Caching in GitLab CI/CD — GitLab official documentation, Cache vs Artifacts and caching strategies
6. Advanced configuration — GitLab official documentation, config.toml advanced configuration parameters