Overview

Many teams perceive Docker Compose as merely a “local development orchestration tool.” In reality, for small to medium production scenarios (single node or a handful of nodes), Compose remains a highly cost-effective solution. Its syntax is simple, the learning curve is low, and it doesn’t require a full K8s cluster operations team—yet it handles multi-service orchestration, dependency management, health checks, and resource limits effectively.

This article skips basic Compose syntax and focuses on real production pain points: how to manage service dependencies without cascading startup failures, how to write reliable health checks, how to avoid hardcoding secrets in compose files, how to prevent logs from filling up disk space, and when to migrate from Compose to K8s.

This article is based on Docker Compose V2 (docker compose subcommand). V1 (docker-compose standalone binary) is no longer maintained. Reference: Compose Specification

Multi-Service Orchestration

Production-Grade Compose File Structure

A typical production application includes at minimum: application service, database, cache, and reverse proxy. Here’s a complete web application orchestration example:

# docker-compose.yml
name: myapp

services:
  # ========== Reverse Proxy ==========
  nginx:
    image: nginx:1.25-alpine
    container_name: myapp-nginx
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./nginx/conf.d:/etc/nginx/conf.d:ro
      - cert_data:/etc/letsencrypt:ro
      - log_data:/var/log/nginx
    depends_on:
      web:
        condition: service_healthy
    networks:
      - frontend
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

  # ========== Application Service ==========
  web:
    build:
      context: .
      dockerfile: Dockerfile
      args:
        - GIT_COMMIT=${GIT_COMMIT:-unknown}
    image: myapp/web:${TAG:-latest}
    container_name: myapp-web
    restart: unless-stopped
    environment:
      - DB_HOST=postgres
      - DB_PORT=5432
      - DB_NAME=${DB_NAME}
      - REDIS_HOST=redis
      - ENV=production
    env_file:
      - .env.production
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:8080/health"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s
    deploy:
      resources:
        limits:
          cpus: "2.0"
          memory: 1G
        reservations:
          cpus: "0.5"
          memory: 256M
    networks:
      - frontend
      - backend
    logging:
      driver: json-file
      options:
        max-size: "20m"
        max-file: "5"

  # ========== Database ==========
  postgres:
    image: postgres:16-alpine
    container_name: myapp-postgres
    restart: unless-stopped
    environment:
      POSTGRES_DB: ${DB_NAME}
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    volumes:
      - pg_data:/var/lib/postgresql/data
      - ./sql/init:/docker-entrypoint-initdb.d:ro
    secrets:
      - db_password
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${DB_USER} -d ${DB_NAME}"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 10s
    networks:
      - backend
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "3"

  # ========== Cache ==========
  redis:
    image: redis:7-alpine
    container_name: myapp-redis
    restart: unless-stopped
    command: >
      redis-server
      --requirepass ${REDIS_PASSWORD}
      --maxmemory 256mb
      --maxmemory-policy allkeys-lru
      --appendonly yes      
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD}", "ping"]
      interval: 10s
      timeout: 3s
      retries: 3
    networks:
      - backend

networks:
  frontend:
    driver: bridge
  backend:
    driver: bridge
    internal: true   # Backend network not exposed to host

volumes:
  pg_data:
    driver: local
  redis_data:
    driver: local
  cert_data:
    driver: local
  log_data:
    driver: local

secrets:
  db_password:
    file: ./secrets/db_password.txt

Network Isolation Design

The above configuration uses two networks:

NetworkPurposeCharacteristics
frontendNginx ↔ WebPorts exposed to host
backendWeb ↔ DB/Redisinternal: true, no routing to host

internal: true is a frequently overlooked security hardening measure. It prevents containers on the backend network from accessing external networks—even if a container is compromised, the attacker cannot directly connect to a C2 server or create a reverse shell.

Multi-Environment Management

Don’t maintain a complete compose file for each environment. Use the override mechanism:

# Directory structure
# docker-compose.yml          # Base configuration
# docker-compose.override.yml  # Dev environment overrides (auto-loaded)
# docker-compose.prod.yml      # Production overrides
# docker-compose.staging.yml   # Staging overrides

# docker-compose.prod.yml
services:
  web:
    environment:
      - ENV=production
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: "2.0"
          memory: 1G

  postgres:
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    deploy:
      resources:
        limits:
          cpus: "4.0"
          memory: 4G

# Start production environment
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

Health Checks

Why Health Checks Are Mandatory

Without healthcheck, depends_on only guarantees container startup order, not service readiness. The classic pitfall:

web container starts → tries to connect to postgres → postgres process is up but not accepting connections → web crashes → restart → retry → loop

With health checks configured, depends_on can use condition: service_healthy, ensuring dependent services are truly ready before starting.

Health Check Writing Essentials

healthcheck:
  test: ["CMD", "wget", "--spider", "-q", "http://localhost:8080/health"]
  interval: 10s       # Check every 10 seconds
  timeout: 5s         # 5 seconds timeout = failure
  retries: 3          # 3 consecutive failures = unhealthy
  start_period: 30s   # Don't count failures within 30s of startup (warmup time)

Health check commands for different services:

Service TypeHealth Check CommandNotes
HTTP APIwget --spider -q http://localhost:PORT/healthCheck HTTP status code
PostgreSQLpg_isready -U user -d dbOfficial readiness check tool
Redisredis-cli -a password pingReturns PONG = healthy
MySQLmysqladmin ping -h localhostReturns mysqld is alive
MongoDBmongosh --eval "db.adminCommand('ping')"Health check command
gRPCgrpc_health_probe -addr=localhost:PORTRequires grpc_health_probe

Note: Use CMD instead of CMD-SHELL in the test field to avoid shell injection risk and improve execution efficiency. Use CMD-SHELL only when shell features (pipes, variable expansion) are needed.

/health Endpoint Design

A health check endpoint should do more than return 200—it should perform real dependency checks:

// Go example: deep health check
func healthHandler(w http.ResponseWriter, r *http.Request) {
    ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
    defer cancel()

    // Check database
    if err := db.PingContext(ctx); err != nil {
        http.Error(w, "database unreachable", http.StatusServiceUnavailable)
        return
    }

    // Check Redis
    if _, err := redis.Ping(ctx).Result(); err != nil {
        http.Error(w, "redis unreachable", http.StatusServiceUnavailable)
        return
    }

    w.WriteHeader(http.StatusOK)
    fmt.Fprint(w, "ok")
}

But be careful: deep health checks shouldn’t be too heavy. If the health check endpoint itself queries downstream databases, it can cause cascading failures. In production, separate liveness and readiness:

  • /health/live: Only check if the process is alive (just return 200)
  • /health/ready: Check dependency readiness (DB, Redis connectivity)

Dependency Management

Three depends_on Conditions

depends_on:
  postgres:
    condition: service_started      # Container process started (default)
  redis:
    condition: service_healthy      # Health check passed
  migration:
    condition: service_completed_successfully  # Container finished with exit code 0

service_completed_successfully is a very practical condition, commonly used for database migrations:

services:
  migration:
    image: myapp/web:${TAG:-latest}
    command: ["./migrate", "up"]
    depends_on:
      postgres:
        condition: service_healthy
    restart: "no"  # Exit after migration, don't restart

  web:
    depends_on:
      migration:
        condition: service_completed_successfully

Circular Dependency Problem

Compose doesn’t support circular dependencies. If A depends on B and B depends on A, Compose will error. The solution is to introduce an init container to break the cycle.

Startup Order Pitfall

Wrong order: nginx → web → postgres → redis!
Correct order: postgres/redis → web → nginx

The dependency chain must be bottom-up: infrastructure → application → gateway. In the complete example above, the dependency chain is:

postgres/redis (no dependencies)
    ↓
web (depends on postgres/redis healthy)
    ↓
nginx (depends on web healthy)

Resource Limits

deploy.resources vs Docker Compose V2 Resource Limits

In Compose V2, resource limits go under deploy.resources, very similar to K8s resource configuration:

deploy:
  resources:
    limits:
      cpus: "2.0"      # Max 2 cores
      memory: 1G       # Max 1GB memory
      pids: 100        # Max 100 processes
    reservations:
      cpus: "0.5"      # Reserved 0.5 core
      memory: 256M     # Reserved 256MB memory
ConfigEffectProduction Recommendation
limits.cpusCPU cap, throttled when exceeded1.5-2x application CPU
limits.memoryMemory cap, OOM Kill when exceeded1.2-1.5x app memory peak
limits.pidsProcess count cap, prevents fork bombs100-500
reservations.cpusCPU reservation, guarantees minimum compute0.5-1x app CPU average
reservations.memoryMemory reservation, guarantees minimum memory0.5-1x app memory average

Note: reservations in non-Swarm mode only provides soft guarantees (cgroup settings), without resource scheduling. True hard guarantees require Swarm or K8s scheduler.

Understanding OOM Behavior

When container memory exceeds limits.memory, the kernel OOM-kills the process with the highest memory usage in that container. When PID 1 is killed, the container exits, and the restart policy determines whether it restarts.

# Check if container was OOM killed
docker inspect myapp-web --format='{{.State.OOMKilled}}'
docker inspect myapp-web --format='{{.State.ExitCode}}'  # 137 = 128+9(SIGKILL)

Rule of thumb: if an application is frequently OOM-killed, first investigate memory leaks, then consider increasing the limit. Don’t just add memory first—that’s only masking the problem.

Logging Configuration

Log Driver Selection

logging:
  driver: json-file    # Default, writes local files
  options:
    max-size: "20m"    # Max 20MB per log file
    max-file: "5"      # Keep at most 5 files
DriverUse CaseCharacteristics
json-fileSingle node / small scaleSimple, needs max-size/max-file
fluentdCentralized loggingReal-time push to Fluentd
gelfGraylogGELF format
syslogLegacy logging systemsRFC 5424
journaldsystemd environmentsIntegrates with journald
localOptimized storageBinary format, efficient rotation

Production red line: If the json-file driver is used without max-size and max-file, log files grow indefinitely. This is one of the most common causes of disk-full incidents. Global defaults can be set in /etc/docker/daemon.json.

Global Log Configuration

// /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "20m",
    "max-file": "5"
  }
}

# Restart Docker after changes
systemctl restart docker

Centralized Logging Solution

Single-node json-file is fine, but multi-node setups should centralize log collection. Integrate Fluentd in Compose:

services:
  web:
    logging:
      driver: fluentd
      options:
        fluentd-address: "localhost:24224"
        fluentd-async: "true"
        fluentd-buffer-limit: "8192"
        tag: "myapp.web"

  fluentd:
    image: fluent/fluentd:v1.16-debian
    volumes:
      - ./fluentd/conf:/fluentd/etc:ro
    ports:
      - "24224:24224"
      - "24224:24224/udp"

Secret Management

Three Approaches Compared

ApproachSecurityComplexityUse Case
Environment variablesLowLowDevelopment
.env fileMediumLowSmall-scale production
Docker SecretsHighMediumProduction

What Not to Do

# ❌ Hardcoded password
environment:
  - POSTGRES_PASSWORD=MySecret123

# ❌ Committing .env file to Git
# .gitignore must include .env*

Using Docker Secrets

# Create secret file
echo "my-super-secret-password" > ./secrets/db_password.txt
chmod 600 ./secrets/db_password.txt

# .gitignore
echo "secrets/" >> .gitignore

# docker-compose.yml
secrets:
  db_password:
    file: ./secrets/db_password.txt

services:
  postgres:
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password

Inside the container, secrets are mounted at /run/secrets/<secret_name> as a tmpfs filesystem—not written to disk. The application reads the file to get the password:

func loadSecret(name string) (string, error) {
    data, err := os.ReadFile(fmt.Sprintf("/run/secrets/%s", name))
    if err != nil {
        return "", err
    }
    return strings.TrimSpace(string(data)), nil
}

External Secret Management

For larger-scale deployments, integrate Vault or cloud-provider secret management services:

# Use dotenv to pull secrets from Vault
services:
  web:
    env_file:
      - .env.production  # Dynamically generated by deployment script from Vault

# Deployment script example: generate .env from Vault
vault kv get -format=json secret/myapp/prod | \
  jq -r '.data.data | to_entries[] | "\(.key)=\(.value)"' > .env.production

docker compose up -d

# Delete after deployment
rm .env.production

Compose and Swarm

When to Use Swarm

ScenarioRecommended
Single node, <10 containersDocker Compose
2-5 nodes, need HADocker Swarm
>5 nodes, complex schedulingKubernetes

Swarm’s advantage is its extremely low learning curve—Compose files work almost directly, just add a few Swarm-specific fields.

Swarm Deployment Example

# docker-compose.swarm.yml
services:
  web:
    image: myapp/web:${TAG:-latest}
    deploy:
      replicas: 3
      update_config:
        parallelism: 1
        delay: 10s
        failure_action: rollback
        order: start-first
      rollback_config:
        parallelism: 1
        order: start-first
      restart_policy:
        condition: on-failure
        max_attempts: 3
      placement:
        constraints:
          - node.role == worker
          - node.labels.zone == east
      resources:
        limits:
          cpus: "2.0"
          memory: 1G
    networks:
      - overlay_net

networks:
  overlay_net:
    driver: overlay
    attachable: true

# Initialize Swarm
docker swarm init

# Deploy
docker stack deploy -c docker-compose.swarm.yml myapp

# View services
docker service ls
docker service ps myapp_web

# Rolling update
docker service update --image myapp/web:v2 myapp_web

# Rollback
docker service rollback myapp_web

Swarm Limitations

  • No native Ingress controller, only simple load balancing via Swarm routing mesh
  • No native autoscaling, requires external tools
  • No Operator ecosystem, weak advanced operations capabilities
  • Community activity continues to decline, new feature development is slow

Since 2024, Mirantis (Swarm’s commercial supporter) has transitioned Swarm maintenance to the community. While it won’t be immediately discontinued, new projects should prioritize K8s. Reference: Mirantis official statement

Migrating from Compose to K8s

Migration Assessment Checklist

Ask yourself these questions before migrating:

  1. Has the current service scale exceeded Swarm’s comfort zone (>5 nodes / >50 containers)?
  2. Do you need HPA autoscaling?
  3. Do you need multi-cluster disaster recovery?
  4. Does the team have K8s operations capability?
  5. Can you accept K8s operational complexity?

If more than 2 answers are “yes,” you should start planning the migration.

Using Kompose for Automatic Conversion

# Install Kompose
curl -L https://github.com/kubernetes/kompose/releases/download/v1.31.2/kompose-linux-amd64 -o kompose
chmod +x kompose
mv kompose /usr/local/bin/

# Convert
kompose convert -f docker-compose.yml -o k8s/

# Generated files
# k8s/web-deployment.yaml
# k8s/web-service.yaml
# k8s/postgres-deployment.yaml
# k8s/postgres-service.yaml

Kompose Conversion Limitations

Kompose handles 70% of mechanical conversion, but the following require manual work:

ComposeK8s EquivalentNotes
depends_oninit containerK8s has no native service dependency, use init container
healthchecklivenessProbe/readinessProbeDifferent syntax, manual rewrite needed
deploy.resourcesresources.limits/requestsDifferent field names
secretsSecret + volumeMountNeed to manually create Secret
networksNetworkPolicyDefault K8s network is fully open, need manual policies
deploy.replicasDeployment replicasDirect mapping
volumesPersistentVolumeClaimNeed to define StorageClass

Key Differences in Manual Migration

# Compose depends_on  →  K8s init container
# Compose healthcheck  →  K8s probe
# Compose restart     →  K8s deployment strategy
# Compose deploy.replicas  →  K8s deployment replicas
# Compose networks     →  K8s NetworkPolicy + Service

Post-Migration K8s Configuration Example

# web-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-web
  labels:
    app: myapp
    component: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      component: web
  template:
    metadata:
      labels:
        app: myapp
        component: web
    spec:
      initContainers:
      - name: wait-for-db
        image: postgres:16-alpine
        command: ['sh', '-c', 'until pg_isready -h postgres -U myuser; do sleep 2; done']
      containers:
      - name: web
        image: myapp/web:latest
        ports:
        - containerPort: 8080
        env:
        - name: DB_HOST
          value: postgres
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: myapp-secret
              key: db-password
        resources:
          requests:
            cpu: 500m
            memory: 256Mi
          limits:
            cpu: "2"
            memory: 1Gi
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

Production Operations Command Cheat Sheet

# ========== Daily Operations ==========
docker compose up -d                    # Start all services in background
docker compose up -d --build web        # Rebuild and start specific service
docker compose down                     # Stop and remove containers, networks
docker compose down -v                  # Also remove volumes (use with caution!)
docker compose restart web              # Restart specific service
docker compose stop web                 # Stop without removing
docker compose start web                # Start stopped service

# ========== Status ==========
docker compose ps                       # View service status
docker compose ps --format json         # JSON format output
docker compose top                      # View processes inside containers
docker compose logs -f web              # Follow logs
docker compose logs --since 10m web     # Last 10 minutes of logs
docker compose logs --tail 100 web      # Last 100 lines

# ========== Debugging ==========
docker compose exec web sh              # Enter container
docker compose exec postgres psql -U myuser mydb  # Connect to database
docker compose run --rm web ./migrate up  # Run one-off command

# ========== Config Validation ==========
docker compose config                   # View final merged config
docker compose config --services        # List all service names
docker compose config --volumes         # List all volumes

Common Production Incidents and Prevention

1. Disk Full

Cause: Logs without max-size/max-file, or data volumes without monitoring.

Prevention:

# Regularly check disk
df -h /var/lib/docker

# Configure alerts
# Alert when /var/lib/docker usage > 80%

2. Frequent Container Restarts

Cause: Improper health check config (start_period too short), or dependencies not ready.

Prevention:

healthcheck:
  start_period: 30s  # Allow enough startup time
  retries: 5          # Give more chances

3. Secret Leakage

Cause: .env file committed to Git, or password hardcoded in compose file.

Prevention:

# .gitignore
echo ".env*" >> .gitignore
echo "secrets/" >> .gitignore

# Check for plaintext secrets with docker compose config
docker compose config | grep -i password

4. Version Inconsistency

Cause: Different environments using different versions of compose files, configuration drift.

Prevention: All compose files under version control, validate config with docker compose config in CI/CD.

Summary

Docker Compose is perfectly viable in production—the key is getting these things right:

  1. Health checks are mandatory: Dependency management without health checks is meaningless. Set start_period generously, and make the test command do a real readiness check, not just port probing.
  2. Resource limits must be configured: A container without limits is a ticking time bomb—one memory leak can bring down the entire host.
  3. Logs must be size-limited: max-size and max-file are the baseline; global config in daemon.json as a safety net.
  4. Secrets must not be hardcoded: Docker Secrets is the simplest approach; use Vault at larger scale.
  5. Networks must be isolated: Use internal: true for backend networks—don’t take the easy way out with a single bridge network.
  6. Migration needs planning: When scale exceeds 5 nodes or advanced scheduling is needed, migrate to K8s decisively. Kompose handles 70% of mechanical work, the rest is manual.

Compose’s core value is simplicity. If your orchestration needs have grown complex enough that you need to constantly check docs to write compose files, it’s time to migrate. Tool choice should match problem scale, not chase technology trends.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

  1. Compose Specification — GitHub, referenced for Compose Specification
  2. grpc_health_probe — GitHub, referenced for grpc_health_probe
  3. Mirantis official statement — Mirantis, referenced for Mirantis official statement