Container Runtime: containerd and CRI Deep Dive

Overview

Kubernetes v1.24 officially removed dockershim, meaning Docker Engine is no longer used as the K8s container runtime. Instead, CRI-compliant (Container Runtime Interface) runtimes are used, with containerd being the de facto standard choice.

Many ops teams perceive containerd as merely a “Docker replacement,” but its architecture, image management, and CLI tools differ significantly from Docker. This article provides a deep dive into the CRI specification, containerd architecture, daily operations, and migration from Docker.

Based on containerd v2.0 and Kubernetes v1.30. Reference: containerd Official Documentation

CRI Specification

What Is CRI

CRI (Container Runtime Interface) is a container runtime interface specification defined by K8s. Instead of calling container runtimes directly, K8s communicates with CRI-compliant runtimes via gRPC:

kubelet → CRI gRPC → Container runtime (containerd/CRI-O) → Container

CRI Interface Definition

CRI defines two core services:

Service	Function	Key Methods
`RuntimeService`	Container and Pod lifecycle	`RunPodSandbox`, `CreateContainer`, `StartContainer`, `StopContainer`
`ImageService`	Image management	`ListImages`, `PullImage`, `RemoveImage`, `ImageStatus`

// Simplified CRI interface definition
service RuntimeService {
    rpc RunPodSandbox(RunPodSandboxRequest) returns (RunPodSandboxResponse);
    rpc StopPodSandbox(StopPodSandboxRequest) returns (StopPodSandboxResponse);
    rpc RemovePodSandbox(RemovePodSandboxRequest) returns (RemovePodSandboxResponse);
    rpc CreateContainer(CreateContainerRequest) returns (CreateContainerResponse);
    rpc StartContainer(StartContainerRequest) returns (StartContainerResponse);
    rpc StopContainer(StopContainerRequest) returns (StopContainerResponse);
    rpc RemoveContainer(RemoveContainerRequest) returns (RemoveContainerResponse);
    rpc ListContainers(ListContainersRequest) returns (ListContainersResponse);
    rpc ContainerStats(ContainerStatsRequest) returns (ContainerStatsResponse);
    rpc ListPodSandbox(ListPodSandboxRequest) returns (ListPodSandboxResponse);
}

service ImageService {
    rpc ListImages(ListImagesRequest) returns (ListImagesResponse);
    rpc PullImage(PullImageRequest) returns (PullImageResponse);
    rpc RemoveImage(RemoveImageRequest) returns (RemoveImageResponse);
    rpc ImageStatus(ImageStatusRequest) returns (ImageStatusResponse);
}

Why dockershim Was Removed

Docker Engine doesn’t natively support CRI. K8s used dockershim as an adapter to call Docker. This caused several issues:

Issue	Description
Extra adapter layer	dockershim added a layer of calls, reducing efficiency
Maintenance burden	K8s maintained dockershim code, but Docker updated frequently
Feature limitations	dockershim couldn’t leverage advanced CRI features
Extra processes	Docker Engine included dockerd, containerd, docker-proxy, etc.

After removing dockershim, using containerd directly eliminates the middle layer:

# Old architecture (Docker)
kubelet → dockershim → dockerd → containerd → containerd-shim → runc → Container

# New architecture (containerd)
kubelet → CRI Plugin → containerd → containerd-shim → runc → Container

Docker vs containerd vs CRI-O

Three Major Runtime Comparison

Feature	Docker Engine	containerd	CRI-O
CRI support	Needs dockershim	Native support	Native support
Architecture	Multi-process (dockerd + containerd)	Single process	Single process
Image format	Docker Image	OCI Image	OCI Image
CLI	docker	nerdctl / ctr	crictl
Image storage	Independent	Independent	Independent
Network management	docker-proxy	CNI	CNI
Build capability	docker build	nerdctl build / buildkit	buildah
Use case	Development	K8s production	OpenShift
Resource overhead	High	Low	Low

Architecture Comparison

# Docker Engine architecture
dockerd (HTTP API + image build + volume management)
  └── containerd (container lifecycle)
        └── containerd-shim (container process management)
              └── runc (OCI runtime)

# containerd architecture
containerd (CRI + container lifecycle + image management)
  └── containerd-shim (container process management)
        └── runc (OCI runtime)

# CRI-O architecture
cri-o (CRI + container lifecycle + image management)
  └── conmon (container process monitoring)
        └── runc (OCI runtime)

Resource Overhead Comparison

Metric	Docker	containerd	CRI-O
Resident memory	~150MB	~50MB	~40MB
Disk usage	~200MB	~80MB	~70MB
Startup time	~2s	~0.5s	~0.5s
Image pull speed	Baseline	10-15% faster	10-15% faster

Values are approximate; actual performance depends on node configuration and workload.

containerd Architecture

Core Components

containerd
├── Containerd API    (gRPC API for external calls)
├── Containerd Runtime
│   ├── Tasks Manager   (Container task management)
│   ├── Runtime Shim    (OCI runtime shim)
│   └── runc            (OCI low-level runtime)
├── Containerd Images
│   ├── Pull / Push     (Image pull/push)
│   ├── Image Store     (Image storage)
│   └── Content Store   (Image content storage)
├── Containerd Snapshots
│   └── Snapshotter     (Snapshot management, overlayfs/btrfs)
├── Containerd Metadata
│   └── BoltDB          (Metadata storage)
└── Containerd Events   (Event system)

containerd Configuration File

# /etc/containerd/config.toml
version = 2

root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0

[metrics]
  address = "127.0.0.1:8080"
  grpc_histogram = false

[debug]
  address = ""
  uid = 0
  gid = 0
  level = "info"

# CRI plugin configuration
[plugins."io.containerd.grpc.v1.cri"]
  # sandbox_image = "registry.k8s.io/pause:3.9"
  sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9"

  # Container runtime
  [plugins."io.containerd.grpc.v1.cri".containerd]
    default_runtime_name = "runc"
    snapshotter = "overlayfs"

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
      runtime_type = "io.containerd.runc.v2"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
        SystemdCgroup = true        # Use systemd cgroup

  # Image pull configuration
  [plugins."io.containerd.grpc.v1.cri".registry]
    config_path = "/etc/containerd/certs.d"   # Image registry config directory

  # Image GC policy
  [plugins."io.containerd.grpc.v1.cri".image_decryption]
    key_model = "node"

  # Log configuration
  [plugins."io.containerd.grpc.v1.cri".containerd.log]
    max_size = "100MB"
    max_files = 3

systemd cgroup Driver

K8s v1.26+ requires the container runtime to use the systemd cgroup driver:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

Without this configuration, mismatched cgroup drivers between kubelet and containerd causes instability.

Image Management

Image Pull Acceleration Configuration

# Method 1: Configure mirrors in config.toml (legacy, deprecated)
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
  endpoint = ["https://mirror.ccs.tencentyun.com", "https://registry-1.docker.io"]

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.k8s.io"]
  endpoint = ["https://registry.cn-hangzhou.aliyuncs.com/google_containers"]

# Method 2: hosts.toml approach (containerd 1.7+, recommended)
mkdir -p /etc/containerd/certs.d/docker.io
cat > /etc/containerd/certs.d/docker.io/hosts.toml << 'EOF'
server = "https://registry-1.docker.io"

[host."https://mirror.ccs.tencentyun.com"]
  capabilities = ["pull", "resolve"]

[host."https://docker.1panel.live"]
  capabilities = ["pull", "resolve"]
EOF

# k8s.io images
mkdir -p /etc/containerd/certs.d/registry.k8s.io
cat > /etc/containerd/certs.d/registry.k8s.io/hosts.toml << 'EOF'
server = "https://registry.k8s.io"

[host."https://registry.cn-hangzhou.aliyuncs.com/google_containers"]
  capabilities = ["pull", "resolve"]
EOF

# Private registry authentication
mkdir -p /etc/containerd/certs.d/harbor.example.com
cat > /etc/containerd/certs.d/harbor.example.com/hosts.toml << 'EOF'
server = "https://harbor.example.com"

[host."https://harbor.example.com"]
  capabilities = ["pull", "resolve"]
  ca = "/etc/containerd/certs.d/harbor.example.com/ca.crt"
  # Or skip TLS verification (not recommended for production)
  # skip_verify = true
EOF

# Point to certs.d directory in config.toml
[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

Private Registry Authentication

# Method 1: K8s Secret (used when Pod pulls images)
kubectl create secret docker-registry harbor-secret \
  --docker-server=harbor.example.com \
  --docker-username=admin \
  --docker-password=Harbor12345 \
  --docker-email=admin@example.com \
  -n production

# Reference in Pod
# imagePullSecrets:
# - name: harbor-secret

# Method 2: Node-level authentication (shared by all Pods)
# Create auth directory
mkdir -p /etc/containerd/certs.d/harbor.example.com

# Login to generate auth file
crictl login harbor.example.com -u admin -p Harbor12345

# Or write manually
cat > /var/lib/kubelet/config.json << 'EOF'
{
  "auths": {
    "harbor.example.com": {
      "auth": "YWRtaW46SGFyYm9yMTIzNDU="
    }
  }
}
EOF

Namespaces

containerd Namespaces

containerd uses namespaces to isolate containers and images from different clients:

Namespace	User	Description
`k8s.io`	kubelet	K8s-managed containers and images
`moby`	Docker	Docker Engine-managed containers (if Docker is installed)
`default`	nerdctl / ctr	Default namespace
`buildkit`	buildkitd	Images used during build

# List all namespaces
ctr namespaces list

# Operate in specified namespace
ctr -n k8s.io images list
ctr -n k8s.io containers list

Important: K8s-managed containers and images are in the k8s.io namespace. When using ctr, add -n k8s.io or you won’t see K8s resources.

CLI Tools

ctr: containerd Native CLI

ctr is containerd’s built-in CLI—most feature-complete but less user-friendly:

# Image operations
ctr -n k8s.io images list                          # List images
ctr -n k8s.io images pull docker.io/library/nginx:1.25-alpine  # Pull image
ctr -n k8s.io images push harbor.example.com/myapp:v1          # Push image
ctr -n k8s.io images delete myapp:v1                            # Delete image
ctr -n k8s.io images export app.tar myapp:v1                    # Export image
ctr -n k8s.io images import app.tar                             # Import image

# Container operations
ctr -n k8s.io containers list                       # List containers
ctr -n k8s.io tasks list                            # List running tasks

crictl: K8s Container Debugging Tool

crictl is a CRI debugging tool maintained by the K8s community, aligned with K8s concepts:

# View Pods
crictl pods

# View containers
crictl ps
crictl ps -a                          # Including stopped
crictl ps --name nginx                # Filter by name

# View images
crictl images
crictl images --digests               # Show digests

# View container logs
crictl logs <container-id>
crictl logs -f <container-id>         # Follow logs

# Execute command in container
crictl exec -it <container-id> sh

# View container status
crictl inspect <container-id>
crictl stats

# Pull image
crictl pull nginx:1.25-alpine

# /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false

nerdctl: Docker-Compatible CLI

nerdctl is a Docker-compatible CLI developed by the containerd community, recommended as a Docker CLI replacement:

# Basic docker-compatible commands
nerdctl ps                             # List containers
nerdctl images                         # List images
nerdctl pull nginx:1.25-alpine         # Pull image
nerdctl run -d --name nginx -p 80:80 nginx:1.25-alpine  # Run container
nerdctl exec -it nginx sh              # Enter container
nerdctl logs -f nginx                  # View logs
nerdctl stop nginx                     # Stop container
nerdctl rm nginx                       # Remove container
nerdctl rmi nginx:1.25-alpine          # Remove image

# Build images (requires buildkitd)
nerdctl build -t myapp:v1 .
nerdctl build -t myapp:v1 --platform linux/amd64,linux/arm64 .

# compose support
nerdctl compose up -d
nerdctl compose down

# Specify namespace
nerdctl --namespace=k8s.io ps
nerdctl --namespace=default ps

Tool Comparison

Feature	ctr	crictl	nerdctl
Image management	Yes	Yes	Yes
Container management	Yes	Read-only	Yes
Container creation	Yes	No	Yes
Image building	No	No	Yes
compose	No	No	Yes
Log viewing	No	Yes	Yes
exec	No	Yes	Yes
K8s aligned	No	Yes	No
Docker compatible	No	No	Yes

Recommendation: Use crictl for daily ops (aligned with K8s concepts), nerdctl as Docker CLI replacement, ctr for low-level debugging.

Snapshotter

Role of the Snapshotter

The snapshotter manages the layering of container filesystems. Each image layer pulled creates a snapshot; running a container creates a read-write layer on top.

Supported Snapshotters

Snapshotter	Description	Use Case
`overlayfs`	Default, based on OverlayFS	General (recommended)
`btrfs`	Based on Btrfs filesystem	Requires Btrfs support
`zfs`	Based on ZFS filesystem	Requires ZFS support
`native`	Simple file copy	Test environments
`stargz`	Lazy-loading images	Large image fast startup
`nydus`	Image acceleration (Ant Group open source)	Large image fast startup

overlayfs Configuration

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"

# View overlayfs mounts
mount | grep overlay

# View image layers
ctr -n k8s.io snapshots list

Stargz / Nydus Lazy Loading

Traditional image pulling downloads the entire image—large images (GB-level) pull slowly. Stargz and Nydus solve this with lazy loading:

# Enable stargz snapshotter
[plugins."io.containerd.snapshotter.v1.stargz"]
  root_path = "/var/lib/containerd-stargz-grpc"

# Traditional image pull: full download → start container (slow)
# Lazy loading: metadata download → start container → download on demand (fast)

Performance Comparison

Image Pull Performance

# Test conditions: 1GB image, 100Mbps internal network
Docker:        ~85s  (includes dockerd scheduling overhead)
containerd:    ~72s  (direct CRI call)
CRI-O:         ~73s  (direct CRI call)
containerd + stargz: ~15s (lazy loading, on-demand pull)

Container Startup Performance

# Test conditions: nginx:alpine, 100 concurrent containers
Docker:        ~8s
containerd:    ~5s
CRI-O:         ~5s

Memory Usage

# Idle state
Docker (dockerd + containerd): ~150MB
containerd:                     ~50MB
CRI-O:                          ~40MB

Migrating from Docker to containerd

Migration Assessment

Check Item	Description
Image building	Docker Build replaced by BuildKit / nerdctl build
CLI tools	docker commands replaced by crictl / nerdctl
Image registry	Authentication configuration differs
Monitoring scripts	Scripts depending on docker CLI need updating
Log driver	containerd doesn’t support Docker’s log drivers
docker.sock	Applications depending on Docker socket need modification

Migration Steps

# 1. Install containerd
apt-get update && apt-get install -y containerd

# 2. Generate default config
containerd config default > /etc/containerd/config.toml

# 3. Modify config (SystemdCgroup + image acceleration)
# Edit /etc/containerd/config.toml, ensure:
# SystemdCgroup = true
# config_path = "/etc/containerd/certs.d"

# 4. Configure image acceleration
mkdir -p /etc/containerd/certs.d/docker.io
cat > /etc/containerd/certs.d/docker.io/hosts.toml << 'EOF'
server = "https://registry-1.docker.io"
[host."https://mirror.ccs.tencentyun.com"]
  capabilities = ["pull", "resolve"]
EOF

# 5. Restart containerd
systemctl restart containerd
systemctl enable containerd

# 6. Modify kubelet config
# /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock"

# 7. Restart kubelet
systemctl restart kubelet

# 8. Verify
kubectl get nodes -o wide
# Verify CONTAINER-RUNTIME column shows containerd://

Image Migration

# Export Docker image
docker save -o myapp.tar myapp:v1

# Import to containerd
ctr -n k8s.io images import myapp.tar

# Or use nerdctl
nerdctl -n k8s.io load -i myapp.tar

Script Migration

# Docker command → containerd equivalent mapping

# List containers
docker ps                    → crictl ps
# List images
docker images                → crictl images
# Pull image
docker pull nginx            → crictl pull nginx
# View logs
docker logs <id>             → crictl logs <id>
# Enter container
docker exec -it <id> sh      → crictl exec -it <id> sh
# Inspect container
docker inspect <id>          → crictl inspect <id>
# Container stats
docker stats                 → crictl stats
# Build image
docker build -t app .        → nerdctl build -t app .
# Run container
docker run -d nginx          → nerdctl run -d nginx
# Remove image
docker rmi nginx             → crictl rmi nginx

Daily Operations

Image GC Configuration

# K8s image GC is managed by kubelet, not containerd config
# /var/lib/kubelet/config.yaml
imageGCHighThresholdPercent: 85    # Trigger GC when disk usage exceeds 85%
imageGCLowThresholdPercent: 80     # Stop GC at 80%
imageMinimumGCAge: 2m              # Image must exist at least 2min before GC

Log Management

containerd doesn’t have Docker’s rich log drivers. Container logs are managed by kubelet, output to /var/log/pods/:

# View container logs
crictl logs <container-id>

# K8s Pod log path
ls /var/log/pods/<namespace>_<pod-name>_<pod-uid>/

# Log rotation config (kubelet)
# /var/lib/kubelet/config.yaml
containerLogMaxSize: "100Mi"
containerLogMaxFiles: 5

Performance Tuning

# containerd performance tuning
version = 2

# Increase gRPC concurrency
[grpc]
  max_concurrent_streams = 1000

# Adjust image pull concurrency
[plugins."io.containerd.grpc.v1.cri"]
  max_concurrent_downloads = 5       # Concurrent image downloads (default 3)

# overlayfs performance
[plugins."io.containerd.snapshotter.v1.overlayfs"]
  upperdir_capability = true

Troubleshooting

# Check containerd status
systemctl status containerd

# View containerd logs
journalctl -u containerd -f

# Check CRI status
crictl info

# View containerd metrics
curl -s http://localhost:8080/metrics | grep containerd

# Common issues
# 1. Pod creation failure - ImagePullBackOff
crictl images | grep <image>       # Check if image exists
ctr -n k8s.io images pull <image>  # Manual pull test

# 2. Container startup failure
crictl inspect <container-id>      # View container details
journalctl -u containerd | grep <container-id>  # View logs

# 3. Slow image pull
cat /etc/containerd/certs.d/*/hosts.toml  # Check image acceleration config

Summary

containerd has become the de facto standard container runtime for K8s. Key takeaways:

CRI is the specification, containerd is the implementation: Understanding CRI helps understand how K8s interacts with runtimes.
Three CLIs serve different purposes: crictl for K8s ops (primarily read-only), nerdctl as Docker CLI replacement, ctr for low-level debugging.
Image acceleration must be configured: In environments with restricted access to Docker Hub, configure image acceleration using the certs.d directory approach (hosts.toml); the legacy configuration method is deprecated.
SystemdCgroup must be enabled: K8s requires consistent cgroup drivers; without it, instability occurs.
Namespace isolation: K8s containers and images are in the k8s.io namespace; add -n k8s.io when operating.
Migration needs planning: Migrating from Docker to containerd requires assessing applications depending on Docker socket, monitoring scripts, etc.—don’t switch directly.
Use lazy loading for large images: Stargz / Nydus can significantly accelerate large image startup, suitable for CI/CD scenarios.

containerd’s design is simpler and more efficient than Docker, but it also means some Docker conveniences (like docker build, rich log drivers) require alternative solutions. Understanding these differences is key to smooth migration.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

containerd Official Documentation — Containerd, referenced for containerd Official Documentation

Overview#

CRI Specification#

What Is CRI#

CRI Interface Definition#

Why dockershim Was Removed#

Docker vs containerd vs CRI-O#

Three Major Runtime Comparison#

Architecture Comparison#

Resource Overhead Comparison#

containerd Architecture#

Core Components#

containerd Configuration File#

systemd cgroup Driver#

Image Management#

Image Pull Acceleration Configuration#

Private Registry Authentication#

Namespaces#

containerd Namespaces#

CLI Tools#

ctr: containerd Native CLI#

crictl: K8s Container Debugging Tool#

nerdctl: Docker-Compatible CLI#

Tool Comparison#

Snapshotter#

Role of the Snapshotter#

Supported Snapshotters#

overlayfs Configuration#

Stargz / Nydus Lazy Loading#

Performance Comparison#

Image Pull Performance#

Container Startup Performance#

Memory Usage#

Migrating from Docker to containerd#

Migration Assessment#

Migration Steps#

Image Migration#

Script Migration#

Daily Operations#

Image GC Configuration#

Log Management#

Performance Tuning#

Troubleshooting#

Summary#

References & Acknowledgments#