Zero-Failure K8s Migration of 120 Microservices: 6 Pitfalls Nobody Talks About from a 3-Month Real Migration

Overview

Migrating 120+ microservices from traditional VM deployments to Kubernetes sounds like a standard project: write Dockerfiles, build a cluster, migrate services, shift traffic. In planning meetings, everyone thinks three months is enough. In practice, you discover that the real challenges aren’t about “how to get services running” — that’s just helm install — but about pitfalls nobody warned you about: circular dependencies in service graphs, in-memory state loss during stateful migration, CNI performance cliffs at scale, DNS caching traps during canary traffic shifts, ConfigMap bloat, and monitoring metric gaps after migration.

This article is a postmortem of my experience leading the K8s full migration of 120+ microservices at a new-energy logistics platform. Three months, zero incidents (where “zero” means no P0/P1-level business disruptions during migration, not zero errors), autoscaling from 25 minutes down to 1 minute, and deployment efficiency up 50%. But the process was far less smooth than the results suggest — dependency mapping alone took three weeks, CNI performance troubleshooting ate a week, and intermittent 502s from DNS caching during canary shifting kept the team up for two nights.

My goal isn’t to teach K8s basics. It’s to break down the 6 pitfalls that actually stalled us, each with architecture decision processes and alternative comparisons. If you’re planning a large-scale microservice K8s migration, these lessons will save you from repeating our mistakes.

Pitfall 1: Service Dependency Mapping — What Looks Linear Is Actually a Cyclic Graph

The Problem: Dependency Detection Found Dead Cycles Halfway Through

The first migration step was mapping service dependencies to determine migration order. Standard approach: find services with no downstream dependencies, migrate them first, then work up the layers. We grepped config files for database connections and API call endpoints, drew a dependency topology graph.

Week one went smoothly. Week two, problems surfaced — config-file dependencies were just the tip of the iceberg. Runtime dynamic dependencies, calls routed through service discovery (Eureka/Nacos), and asynchronous chains triggered by message queues weren’t in static configs. After adding all dependencies, we found three circular references in the graph: A calls B, B calls C, C calls back to A.

This isn’t surprising. After two or three years of microservice architecture evolution, circular dependencies are almost inevitable. But during migration, they’re a fatal problem: you can’t migrate in batches. A, B, and C must migrate simultaneously, otherwise cross-platform calls (K8s services calling non-K8s services) introduce network latency and DNS resolution overhead that become performance bottlenecks.

The Solution: Runtime Call Chain Analysis + Batch Strategy Adjustment

Static config inspection isn’t enough. We ultimately used two toolsets together:

Toolset 1: Jaeger Distributed Tracing

One month before migration, we instrumented all services with OpenTelemetry SDK and collected full call chains through Jaeger. After running for a week, we obtained the real runtime service call graph. Dependencies absent from static configs were fully exposed — 14 asynchronous call chains triggered by message queues alone, 3 of which were circular dependencies.

Toolset 2: Custom Go Script for Topology Analysis

Jaeger’s call chains are request-granular and need aggregation for service-level topology. I wrote a Go script pulling Jaeger’s API, aggregating call relationships by service, and outputting a dependency matrix with topological sort results:

// Core logic for dependency topology analysis
type ServiceGraph struct {
    Nodes map[string]*ServiceNode     // service name -> node
    Edges map[string][]string         // service name -> downstream dependencies
}

// Detect circular dependencies (Tarjan's algorithm for strongly connected components)
func (g *ServiceGraph) FindCycles() [][]string {
    index := 0
    stack := []string{}
    onStack := make(map[string]bool)
    sccs := [][]string{}
    indices := make(map[string]int)
    lowlinks := make(map[string]int)

    var strongconnect func(v string)
    strongconnect = func(v string) {
        indices[v] = index
        lowlinks[v] = index
        index++
        stack = append(stack, v)
        onStack[v] = true

        for _, w := range g.Edges[v] {
            if _, ok := indices[w]; !ok {
                strongconnect(w)
                lowlinks[v] = min(lowlinks[v], lowlinks[w])
            } else if onStack[w] {
                lowlinks[v] = min(lowlinks[v], indices[w])
            }
        }

        if lowlinks[v] == indices[v] {
            scc := []string{}
            for {
                w := stack[len(stack)-1]
                stack = stack[:len(stack)-1]
                onStack[w] = false
                scc = append(scc, w)
                if w == v {
                    break
                }
            }
            if len(scc) > 1 {
                sccs = append(sccs, scc)
            }
        }
    }

    for v := range g.Nodes {
        if _, ok := indices[v]; !ok {
            strongconnect(v)
        }
    }
    return sccs
}

The core idea: use Tarjan’s algorithm to find strongly connected components (SCCs). An SCC containing multiple services indicates circular dependencies — those services must migrate as a unit.

After analysis, 120 services were regrouped:

Architecture Trade-off: Why Not Migrate Everything at Once?

Some argue: if there are circular dependencies, just migrate everything together. Technically feasible, but I don’t recommend it. Three reasons:

First, rollback cost. Migrating 120 services at once means rolling back everything if something goes wrong. The benefit of canary migration is that issues only require rolling back the affected services, keeping business impact controllable.

Second, validation window. Earlier-migrated services need to run stably on the new platform before serving as dependency targets for later services. If everything migrates simultaneously, all services call each other in an unstable new environment, making troubleshooting exponentially harder. In our migration, the first 34 services ran for 5 days after migration with stable metrics, logs, and alerts confirmed before starting batch two.

Third, team bandwidth. Migration requires extensive manual verification and traffic validation. Migrating 120 at once drowns the team in verification work, and critical anomaly signals get buried in noise.

My recommendation: Circular dependency groups migrate together, linear chains in batches, stateful services separately. Use topological sort for batch ordering, with each SCC as an indivisible migration unit.

Internal link: Service dependency mapping is closely tied to microservice governance. If you haven’t implemented rate limiting and circuit breaking, do it before migration. See Related: Microservice Rate Limiting, Circuit Breaking, and Degradation — during migration, rate limiting and circuit breaking are your last line of defense.

Pitfall 2: In-Memory State Loss in Stateful Services — Not Just a Database Problem

The Problem: Redis Cache Hit Rate Crashed After Migration

When discussing stateful service migration, most people think of databases. Database migration strategies are well-established — primary-replica sync, dual-write, verification, cutover. What really stung us was Redis.

Not that Redis data was lost, but Redis Cluster topology information was lost. In the VM era, we used self-hosted Redis primary-replica (not Cluster mode), with business code directly connecting to Sentinel for failover. During migration, we upgraded to Redis Cluster deployed via StatefulSet. Data was restored from RDB files — looked fine.

Day one after go-live: cache hit rate crashed from 92% to 31%. The reason was simple: Redis Cluster clients need to maintain a slot map, and our old Sentinel-based clients didn’t speak the Cluster protocol. We updated client configs but missed one edge service — it used its own Redis client library that didn’t support Cluster protocol, routing all requests to the first node.

The Deeper Problem: In-Process Caches

Redis issues were easy to find — logs were full of MOVED exceptions. The truly insidious problem was in-process caches.

Several Java services used Caffeine for local caching of user permissions and routing policies. These caches auto-rebuilt on process restart, seemingly no migration issue. But after cutover, these three services saw P99 latency spike from 50ms to 800ms.

After hours of investigation, we discovered: the cache rebuild logic depended on another service’s startup order. In the VM era, deployment scripts guaranteed config service started first, cache service second, so caches loaded correctly. But K8s Pod scheduling doesn’t guarantee startup order (even with init containers), and network readiness timing varies. The config service Pod started first but wasn’t ready (readiness probe hadn’t passed), so the cache service started pulling configs, got empty values, and cached those empties.

The Solution: Three-Layer Stateful Service Handling

Stateful service migration needs three layers, each with completely different strategies:

Layer 1: Persistent Data (Databases, Message Queues)

Layer 2: In-Memory State (In-Process Caches, Session State)

In-process caches are the most easily overlooked during migration. My recommendation: run a cache dependency audit before migration. For each service, check three questions:

1. Does it have in-process caches? (Search for Caffeine, Guava Cache, Ehcache, go-cache keywords)
2. What’s the cache data source? (Config service, database, other APIs?)
3. Does cache rebuild depend on startup order?

If the third answer is “yes,” you must handle startup ordering in the migration plan. In K8s, don’t rely on deployment scripts for startup order — the correct approach is adding retry and degradation logic in the application layer. If config is unavailable, start with defaults and rebuild cache asynchronously in the background.

# The "correct" way to handle K8s startup ordering isn't initContainer
# but application-layer fault tolerance
# Here's the readinessProbe configuration strategy:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cache-service
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    spec:
      containers:
      - name: app
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
          failureThreshold: 3
        # readinessProbe only returns 200 when cache loading is complete
        # Before that, requests return 503 and Service won't route traffic
        # This way, even if a Pod starts, it won't receive traffic until cache is ready

Key insight: readinessProbe isn’t just “is the process alive” — it’s “is the service ready.” Our readiness endpoint checks whether the cache has finished loading — returns 503 if not, and K8s Service won’t route traffic to that Pod. This is far more reliable than depending on startup order.

Layer 3: Filesystem State (Local Files, Temp Files)

Some services write temp files, lock files, PID files to local disk. After migrating to K8s, the container filesystem is ephemeral — Pod recreation wipes files. These services need to be refactored to use PVCs (PersistentVolumeClaims) or move file storage to object storage.

We wrote a Go script to batch-scan all service deployment scripts for local file writes:

// Scan deployment scripts for local file write paths
func FindLocalFileWrites(scriptsDir string) []string {
    var results []string
    // Common file write patterns
    patterns := []string{
        `/tmp/`, `/var/log/`, `/data/`, `/opt/`,
        `nohup.*\.log`, `pid`, `\.lock`, `\.pid`,
    }
    files, _ := filepath.Glob(filepath.Join(scriptsDir, "**/*.sh"))
    for _, f := range files {
        content, _ := os.ReadFile(f)
        for _, p := range patterns {
            re := regexp.MustCompile(p)
            if re.Match(content) {
                results = append(results, fmt.Sprintf("%s: matched %s", f, p))
            }
        }
    }
    return results
}

The scan found 17 services writing local files: 12 were log files (solved with stdout + log collection, see Related: K8s Log Collection Strategy), 3 were lock files (switched to Redis distributed locks), 2 were temp upload files (switched to PVC).

Pitfall 3: CNI Performance Cliff — Hidden Bottleneck at Scale

The Problem: Periodic Latency Spikes After 80+ Nodes

Everything was fine at 30 nodes. After the second batch pushed node count to 80, periodic latency spikes appeared. Not all requests were slow — every few minutes, a burst of 200ms+ slow requests appeared, while normal latency was 2-5ms.

Monitoring showed slow requests concentrated on cross-node communication. Same-node Pod-to-Pod communication was perfectly normal. This pattern is classic — CNI plugins at a certain node count threshold see routing table and iptables rule explosion, causing kernel network stack processing delays.

We used Calico (BGP mode). At 80 nodes, each node’s routing table had 80 BGP routes, plus iptables rules from NetworkPolicy — single-node iptables rules exceeded 2,000. Not a problem at 30 nodes, but at 80, iptables linear matching performance became a bottleneck.

Diagnosis Process

Troubleshooting took three days. First excluded application-layer issues (latency distribution wasn’t from application logic), then tcpdump confirmed network-layer delay, finally perf top showed kernel-mode ipt_do_table function consuming 15% CPU — that’s the iptables rule matching overhead.

# Three-step CNI performance diagnosis

# Step 1: Confirm network delay is in kernel mode
perf top -g
# If ipt_do_table or nf_hook_slow > 5%, iptables is the bottleneck

# Step 2: Count iptables rules
iptables -L -n --line-numbers | wc -l
# 80-node Calico cluster, ~2000-3000 rules per node

# Step 3: Measure cross-node Pod communication latency
# Inside a Pod:
kubectl exec -it <pod-name> -- bash -c '
  for i in $(seq 1 100); do
    ping -c 1 <target-pod-ip> | grep "time=" | awk -F"time=" "{print \$2}"
  done
'
# High latency variance (min 1ms, max 200ms) indicates CNI rule matching performance jitter

Architecture Decision: Calico vs Cilium vs Flannel

Two paths: switch CNI or optimize the existing one. We did a detailed comparison:

My recommendation:

• Under 50 nodes: Calico is sufficient, BGP mode performs well, NetworkPolicy is complete
• 50-200 nodes: Cilium — eBPF bypasses iptables, clear advantage at scale
• No NetworkPolicy needed: Flannel is simplest, but I don’t recommend bare Flannel for production

We chose Cilium. Switching CNI is risky, but we validated for two weeks on a test cluster first, including fault injection, load testing, and NetworkPolicy testing. The migration used one weekend maintenance window: pre-configure Cilium CNI on all nodes → drain nodes one by one + reschedule Pods → verify cross-node communication → full confirmation.

Post-switch results were dramatic:

Before Cilium (80 nodes, Calico):
  Cross-node Pod latency P50: 3ms  P99: 180ms  Max: 320ms
  iptables rules: 2,347
  Kernel network processing CPU: 15%

After Cilium (80 nodes, Cilium eBPF):
  Cross-node Pod latency P50: 1.5ms  P99: 8ms  Max: 15ms
  eBPF programs: 47
  Kernel network processing CPU: 3%

P99 from 180ms to 8ms — a qualitative difference for the business. User-perceived “stuttering” disappeared.

Bonus: Cilium’s Unexpected Benefit

Switching to Cilium also brought Hubble observability. Hubble uses eBPF for Sidecar-free network traffic monitoring — you can see Pod-to-Pod call relationships and latency distributions directly. This capability proved invaluable during NetworkPolicy troubleshooting later in the migration.

If you’re planning to switch CNI during migration, my advice: choose your target CNI before migration starts. Don’t migrate first then switch. Mid-migration CNI changes mean re-assigning all Pod IPs — an additional full-cluster risk.

Pitfall 4: DNS Caching Trap During Canary Traffic Shifting

The Problem: Intermittent 502s After Shifting 10% Traffic

At batch four, we started canary traffic shifting. Standard approach: Ingress weight-based routing — 10% to K8s, 90% to VMs. Observed 30 minutes, no issues, increased to 30%. Then 502s started appearing intermittently.

Not persistent 502s — small bursts every few minutes lasting a few seconds, then recovery. Ingress logs showed upstream connection timeouts. But K8s services were perfectly fine — Pods in Running state, readiness probes passing.

Two hours of investigation finally traced it to JVM DNS caching. Java caches DNS resolution results for 30 seconds by default (controlled by networkaddress.cache.ttl). When Ingress routed requests to the K8s cluster, Java apps had cached IPs from the VM-era old services. K8s Service ClusterIPs differ from old VM IPs, so requests went to decommissioned old IPs.

This issue was insidious because it only appeared during the traffic-shifting window. Once traffic fully shifted and old IPs were decommissioned, DNS cache entries expired and the problem vanished. But during the transition, it kept generating intermittent failures.

The Solution: Three-Layer DNS Cache Governance

Layer 1: Application-Layer DNS TTL Adjustment

// Java DNS cache config (add to JVM startup args)
-Dnetworkaddress.cache.ttl=1
// Or set in code
java.security.Security.setProperty("networkaddress.cache.ttl", "1");

Reducing DNS cache to 1 second. This adds minimal latency (< 1ms per request) but ensures fresh DNS records during canary shifting.

Layer 2: CoreDNS Configuration Optimization

# CoreDNS ConfigMap optimization
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        # Cache: 1000 entries, 30-second TTL
        cache {
            1000
            30
        }
        # Negative cache: NXDOMAIN cached for 5 seconds to prevent DNS storms
        cache {
            denyleft 5
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        loop
        reload
        loadbalance
    }    

Two key points: positive cache 30 seconds avoids upstream queries for every request; negative cache 5 seconds prevents NXDOMAIN responses from overwhelming CoreDNS.

Layer 3: Dual-IP Strategy During Transition

During canary shifting, DNS records return both old and new IPs simultaneously. Client load balancers hold both IP sets and distribute traffic per Ingress weights:

# DNS record during canary shifting
# VM LB IP: 10.0.1.100
# K8s Ingress LB IP: 10.0.2.200
# DNS returns two A records, client LB distributes by weight

api.example.com.  30  IN  A  10.0.1.100   # Old environment (weight 90%)
api.example.com.  30  IN  A  10.0.2.200   # K8s cluster (weight 10%)

This requires client support for DNS round-robin with connection-level load balancing. If clients only take the first DNS IP (many older HTTP clients do this), use weighted DNS rotation — CoreDNS’s loadbalance plugin rotates IP order in responses, enabling simple weighted traffic distribution with client DNS round-robin.

Pitfall 5: ConfigMap Bloat — Environment Variable Explosion

The Problem: One Service’s ConfigMap Grew to 300 Lines

In the VM era, config management used Spring Cloud Config (Java services) and Consul KV (Go services). After K8s migration, the team decided to standardize on ConfigMap + Secret.

During migration, one Java service’s ConfigMap grew from 40 lines to 300. The reason: placeholders in config files (${DB_HOST}, ${REDIS_PORT}) previously injected by deployment scripts now needed to go into ConfigMaps. Plus environment differences (dev/staging/prod) — the same service needed 3 ConfigMaps, each 300 lines.

300-line ConfigMaps are problematic: high change risk (reviewing the entire file for one line change), difficult version management (Git diff can’t show which config item changed), and slow deployments (ConfigMap update requires restarting all using Pods).

The Solution: Layered Configuration + Runtime Injection

Layered strategy: Split configs into three layers managed separately.

Runtime injection: Don’t hardcode all configs in ConfigMaps — use envFrom + configMapKeyRef for on-demand injection:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  template:
    spec:
      containers:
      - name: app
        envFrom:
        # Layer 1: Common config, shared across all environments
        - configMapRef:
            name: common-config
        # Layer 2: Environment-specific config
        - configMapRef:
            name: order-service-prod
        # Layer 3: Sensitive config
        - secretRef:
            name: order-service-secrets
        env:
        # Layer 4: Pod-level dynamic config via Downward API
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        # Layer 5: Runtime config center
        # App pulls dynamic config at startup from config center
        # Changes pushed via long connection, no Pod restart needed
        - name: CONFIG_CENTER_URL
          value: "http://nacos.infra:8848"

Benefit: common config maintained once, environment-specific config only contains per-env differences. ConfigMaps shrink from 300 lines to 60-80.

Hot config reload: Some configs don’t need Pod restarts to take effect (log level, rate limit threshold). These shouldn’t go in ConfigMaps — ConfigMap updates require Pod restarts. Use a config center (Nacos/Apollo) with Watch mechanism:

// Go service hot config reload example
func WatchConfig(nacosClient *nacos.Client, dataId string) {
    go func() {
        for {
            config, err := nacosClient.GetConfig(dataId)
            if err != nil {
                log.Printf("Failed to get config: %v", err)
                time.Sleep(5 * time.Second)
                continue
            }
            // Atomic update of runtime config
            if err := UpdateRuntimeConfig(config); err != nil {
                log.Printf("Config update failed: %v", err)
            }
            // Block until next change notification
            nacosClient.ListenConfig(dataId)
        }
    }()
}

Bonus: K8s ConfigMap Update Trap

K8s ConfigMap updates aren’t automatically picked up by using Pods. Two options:

1. Restart Pods: kubectl rollout restart deployment/<name> — simple but interrupts long connections
2. Sidecar container watching for changes: Add a sidecar that watches ConfigMap changes and signals the main container to reload

I recommend the second, but only for frequently-changing configs. Most services have low config change frequency (weekly) — just restart, no need for sidecar complexity.

Pitfall 6: Monitoring Metric Migration Gap

The Problem: All Alert Rules Silently Failed, Nobody Noticed

This pitfall was discovered after the fact. Before migration, all services had Prometheus + Grafana monitoring with 200+ alert rules. After K8s migration, service IPs changed (VM IPs to Pod IPs), and IP-based alert rules all failed — Prometheus scraped new IPs, but alert rules still matched old IPs.

More insidiously, label gaps. K8s Prometheus auto-labels are namespace, pod, container, while VM-era labels were instance, host, service. Many PromQL queries and Grafana dashboards filtered by old labels — after migration, queries returned empty results, dashboards went blank.

But nobody noticed. During migration, everyone focused on traffic and latency, not monitoring dashboards. A week after migration, several services’ CPU and memory monitoring had been broken for three days, and no alerts fired.

The Solution: Label Mapping + Alert Rule Migration

Step 1: Label Mapping

Added mapping rules in Prometheus relabel config, mapping K8s labels to the old label scheme:

# Prometheus scrape config with relabel
scrape_configs:
- job_name: 'k8s-pods'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  # Map namespace -> service label (by namespace)
  - source_labels: [__meta_kubernetes_namespace]
    target_label: service
    regex: '(.+)'
    replacement: '${1}'
  # Map pod name -> instance label
  - source_labels: [__meta_kubernetes_pod_name]
    target_label: instance
  # Map node name -> host label
  - source_labels: [__meta_kubernetes_pod_node_name]
    target_label: host
  # Extract service name from pod annotation
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_service_name]
    target_label: service
  # Preserve container name
  - source_labels: [__meta_kubernetes_pod_container_name]
    target_label: container

Benefit: old PromQL queries and Grafana dashboards work unchanged — new data automatically matches old labels. During migration, you can see both old and new environment data for comparison validation.

Step 2: Alert Rule Migration

Alert rules can’t be simply copied. K8s metric collection methods differ — some rules need adjustment. For example, CPU usage in the VM era came from node_exporter, while K8s uses cAdvisor or kube-state-metrics — metric names and labels differ.

I created an alert rule migration mapping table:

Step 3: Dual-Environment Parallel Monitoring

During migration, both old and new environments are monitored in parallel, distinguished by labels. A verification script compares metrics between environments:

#!/bin/bash
# Metric consistency check during migration
# Compare the same service's metrics across old and new environments

OLD_ENV="instance=~\"10\\.0\\.1\\.\\d+\""
NEW_ENV="namespace=\"production\""

METRICS=(
    "http_request_duration_seconds_p99"
    "http_requests_total"
    "process_cpu_seconds_total"
    "process_resident_memory_bytes"
)

for metric in "${METRICS[@]}"; do
    echo "=== ${metric} ==="
    echo "Old env: $(curl -s "http://prometheus:9090/api/v1/query?query=${metric}{${OLD_ENV}}" | jq '.data.result[0].value[1]')"
    echo "New env: $(curl -s "http://prometheus:9090/api/v1/query?query=${metric}{${NEW_ENV}}" | jq '.data.result[0].value[1]')"
    echo ""
done

Internal link: Monitoring system migration and alert strategy design are strongly linked. If you’re doing alert rule migration, see Related: Alert Strategy Design — From Noise to Signal — use migration as an opportunity to clean up junk alerts.

Pitfall 7: Capacity Planning — You Thought You Got It Right, But You Were Off 40%

The Problem: Cluster Ran Out of Resources Three Days After Migration

Pre-migration capacity planning: mapped VM-era CPU and memory requests directly to K8s requests and limits. 120 services, 200 CPU cores total, 480GB memory. Planned 8 nodes at 32 cores 64GB each — 256 cores 512GB total, seemingly 20% headroom.

Three days after migration, cluster utilization hit 85%. HPA scaled frequently, node pools maxed out. Why?

We missed three things:

First, K8s’s own resource overhead. kubelet, kube-proxy, CNI plugins, CoreDNS, log collection agents — these system components consume 3-5 CPU cores and 2-4GB memory per node. 8 nodes = 24-40 cores, 16-32GB system overhead — directly eating the “headroom.”

Second, requests vs limits difference causing fragmentation. We set requests=2 cores, limits=4 cores per service. K8s scheduler allocates by requests, but containers may use up to limits at runtime. A node’s requests may be “full” (32 cores allocated), but actual CPU usage only 50%. New Pods can’t schedule — because the scheduler looks at requests sum, not actual usage.

Third, high Pod density causing I/O contention. We packed 20-25 Pods per node — CPU and memory sufficient, but disk I/O became the bottleneck. A dozen Java services simultaneously doing GC log writes and JVM heap dumps saturated disk IOPS, slowing application log writes and cascading into request processing latency.

The Solution: Three-Layer Capacity Model

Layer 1: Node-Level Capacity Planning

# Calculate node-level actually available resources
NODE_CPU=32
NODE_MEM=64  # GB

# System reserve (kubelet itself)
SYS_RESERVE_CPU=2
SYS_RESERVE_MEM=2  # GB

# kube reserve (OS daemon processes)
KUBE_RESERVE_CPU=1
KUBE_RESERVE_MEM=1  # GB

# Eviction threshold (reserved for disk pressure)
EVICT_MEM=0.5  # GB

# Actually allocatable to Pods
AVAILABLE_CPU=$((NODE_CPU - SYS_RESERVE_CPU - KUBE_RESERVE_CPU))
AVAILABLE_MEM=$((NODE_MEM - SYS_RESERVE_MEM - KUBE_RESERVE_MEM - EVICT_MEM))

echo "Node allocatable: ${AVAILABLE_CPU} cores, ${AVAILABLE_MEM}GB"
# Output: Node allocatable: 29 cores, 60.5GB

Layer 2: Pod Density Control

Not all services should run together. I/O-intensive services (databases, log aggregators) and CPU-intensive services (API gateways, compute services) need separation. We used node taints and tolerations for isolation:

# Compute-intensive node pool
apiVersion: v1
kind: Node
metadata:
  name: compute-node-1
  labels:
    node-type: compute
spec:
  taints:
  - key: workload
    value: compute
    effect: NoSchedule
---
# I/O-intensive node pool (with local SSD)
apiVersion: v1
kind: Node
metadata:
  name: io-node-1
  labels:
    node-type: io-intensive
spec:
  taints:
  - key: workload
    value: io-intensive
    effect: NoSchedule

Pods select target node pools via toleration, preventing different workload types from interfering. After workload-based pool separation, I/O contention completely disappeared — Java service GC log write latency dropped from 50ms to 2ms.

Layer 3: Capacity Monitoring and Forecasting

Planning isn’t enough — continuous resource usage trend monitoring during migration. A Go script collects daily cluster utilization and predicts when scaling is needed:

// Cluster capacity trend prediction
type ClusterCapacity struct {
    TotalCPU     float64
    TotalMem     float64
    UsedCPU      float64
    UsedMem      float64
    PodCount     int
    Timestamp    time.Time
}

func (c *ClusterCapacity) Utilization() (cpuPct, memPct float64) {
    if c.TotalCPU > 0 {
        cpuPct = c.UsedCPU / c.TotalCPU * 100
    }
    if c.TotalMem > 0 {
        memPct = c.UsedMem / c.TotalMem * 100
    }
    return
}

// Predict resource exhaustion based on 7-day trend
func PredictExhaustion(history []ClusterCapacity) (cpuDays, memDays int) {
    if len(history) < 2 {
        return -1, -1
    }
    first := history[0]
    last := history[len(history)-1]
    days := last.Timestamp.Sub(first.Timestamp).Hours() / 24

    cpuGrowthRate := (last.UsedCPU - first.UsedCPU) / days
    memGrowthRate := (last.UsedMem - first.UsedMem) / days

    if cpuGrowthRate > 0 {
        remainingCPU := (last.TotalCPU * 0.8) - last.UsedCPU // 80% threshold
        cpuDays = int(remainingCPU / cpuGrowthRate)
    }
    if memGrowthRate > 0 {
        remainingMem := (last.TotalMem * 0.8) - last.UsedMem
        memDays = int(remainingMem / memGrowthRate)
    }
    return
}

Runs daily — if resources will hit 80% within 7 days, auto-alerts for node expansion. During migration, it successfully predicted two resource bottlenecks, both resolved with pre-emptive node additions.

Pitfall 8: Performance Validation — Slower After Migration but Metrics Show Normal

The Problem: Users Report “It’s Laggy” But All Metrics Are Normal

After migration, business teams reported “page loading is slower.” Checked monitoring — CPU normal, memory normal, network latency normal, P99 normal. All metrics within pre-migration baselines.

Two days of investigation. Finally found: DNS resolution latency. VM-era DNS resolution was under 1ms (local dnsmasq). K8s CoreDNS was also fast (P50 2ms), but P99 had 15-20ms spikes. This wasn’t in standard monitoring — most monitoring only looks at application-layer metrics, not DNS resolution latency.

Why does 15ms DNS latency make pages slow? Because a page load might trigger 5-8 backend API calls, each triggering 2-3 internal service calls. If each call does DNS resolution, cumulative latency is 5×8×15ms = 600ms. Adding browser rendering and network transfer, user-perceived total latency increased 500-800ms — enough to turn “instant” into “noticeably slow.”

The Solution: DNS Performance Optimization + Latency Breakdown Monitoring

Step 1: NodeLocal DNS Cache

NodeLocal DNS Cache is the K8s community-recommended DNS optimization. It runs a DNS cache DaemonSet on each node — Pod DNS requests hit local cache first (latency < 1ms), only forwarding to CoreDNS on cache miss.

# NodeLocal DNS Cache deployment (simplified)
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-local-dns
  namespace: kube-system
spec:
  template:
    spec:
      containers:
      - name: node-cache
        image: registry.k8s.io/dns/k8s-dns-node-cache:1.22.20
        args:
        - -localip=169.254.20.10
        - -conf=/etc/Corefile
        - -upstream=10.96.0.10  # CoreDNS ClusterIP
        ports:
        - containerPort: 53
          hostPort: 53
        # Each Pod's resolv.conf points to 169.254.20.10
        # DNS requests hit local cache first

Post-deployment results:

P99 from 18ms to 1.2ms — cumulative latency from 600ms to 40ms, user-perceived difference disappeared.

Step 2: Latency Breakdown Monitoring

After migration, don’t just look at application-layer metrics — do latency breakdown: split a request’s total latency into DNS resolution + connection establishment + request sending + server processing + response transport, monitoring each stage.

We added DNS resolution time collection via custom OpenTelemetry Spans and built a latency breakdown dashboard in Grafana:

# DNS resolution latency P99 (via custom Span)
histogram_quantile(0.99, 
  sum(rate(dns_resolution_duration_seconds_bucket[5m])) by (le, service)
)

# Connection establishment latency P99
histogram_quantile(0.99,
  sum(rate(tcp_connect_duration_seconds_bucket[5m])) by (le, service)
)

# Server processing latency P99
histogram_quantile(0.99,
  sum(rate(http_server_duration_seconds_bucket{le!="+Inf"}[5m])) by (le, service)
)

The monitoring system evolved from “look at total latency” to “look at each stage’s latency.” When issues arise, instead of “it’s slow,” you see “which stage is slow” — fast root cause identification.

Architecture Decision Summary for the Full Migration

Connecting all 8 pitfalls, the core decision points for large-scale microservice K8s migration are:

Decision 1: Migration Batch Strategy

Our approach: topological batching + traffic canary — determine batches by dependency topology, then use traffic canary within each batch (10% → 30% → 50% → 100%), observing 30-60 minutes per step.

Decision 2: Stateful Service Migration Timing

When should stateful services migrate? I recommend batch 2 or 3, because:

• Batch 1 migrates dependency-free leaf services to validate K8s cluster basics (network, storage, scheduling)
• Stateful migration depends on stable cluster operation — can’t be in batch 1 (if cluster config has issues, stateful data may corrupt)
• Stateful services can’t be last — downstream services may depend on them, blocking traffic shifting

Decision 3: NetworkPolicy Gradual Rollout

Don’t enable full NetworkPolicy from day one. Default-deny-all is dangerous in production — you never know which service has undocumented dependencies.

Three-step approach:

1. Pre-migration: Default allow, only minimal Namespace isolation
2. During migration: Add NetworkPolicy per service — verify no communication impact before tightening
3. Post-migration: Full default-deny + whitelist policies

Decision 4: Rollback Plan Design

Migration rollback must be designed before starting. Our three-tier rollback:

Key design: all rollback levels are traffic-level — no need to delete Pods or roll back deployments. Old environment stays running until new environment is 100% stable for a week.

Summary

Migrating 120+ microservices to K8s — the technical approach itself isn’t complex. Dockerfiles, Helm Charts, CI/CD pipelines are all standard. The real challenges are in the detail decisions during migration:

1. Use runtime data for dependency mapping, not just static configs. Jaeger tracing + topology analysis is essential. Circular dependency groups must migrate together.
2. Handle stateful services in three layers: persistent data via primary-replica sync, in-memory state via readinessProbe, filesystem state via dedicated audit.
3. Choose CNI before migration — Cilium for 50+ nodes. Calico’s iptables has a performance cliff at scale; eBPF is more stable.
4. Handle DNS caching during canary shifting: application-layer TTL, CoreDNS optimization, dual-IP transition strategy — three-pronged approach.
5. Layer config management: common, environment-specific, sensitive configs in different K8s resources — avoid single ConfigMap bloat.
6. Don’t miss monitoring metric migration: label mapping + alert rule mapping table + dual-environment parallel validation — three steps to prevent monitoring gaps.
7. Capacity planning needs 40%+ headroom for system overhead, request/limit fragmentation, and I/O contention. Use workload-based node pool separation.
8. DNS performance is a hidden bottleneck — NodeLocal DNS Cache reduces P99 from 18ms to 1.2ms. Build latency breakdown monitoring for each request stage.

The deepest lesson: migration isn’t a technical problem, it’s an engineering management problem. Each pitfall isn’t a deep technical mystery, but each one bites you when you’re not looking. What you can do is map dependencies thoroughly, audit stateful services, pick the right CNI, and design rollback plans before starting. Then stay vigilant during migration and respond fast.

Post-migration results: autoscaling from 25 minutes to 1 minute (HPA + Cluster Autoscaler), deployment efficiency up 50% (CI/CD pipeline standardization), service availability from 99.5% to 99.9%. These numbers look great, but only those who’ve been through the process know how many pitfalls are behind them.

Hope these lessons help colleagues doing or planning K8s migration. Feel free to reach out with questions.

References & Acknowledgments

The following resources were referenced during the writing of this article. Credit to the original authors:

1. Optimizing Stateful Microservice Migration in Kubernetes with MS2M and Forensic Checkpointing — IEEE, proposed the MS2M framework and Forensic Container Checkpointing for stateful microservice migration; referenced for stateful service migration approaches
2. Enhancing the Kubernetes Platform with a Load-Aware Orchestration Strategy — Springer Nature, analyzed microservice scheduling optimization in K8s; referenced for large-scale microservice scheduling architecture
3. Stateful workloads in Azure Kubernetes Service (AKS) — Microsoft Learn, provided best practices for migrating stateful workloads to K8s and storage selection guidance
4. Locality-Aware QoS Optimization for Microservices Scheduling in Kubernetes Cluster — IEEE, LOCUS scheduler design; referenced for microservice dependency and scheduling performance analysis
5. Architecture best practices for Azure Kubernetes Service (AKS) — Microsoft, AKS architecture best practices; referenced for workload isolation and multi-cluster management trade-offs