Kubernetes Cluster Upgrade Strategy: From Planning to Zero-Downtime Execution

Release ──────────────────────────────────────── EOL
│ ←──────────── ~14 months ──────────────────→ │
│                                                │
├── Month 1: Community validation phase            │
├── Months 2-6: Mainstream adoption phase          │
├── Months 7-12: Mature stable phase               │
└── Months 13-14: Security patches only, EOL countd│

Control plane version: v1.36
    │
    ├── kubelet: max v1.36, min v1.35 (1 minor behind)
    ├── kube-proxy: same version as kubelet
    └── kubectl: max v1.37, min v1.35 (±1 minor)

## Pre-Upgrade Assessment Checklist

### Cluster Status
- [ ] Current version (kubeadm version + kubectl version)
- [ ] Target version
- [ ] Number and distribution of cluster nodes
- [ ] Number and types of running workloads
- [ ] List of existing Custom Resource Definitions (CRDs)

### Compatibility Check
- [ ] API deprecation check for target version
- [ ] Third-party component compatibility (CNI, CSI, Ingress Controller)
- [ ] Application API versions not being removed
- [ ] kubelet version skew within allowed range
- [ ] etcd version compatibility

### Backup & Recovery
- [ ] Full etcd data backup
- [ ] Cluster configuration file backup
- [ ] Certificate expiration check
- [ ] Backup restore drill verified

### Resources & Capacity
- [ ] Node resource headroom sufficient (at least 20%)
- [ ] PodDisruptionBudget configured correctly
- [ ] Multi-replica deployment confirmed
- [ ] Pod anti-affinity rules confirmed

### Operations
- [ ] Upgrade window determined
- [ ] Rollback plan prepared
- [ ] Monitoring and alerting configured
- [ ] Emergency contacts on standby

# Method 1: Use deprecation metrics
# Check for deprecated API resources currently in use
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

# Method 2: Use kube-no-trouble (kubent)
# GitHub: https://github.com/doitintl/kube-no-trouble

# Download and run
wget https://github.com/doitintl/kube-no-trouble/releases/latest/download/kubent-linux-amd64.tar.gz
tar xzf kubent-linux-amd64.tar.gz
./kubent

# Sample output:
# [ERROR] Deployment "my-app" in namespace "default": uses deprecated API v1beta1
# [ERROR] ClusterRole "my-role" in namespace "": uses deprecated API v1beta1
# Migrate all deprecated APIs to new versions before upgrading

# Method 3: Query the API server directly
kubectl get --raw='/api/v1' | jq '.resources[] | select(.deprecationWarning != null) | {name, deprecationWarning}'

#!/bin/bash
# etcd_backup.sh - Production-grade etcd backup script

ETCD_ENDPOINTS="https://127.0.0.1:2379"
ETCD_CERT="/etc/etcd/pki/etcd-peer.crt"
ETCD_KEY="/etc/etcd/pki/etcd-peer.key"
ETCD_CACERT="/etc/etcd/pki/ca.crt"
BACKUP_DIR="/var/backups/etcd"
DATE=$(date +%Y%m%d_%H%M%S)

mkdir -p "${BACKUP_DIR}"

# Take snapshot
ETCDCTL_API=3 etcdctl snapshot save "${BACKUP_DIR}/etcd-snapshot-${DATE}.db" \
    --endpoints="${ETCD_ENDPOINTS}" \
    --cert="${ETCD_CERT}" \
    --key="${ETCD_KEY}" \
    --cacert="${ETCD_CACERT}"

# Verify snapshot integrity
ETCDCTL_API=3 etcdctl snapshot status "${BACKUP_DIR}/etcd-snapshot-${DATE}.db" \
    --write-out=table

# Clean up backups older than 7 days
find "${BACKUP_DIR}" -name "etcd-snapshot-*.db" -mtime +7 -delete

echo "Backup completed: etcd-snapshot-${DATE}.db"

# Check all K8s certificate expiration dates
sudo kubeadm certs check-expiration

# Sample output:
# CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY
# admin.conf                 Jul 11, 2027 10:00 UTC   364d            ca
# apiserver                 Jul 11, 2027 10:00 UTC   364d            ca
# apiserver-etcd-client      Jul 11, 2027 10:00 UTC   364d            etcd-ca
# ...

# If certificates are near expiration (<90 days), renew first
sudo kubeadm certs renew all

# Restart affected components
sudo systemctl restart kube-apiserver kube-controller-manager kube-scheduler

# ============================================
# Step 1: Upgrade kubeadm
# ============================================

# Add Kubernetes APT repository (if using APT)
sudo apt-mark unhold kubeadm
sudo apt-get update && sudo apt-get install -y kubeadm=1.36.0-00
sudo apt-mark hold kubeadm

# Verify version
kubeadm version
# kubeadm version: &version.Info{Major:"1", Minor:"36", ...}

# ============================================
# Step 2: Upgrade plan check
# ============================================

# Check upgrade plan
sudo kubeadm upgrade plan

# Sample output:
# Components that must be upgraded manually after you have upgraded the control plane:
# COMPONENT   CURRENT    TARGET
# kubelet     v1.35.0    v1.36.0
#
# Upgrade to the latest version in the v1.36 series:
# COMPONENT                 CURRENT    TARGET
# kube-apiserver            v1.35.0    v1.36.0
# kube-controller-manager   v1.35.0    v1.36.0
# kube-scheduler            v1.35.0    v1.36.0
# kube-proxy                v1.35.0    v1.36.0
# etcd                      3.5.16     3.7.0

# ============================================
# Step 3: Apply upgrade
# ============================================

# On the first control plane node
sudo kubeadm upgrade apply v1.36.0

# kubeadm automatically performs:
# 1. Pre-checks cluster health
# 2. Upgrades etcd (if needed)
# 3. Sequentially upgrades control plane components (API server, controller manager, scheduler)
# 4. Updates kube-proxy ConfigMap
# 5. Updates kubelet configuration

# On other control plane nodes (no version number needed)
sudo kubeadm upgrade node

# ============================================
# Step 4: Upgrade CNI plugin
# ============================================

# CNI plugins must be upgraded manually! kubeadm does not manage CNI
# Check current CNI version
kubectl get pods -n kube-system -l k8s-app=calico-node -o yaml | grep image

# Upgrade Calico
kubectl apply -f https://docs.projectcalico.org/manifests/calico-v3.28.yaml
# Or upgrade Cilium
helm upgrade cilium cilium/cilium --namespace kube-system \
    --version 1.16.0 --reuse-values

# ============================================
# Step 5: Upgrade kubelet (all nodes)
# ============================================

# Execute on each node
sudo apt-mark unhold kubelet kubectl
sudo apt-get update && sudo apt-get install -y \
    kubelet=1.36.0-00 kubectl=1.36.0-00
sudo apt-mark hold kubelet kubectl

# Restart kubelet
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# Verify node status
kubectl get nodes
# NAME          STATUS   ROLES           AGE    VERSION
# master-01     Ready    control-plane   365d   v1.36.0
# worker-01     Ready    <none>          365d   v1.35.0  <- not yet upgraded
# worker-02     Ready    <none>          365d   v1.35.0

# ============================================
# Step 6: Drain and upgrade worker nodes (one at a time)
# ============================================

# Drain node (migrate pods to other nodes)
kubectl drain worker-01 --ignore-daemonsets --delete-emptydir-data

# Upgrade kubeadm (skip if already upgraded)
# sudo apt-get install -y kubeadm=1.36.0-00

# Execute upgrade on worker node
sudo kubeadm upgrade node

# Upgrade kubelet
sudo apt-get install -y kubelet=1.36.0-00 kubectl=1.36.0-00
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# Restore scheduling
kubectl uncordon worker-01

# Verify
kubectl get nodes

# Verify all nodes are at the same version
kubectl get nodes -o wide

# Verify core component status
kubectl get pods -n kube-system -o wide

# Verify component versions
kubectl version --short
# Client Version: v1.36.0
# Server Version: v1.36.0

# Check cluster events
kubectl get events -n kube-system --sort-by='.lastTimestamp' | tail -20

# Verify critical workloads
kubectl get pods --all-namespaces --field-selector=status.phase!=Running

                    ┌───────────────┐
                    │  Global LB    │
                    │  / DNS        │
                    └───────┬───────┘
                            │
              ┌─────────────┴──────────────┐
              │                              │
    ┌─────────▼─────────┐       ┌──────────▼──────────┐
    │  Blue Cluster      │       │  Green Cluster      │
    │  (v1.35 - Current)  │       │  (v1.36 - Target)   │
    │                     │       │                      │
    │  ┌───┐  ┌───┐      │       │  ┌───┐  ┌───┐      │
    │  │CP1│  │CP2│      │       │  │CP1│  │CP2│      │
    │  └───┘  └───┘      │       │  └───┘  └───┘      │
    │  ┌───┐  ┌───┐      │       │  ┌───┐  ┌───┐      │
    │  │W1 │  │W2 │      │       │  │W1 │  │W2 │      │
    │  └───┘  └───┘      │       │  └───┘  └───┘      │
    └─────────────────────┘       └──────────────────────┘

    Traffic: 100% ─────────────────── Traffic: 0%
                            ↑
                    After cutover: 0%     After cutover: 100%

#!/bin/bash
# blue_green_upgrade.sh - Blue-green upgrade workflow

set -euo pipefail

OLD_CLUSTER="prod-blue"
NEW_CLUSTER="prod-green"
LB_ENDPOINT="lb.example.com"

# ============================================
# Phase 1: Deploy new cluster (Green)
# ============================================

echo "=== Phase 1: Deploy Green Cluster ==="

# Deploy new cluster using Terraform or Ansible
# New cluster uses target version v1.36
terraform apply -var="cluster_name=${NEW_CLUSTER}" \
                -var="k8s_version=v1.36.0" \
                -auto-approve

# Wait for cluster to be ready
echo "Waiting for cluster to be ready..."
until kubectl --context=${NEW_CLUSTER} get nodes | grep -c "Ready" | grep -q "3"; do
    echo "  Waiting... $(date)"
    sleep 30
done

# ============================================
# Phase 2: Deploy workloads
# ============================================

echo "=== Phase 2: Deploy Workloads to Green ==="

# Apply all manifests
kubectl --context=${NEW_CLUSTER} apply -f manifests/namespaces/
kubectl --context=${NEW_CLUSTER} apply -f manifests/crds/
kubectl --context=${NEW_CLUSTER} apply -f manifests/deployments/
kubectl --context=${NEW_CLUSTER} apply -f manifests/services/
kubectl --context=${NEW_CLUSTER} apply -f manifests/ingress/

# Wait for all Deployments to be ready
kubectl --context=${NEW_CLUSTER} wait --for=condition=available \
    --all --timeout=600s deployment -A

# ============================================
# Phase 3: Data synchronization
# ============================================

echo "=== Phase 3: Data Migration ==="

# For stateful services, synchronize data
# Example: use Velero for migration or database replication

# Velero migration example
velero backup create pre-upgrade-backup \
    --include-namespaces app,monitoring,logging \
    --kube-context=${OLD_CLUSTER}

velero restore create --from-backup pre-upgrade-backup \
    --kube-context=${NEW_CLUSTER}

# ============================================
# Phase 4: Canary validation (10% traffic)
# ============================================

echo "=== Phase 4: Canary - 10% traffic to Green ==="

# Route 10% traffic to new cluster using Weighted DNS or Service Mesh
# Istio VirtualService example
cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: app-routing
  namespace: istio-system
spec:
  gateways:
  - mesh
  - gateway
  hosts:
  - "app.example.com"
  http:
  - route:
    - destination:
        host: app.blue.svc.cluster.local
        port:
          number: 80
      weight: 90
    - destination:
        host: app.green.svc.cluster.local
        port:
          number: 80
      weight: 10
EOF

# Monitor for 15 minutes
echo "Monitoring for 15 minutes..."
sleep 900

# Check error rate
ERROR_RATE=$(curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{cluster=\"green\",code=~\"5..\"}[5m])/rate(http_requests_total{cluster=\"green\"}[5m])" | jq -r '.data.result[0].value[1]')

if (( $(echo "${ERROR_RATE} > 0.01" | bc -l) )); then
    echo "ERROR: Error rate too high (${ERROR_RATE}), rolling back!"
    exit 1
fi

# ============================================
# Phase 5: Full cutover
# ============================================

echo "=== Phase 5: Full cutover - 100% to Green ==="

# Switch all traffic to new cluster
cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: app-routing
  namespace: istio-system
spec:
  gateways:
  - mesh
  - gateway
  hosts:
  - "app.example.com"
  http:
  - route:
    - destination:
        host: app.green.svc.cluster.local
        port:
          number: 80
      weight: 100
EOF

echo "=== Upgrade complete. Old cluster (Blue) kept for 72h rollback window ==="

Cluster: 100 worker nodes

Upgrade batches:
├── Batch 1: 3 nodes (3%) — minimum validation set
│   └── Observe 30 min, verify core functionality
├── Batch 2: 7 nodes (10%) — expanded validation
│   └── Observe 30 min, verify autoscaling
├── Batch 3: 20 nodes (30%) — large-scale validation
│   └── Observe 1 hour, verify performance metrics
├── Batch 4: 30 nodes (60%) — primary traffic
│   └── Observe 1 hour
└── Batch 5: 40 nodes (100%) — full completion
    └── Final verification

#!/bin/bash
# canary_node_upgrade.sh - Canary node upgrade

set -euo pipefail

TARGET_VERSION="1.36.0-00"
BATCH_SIZES=(3 7 20 30 40)
OBSERVE_TIME_SEC=1800  # 30-minute observation period

# Get nodes for a given batch
get_nodes_for_batch() {
    local batch_num=$1
    local batch_size=${BATCH_SIZES[$((batch_num - 1))]}
    kubectl get nodes -l '!node-role.kubernetes.io/control-plane' \
        -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' \
        | grep "v1.35" \
        | head -n "${batch_size}"
}

# Upgrade a single node
upgrade_node() {
    local node=$1
    echo "  Upgrading node: ${node}"

    # 1. Drain node
    kubectl drain "${node}" --ignore-daemonsets --delete-emptydir-data \
        --grace-period=30 --timeout=300s

    # 2. SSH to node and upgrade
    ssh "${node}" << EOF
        sudo apt-mark unhold kubeadm kubelet kubectl
        sudo apt-get update
        sudo apt-get install -y kubeadm=${TARGET_VERSION} kubelet=${TARGET_VERSION} kubectl=${TARGET_VERSION}
        sudo apt-mark hold kubeadm kubelet kubectl
        sudo kubeadm upgrade node
        sudo systemctl daemon-reload
        sudo systemctl restart kubelet
EOF

    # 3. Uncordon
    kubectl uncordon "${node}"

    # 4. Wait for node Ready
    kubectl wait --for=condition=Ready "node/${node}" --timeout=300s
}

# Health check
health_check() {
    echo "  Running health checks..."

    # Check node status
    local not_ready=$(kubectl get nodes | grep -v "Ready" | wc -l)
    if [ "${not_ready}" -gt 0 ]; then
        echo "  FAIL: ${not_ready} nodes not ready"
        return 1
    fi

    # Check pod restart rate
    local restarts=$(kubectl get pods -A -o json \
        | jq '[.items[] | select(.status.containerStatuses != null) | .status.containerStatuses[] | select(.restartCount > 5)] | length')
    if [ "${restarts}" -gt 3 ]; then
        echo "  FAIL: ${restarts} pods with excessive restarts"
        return 1
    fi

    # Check CrashLoopBackOff
    local crashes=$(kubectl get pods -A --field-selector=status.phase=Running \
        -o json | jq '[.items[] | select(.status.containerStatuses[]? | .state.waiting.reason == "CrashLoopBackOff")] | length')
    if [ "${crashes}" -gt 0 ]; then
        echo "  FAIL: ${crashes} pods in CrashLoopBackOff"
        return 1
    fi

    echo "  Health check PASSED"
    return 0
}

# Main loop
for batch_idx in "${!BATCH_SIZES[@]}"; do
    batch_num=$((batch_idx + 1))
    echo "============================================"
    echo "Batch ${batch_num}/${#BATCH_SIZES[@]}"
    echo "============================================"

    nodes=$(get_nodes_for_batch "${batch_num}")
    if [ -z "${nodes}" ]; then
        echo "No more nodes to upgrade"
        break
    fi

    for node in ${nodes}; do
        upgrade_node "${node}"
    done

    echo "Waiting ${OBSERVE_TIME_SEC}s for observation..."
    sleep "${OBSERVE_TIME_SEC}"

    if ! health_check; then
        echo "BATCH ${batch_num} FAILED! Stopping upgrade."
        echo "Already upgraded nodes remain on v1.36, pending investigation."
        exit 1
    fi
done

echo "All batches completed successfully!"
kubectl get nodes -o wide

# EKS cluster upgrade workflow

# 1. Check available versions
aws eks describe-cluster --name prod-cluster \
    --query 'cluster.version' --output text
# 1.35

aws eks describe-addon-versions \
    --kubernetes-version 1.36 \
    --query 'addons[].addonName' --output text

# 2. Upgrade control plane
aws eks update-cluster-version \
    --name prod-cluster \
    --version 1.36

# 3. Wait for upgrade to complete
aws eks wait cluster-active --name prod-cluster

# 4. Upgrade node groups
aws eks update-nodegroup-version \
    --cluster-name prod-cluster \
    --nodegroup-name prod-ng-1 \
    --kubernetes-version 1.36

# 5. Upgrade managed node group AMI
aws eks update-nodegroup-version \
    --cluster-name prod-cluster \
    --nodegroup-name prod-ng-1 \
    --release-version 1.36.0-20260711

# 6. Monitor upgrade progress
aws eks describe-update \
    --name prod-cluster \
    --update-id <update-id>

# GKE cluster upgrade

# 1. Check available versions
gcloud container get-server-config

# 2. Upgrade control plane
gcloud container clusters upgrade prod-cluster \
    --master --cluster-version 1.36 \
    --region asia-east1

# 3. Upgrade node pools
gcloud container clusters upgrade prod-cluster \
    --node-pool default-pool \
    --cluster-version 1.36 \
    --region asia-east1

# 4. Set maintenance window (for auto-upgrades)
gcloud container clusters update prod-cluster \
    --maintenance-window-start=2026-07-11T02:00:00Z \
    --region asia-east1

# 5. Enable surge upgrade (parallel node upgrade)
gcloud container clusters update prod-cluster \
    --max-surge-upgrade=3 \
    --max-unavailable-upgrade=1 \
    --region asia-east1

# AKS cluster upgrade

# 1. Check available versions
az aks get-upgrades \
    --resource-group prod-rg \
    --name prod-aks \
    --output table

# 2. Upgrade control plane
az aks upgrade \
    --resource-group prod-rg \
    --name prod-aks \
    --kubernetes-version 1.36 \
    --no-wait

# 3. Check upgrade status
az aks show \
    --resource-group prod-rg \
    --name prod-aks \
    --query "provisioningState"

# 4. Upgrade node pool
az aks nodepool upgrade \
    --resource-group prod-rg \
    --cluster-name prod-aks \
    --name nodepool1 \
    --kubernetes-version 1.36

# 5. Set node auto-upgrade channel
az aks update \
    --resource-group prod-rg \
    --name prod-aks \
    --auto-upgrade-channel node-image \
    --node-os-upgrade-channel SecurityPatch

# kubeadm does not support automatic rollback!
# Rollback requires manual component downgrade

# Scenario: rollback from v1.36 to v1.35

# 1. Downgrade kubeadm
sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm=1.35.0-00
sudo apt-mark hold kubeadm

# 2. Restore etcd backup (critical step!)
sudo ETCDCTL_API=3 etcdctl snapshot restore \
    /var/backups/etcd/etcd-snapshot-pre-upgrade.db \
    --data-dir=/var/lib/etcd-restored

# 3. Replace etcd data
sudo systemctl stop etcd
sudo mv /var/lib/etcd /var/lib/etcd-broken
sudo mv /var/lib/etcd-restored /var/lib/etcd
sudo chown -R etcd:etcd /var/lib/etcd
sudo systemctl start etcd

# 4. Downgrade control plane components
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.35.0-00 kubectl=1.35.0-00
sudo apt-mark hold kubelet kubectl
sudo systemctl restart kubelet

# 5. Verify
kubectl get nodes
kubectl get pods -A

# Instant rollback
cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: app-routing
  namespace: istio-system
spec:
  gateways:
  - mesh
  - gateway
  hosts:
  - "app.example.com"
  http:
  - route:
    - destination:
        host: app.blue.svc.cluster.local
        port:
          number: 80
      weight: 100
    - destination:
        host: app.green.svc.cluster.local
        port:
          number: 80
      weight: 0
EOF

echo "Traffic switched back to Blue cluster"

# Use maintenance windows for automated upgrades
# Recommended to execute during business off-peak hours

# View cluster load trends (determine window from historical data)
# Typical window: 2:00 AM - 6:00 AM (local time)

# Configure maintenance window with kubeadm
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
metadata:
  name: config
kubernetesVersion: v1.36.0
---
# Set maintenance window for node upgrades
apiVersion: apps/v1
kind: MaintenanceWindow
metadata:
  name: upgrade-window
spec:
  schedule: "0 2 * * 6"  # Every Saturday 2:00 AM
  duration: 4h

# Set PDB for critical applications
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-server-pdb
  namespace: production
spec:
  minAvailable: 2          # Keep at least 2 replicas available
  # Or use maxUnavailable: 1
  selector:
    matchLabels:
      app: api-server
---
# Strict PDB for database-class applications
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: redis-pdb
  namespace: production
spec:
  maxUnavailable: 0          # No replicas allowed to be unavailable (use rolling updates)
  selector:
    matchLabels:
      app: redis

# Key Prometheus metrics to monitor during upgrades
# Alert rule examples

groups:
- name: upgrade-monitoring
  rules:
  # Node NotReady alert
  - alert: NodeNotReady
    expr: kube_node_status_condition{condition="Ready",status!="true"} == 1
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Node {{ $labels.node }} is not ready"

  # Pod restart alert
  - alert: PodRestarting
    expr: increase(kube_pod_container_status_restarts_total[30m]) > 3
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "Pod {{ $labels.pod }} restarting frequently"

  # API Server latency alert
  - alert: APIServerLatencyHigh
    expr: histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket[5m])) > 1
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "API server P99 latency > 1s"

  # Scheduling failure alert
  - alert: SchedulingFailed
    expr: kube_pod_status_unscheduled == 1
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Pod {{ $labels.pod }} cannot be scheduled"

# argocd-app-cluster-config.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cluster-config
  namespace: argocd
spec:
  source:
    repoURL: https://github.com/org/cluster-config
    targetRevision: release/v1.36
    path: manifests
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
    - ApplyOutOfSyncOnly=true

# Check kubelet status
sudo systemctl status kubelet

# Check kubelet logs
sudo journalctl -u kubelet --since "10 minutes ago" | tail -50

# Common causes:
# 1. CNI plugin not upgraded causing network issues
# 2. Container runtime version incompatible
# 3. Configuration file format changed

# Fix CNI
kubectl get pods -n kube-system -l k8s-app=calico-node
# If CNI pods are not running, manually upgrade CNI
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

# CoreDNS ConfigMap format may change after upgrade
kubectl get configmap coredns -n kube-system -o yaml

# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# If CoreDNS keeps restarting, check config
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

# Fix: update CoreDNS ConfigMap
kubectl edit configmap coredns -n kube-system
# Ensure plugin config is compatible with CoreDNS version

# If resources disappear after upgrade, APIs may have been removed

# Check API server logs
sudo journalctl -u kube-apiserver | grep "deprecated"

# View removed APIs
kubectl api-resources --verbs=list -o name | xargs -n1 kubectl get -A 2>&1 | grep "not found"

# Recovery:
# 1. Temporarily downgrade to old version
# 2. Update apiVersion in resource YAML to new version
# 3. Re-apply
# 4. Upgrade again

# Cluster API upgrade example
# Trigger upgrade by modifying MachineDeployment's version field

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: prod-cluster
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["10.244.0.0/16"]
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: prod-cluster-cp
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: AWSCluster
    name: prod-cluster-infra
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: prod-cluster-cp
  namespace: default
spec:
  version: v1.36.0   # Modify version to trigger upgrade
  replicas: 3
  machineTemplate:
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: AWSMachineTemplate
      name: prod-cluster-cp-template
  kubeadmConfigSpec:
    initConfiguration:
      nodeRegistration:
        kubeletExtraArgs:
          feature-gates: "..."
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: prod-cluster-md-0
  namespace: default
spec:
  clusterName: prod-cluster
  replicas: 5
  template:
    spec:
      version: v1.36.0   # Worker nodes also upgrade
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: KubeadmConfigTemplate
          name: prod-cluster-md-0
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: AWSMachineTemplate
        name: prod-cluster-md-0