Kubernetes Multi-Cluster Management in Practice

Overview

When your business scale grows beyond what a single cluster can handle, multi-cluster becomes an inevitable choice. Reasons include: single cluster node limits (5000 nodes), multi-region deployment, hybrid cloud strategy, fault isolation, and compliance requirements. But multi-cluster management complexity grows exponentially—how to deploy applications across clusters, discover services cross-cluster, synchronize configurations, and handle failover.

This article systematically covers multi-cluster architecture patterns, mainstream management tool comparisons, and practical solutions for cross-cluster service discovery, CI/CD, and disaster recovery failover.

Based on Kubernetes v1.30. The multi-cluster management space is still rapidly evolving; monitor tool maturity continuously.

Why Multi-Cluster

Single Cluster Bottlenecks

Bottleneck	Description
Scale limit	K8s single cluster recommended limit: 5000 nodes, 150K Pods, 300K containers
Failure domain	Single cluster etcd failure affects all workloads
Upgrade risk	Cluster upgrades may impact all workloads
Multi-tenant isolation	Soft isolation is weaker than hard isolation
Regional latency	Cross-region can’t use one cluster
Compliance	Data must not cross regions/borders

Typical Multi-Cluster Scenarios

Scenario	Architecture	Goal
Multi-region DR	One cluster per region, DNS global load balancing	RTO < 5min
Hybrid cloud	Cloud + on-prem	Elastic + compliance
Dev/test/prod isolation	One cluster per environment	Security isolation
Multi-tenant hard isolation	Independent cluster per tenant	Security compliance
Edge computing	Central cluster + edge clusters	Low latency

Multi-Cluster Architecture Patterns

Pattern 1: Hub-Spoke

        ┌─────────┐
        │  Hub    │  ← Management cluster
        │ Cluster │
        └────┬────┘
    ┌────────┼────────┐
    ▼        ▼        ▼
┌──────┐ ┌──────┐ ┌──────┐
│Spoke1│ │Spoke2│ │Spoke3│  ← Work clusters
└──────┘ └──────┘ └──────┘

The hub cluster manages configuration, distributes applications, and collects status. Work clusters only run workloads. This is the most common multi-cluster management pattern.

Pros: Centralized management, consistent configuration, easy operations. Cons: Hub is a single point of failure; hub failure doesn’t affect existing workloads but blocks new deployments.

Pattern 2: Federation

┌─────────────────────────────────────┐
│         Federation Control Plane     │
│  (Unified API, cross-cluster scheduling) │
└───┬──────────┬──────────┬──────────┘
    ▼          ▼          ▼
┌──────┐  ┌──────┐  ┌──────┐
│Clstr1│  │Clstr2│  │Clstr3│
└──────┘  └──────┘  └──────┘

The federation control plane provides a unified API. Users create resources at the federation level, which are automatically distributed to member clusters.

Pros: Unified API, automatic scheduling, cross-cluster service discovery. Cons: Complex architecture, control plane itself requires HA.

Pattern 3: Mesh

  ┌──────┐         ┌──────┐
  │Clstr1│◄───────►│Clstr2│
  └──┬───┘         └───┬──┘
     │                  │
     ▼                  ▼
  ┌──────┐         ┌──────┐
  │Clstr3│◄───────►│Clstr4│
  └──────┘         └──────┘

Clusters are peers, interconnected via service mesh. Suitable for peer-to-peer multi-region deployment.

Pros: No central node, good fault isolation. Cons: Complex management, hard to guarantee consistency.

Mainstream Multi-Cluster Management Tools

Tool Overview

Tool	Status	Core Capability	Maturity	CNCF Status
KubeFed	Archived	Federation API	Stalled	Archived
Cluster API	Active	Cluster lifecycle	High	Incubating
Karmada	Active	Multi-cluster orchestration	High	Incubating
OCM (Open Cluster Management)	Active	Cluster management	Medium	Incubating
Argo CD + ApplicationSet	Active	GitOps multi-cluster deploy	High	Graduated
Submariner	Active	Cross-cluster networking	Medium	Incubating

KubeFed (Archived)

KubeFed was the earliest K8s multi-cluster federation project but was officially archived in 2023.

Not recommended for new projects. Reference: KubeFed Archive Notice

Cluster API

Cluster API (CAPI) focuses on cluster lifecycle management—creating, upgrading, and destroying clusters. It does not handle cross-cluster application deployment.

Core Concepts:

Concept	Description
`Cluster`	Declarative definition of a K8s cluster
`Machine`	A node (Control Plane or Worker)
`MachineDeployment`	Like Deployment, manages a set of Machines
`MachineSet`	Like ReplicaSet
`MachineHealthCheck`	Node health check and auto-repair
`InfrastructureProvider`	Infrastructure provider (AWS/GCP/Azure/vSphere)

Example: Declarative cluster creation:

# cluster.yaml
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: my-cluster
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["10.244.0.0/16"]
    services:
      cidrBlocks: ["10.96.0.0/12"]
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: my-cluster-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSCluster
    name: my-cluster

---
# control-plane.yaml
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: my-cluster-control-plane
spec:
  replicas: 3
  version: v1.30.0
  machineTemplate:
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
      kind: AWSMachineTemplate
      name: my-cluster-control-plane
  kubeadmConfigSpec:
    initConfiguration:
      nodeRegistration:
        kubeletExtraArgs:
          cloud-provider: aws
    clusterConfiguration:
      apiServer:
        extraArgs:
          cloud-provider: aws

---
# worker-nodes.yaml
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: my-cluster-md-0
spec:
  clusterName: my-cluster
  replicas: 3
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: my-cluster
  template:
    spec:
      clusterName: my-cluster
      version: v1.30.0
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: KubeadmConfigTemplate
          name: my-cluster-md-0
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
        kind: AWSMachineTemplate
        name: my-cluster-md-0

Pros: Declarative cluster lifecycle management, multi-infrastructure support, automatic node repair. Cons: Only manages cluster creation, not application deployment; steep learning curve.

Karmada

Karmada (Kubernetes Management Daemon) is an open-source multi-cluster orchestration engine from Huawei, a CNCF incubating project.

Core Capabilities:

Cross-cluster application distribution (PropagationPolicy)
Cross-cluster service discovery
Failover
Resource re-scheduling

Architecture:

Karmada Control Plane
├── karmada-apiserver     (Unified API entry)
├── karmada-controller-manager
├── karmada-scheduler      (Cross-cluster scheduling)
└── karmada-webhook

Member Clusters
├── cluster-1 (push mode)
├── cluster-2 (push mode)
└── cluster-3 (pull mode)

Application distribution example:

# Define application
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  replicas: 10
  selector:
    matchLabels:
      app: myapp
  template:
    spec:
      containers:
      - name: app
        image: myapp:v1

---
# Define propagation policy
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: myapp-propagation
spec:
  resourceSelectors:
  - apiVersion: apps/v1
    kind: Deployment
    name: myapp
  placement:
    clusterAffinity:
      clusterNames:
      - cluster-beijing
      - cluster-shanghai
    replicaScheduling:
      replicaSchedulingType: Divided
      replicaDivisionPreference: Weighted
      weightPreference:
        staticWeightList:
        - targetCluster:
            clusterNames: [cluster-beijing]
          weight: 7
        - targetCluster:
            clusterNames: [cluster-shanghai]
          weight: 3

Failover:

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: myapp-failover
spec:
  resourceSelectors:
  - apiVersion: apps/v1
    kind: Deployment
    name: myapp
  placement:
    clusterAffinity:
      clusterNames:
      - cluster-beijing
      - cluster-shanghai
    spreadConstraints:
    - spreadByLabel: failure-domain.beta.kubernetes.io/region
      maxGroups: 2
      minGroups: 1
  failover:
    application:
      decisionConditions:
        maxUnavailable: 50%
      gracePeriodSeconds: 300
      purgeMode: Graceful

Pros: Feature-rich, supports failover, good Chinese documentation. Cons: Relatively small community, control plane requires HA.

OCM (Open Cluster Management)

OCM is an open-source multi-cluster management framework from Red Hat.

Core Concepts:

Concept	Description
`ManagedCluster`	A managed cluster
`ManagedClusterSet`	A set of clusters
`Placement`	Cluster selection policy
`ManifestWork`	Workload distributed to clusters
`Subscription`	GitOps subscription
`Channel`	Subscription source

Pros: Good OpenShift integration, modular design. Cons: Smaller community, English-focused documentation.

Argo CD ApplicationSet

Argo CD itself is a GitOps tool; ApplicationSet enables multi-cluster deployment:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: myapp-multi-cluster
spec:
  generators:
  - list:
      elements:
      - cluster: cluster-beijing
        url: https://cluster-beijing-api:6443
      - cluster: cluster-shanghai
        url: https://cluster-shanghai-api:6443
  template:
    metadata:
      name: '{{cluster}}-myapp'
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/myapp-deploy
        targetRevision: HEAD
        path: overlays/{{cluster}}
      destination:
        server: '{{url}}'
        namespace: production
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Pros: Seamless Argo CD integration, GitOps pattern, mature and stable. Cons: Only handles application deployment, not cluster lifecycle; no cross-cluster service discovery.

Tool Selection Guide

Requirement	Recommended Tool
Cluster create/upgrade/destroy	Cluster API
Cross-cluster app distribution + failover	Karmada
GitOps multi-cluster deployment	Argo CD + ApplicationSet
Cross-cluster network connectivity	Submariner
Red Hat/OpenShift ecosystem	OCM
Simple multi-cluster deployment (<10 clusters)	Argo CD + ApplicationSet
Complex multi-cluster orchestration (>10 clusters)	Karmada + Cluster API

Cross-Cluster Service Discovery

Option 1: Global DNS

The simplest cross-cluster service discovery method, using CoreDNS multi-cluster plugin:

# Service in Cluster A
svc-a.namespace-a.svc.cluster-beijing.cluster.local

# Service in Cluster B
svc-b.namespace-b.svc.cluster-shanghai.cluster.local

Through global DNS resolution, clusters can access each other’s Services.

Option 2: Service Mesh Multi-Cluster

Istio supports multi-cluster Service Mesh, providing cross-cluster load balancing and failover:

# ServiceExport: Export Service to mesh
apiVersion: networking.istio.io/v1beta1
kind: ServiceExport
metadata:
  name: myapp
  namespace: production
spec:
  hosts:
  - "myapp.production.svc.cluster.local"
  ports:
  - number: 8080
    name: http
    protocol: HTTP

# ServiceEntry: Import in other clusters
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: remote-myapp
spec:
  hosts:
  - "myapp.production.svc.cluster.local"
  location: MESH_INTERNAL
  ports:
  - number: 8080
    name: http
    protocol: HTTP
  resolution: DNS
  endpoints:
  - address: cluster-beijing-api.internal
    ports:
      http: 15443

Option 3: Submariner

Submariner connects cluster Pod networks via IP tunnels:

# Install Submariner
subctl deploy-broker --kubeconfig cluster-a.kubeconfig

# Join clusters
subctl join broker-info.subm --clusterid cluster-a --kubeconfig cluster-a.kubeconfig
subctl join broker-info.subm --clusterid cluster-b --kubeconfig cluster-b.kubeconfig

# Verify connectivity
subctl show all --kubeconfig cluster-a.kubeconfig

Comparison

Option	Complexity	Features	Latency	Use Case
Global DNS	Low	Service discovery	High (cross-cluster)	Simple scenarios
Service Mesh	High	Discovery + LB + policy	Medium	Advanced traffic management
Submariner	Medium	Pod network direct connect	Low	Need Pod-to-Pod connectivity

Unified CI/CD

GitOps Multi-Cluster Deployment

# Directory structure
# ├── base/                 # Base configuration
# │   ├── deployment.yaml
# │   ├── service.yaml
# │   └── kustomization.yaml
# ├── overlays/             # Per-cluster overlays
# │   ├── cluster-beijing/
# │   │   ├── deployment-patch.yaml
# │   │   └── kustomization.yaml
# │   └── cluster-shanghai/
# │       ├── deployment-patch.yaml
# │       └── kustomization.yaml
# └── argocd-apps/         # Argo CD Application
#     └── appset.yaml

# Argo CD ApplicationSet
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: myapp
spec:
  generators:
  - git:
      repoURL: https://github.com/myorg/myapp-deploy
      revision: HEAD
      directories:
      - path: overlays/*
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      project: production
      source:
        repoURL: https://github.com/myorg/myapp-deploy
        targetRevision: HEAD
        path: '{{path}}'
      destination:
        server: '{{cluster.url}}'
        namespace: production
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
        - CreateNamespace=true

Configuration Diff Management

Per-cluster configuration differences are managed via Kustomize patches:

# overlays/cluster-beijing/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base

patches:
- path: deployment-patch.yaml

configMapGenerator:
- name: app-config
  behavior: merge
  literals:
  - REGION=beijing
  - DB_HOST=pg-beijing.internal

# overlays/cluster-beijing/deployment-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 5                   # Beijing cluster: 5 replicas
  template:
    spec:
      nodeSelector:
        topology.kubernetes.io/region: cn-north-1

Disaster Recovery Failover

Multi-Region DR Architecture

            ┌─────────────────────┐
            │   Global DNS / GSLB  │
            │   (Route53 / CloudDNS)│
            └──────┬────────┬──────┘
                   │        │
          ┌─────────┘        └─────────┐
          ▼                            ▼
   ┌──────────────┐            ┌──────────────┐
   │  Beijing      │            │  Shanghai     │
   │  Cluster      │            │  Cluster      │
   │  (Active)     │            │  (Standby)    │
   │  Weight: 100  │            │  Weight: 0    │
   └──────────────┘            └──────────────┘
          │                            │
          ▼                            ▼
   ┌──────────────┐            ┌──────────────┐
   │  Beijing DB  │ ──sync──→  │  Shanghai DB │
   │  (Primary)   │            │  (Replica)    │
   └──────────────┘            └──────────────┘

RTO/RPO Design

Metric	Meaning	Target
RTO	Recovery Time Objective	< 5min
RPO	Recovery Point Objective	< 1min

Failover Procedure

# 1. Detect failure (Prometheus/Blackbox Exporter)
# Beijing cluster fails health check 3 consecutive times

# 2. DNS switch (Route53 Health Check auto-switch)
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123ABC \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "api.example.com",
        "Type": "CNAME",
        "TTL": 60,
        "ResourceRecords": [{"Value": "shanghai-lb.example.com"}]
      }
    }]
  }'

# 3. Promote standby cluster database to primary
# Execute DB failover on Shanghai cluster

# 4. Scale up on standby cluster
kubectl scale deployment myapp -n production --replicas=10 --kubeconfig=shanghai.kubeconfig

# 5. Verify service
curl -f https://api.example.com/health || echo "FAIL"

Automatic Failover

Use Karmada’s failover feature for automatic switching:

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: myapp-failover-policy
spec:
  resourceSelectors:
  - apiVersion: apps/v1
    kind: Deployment
    name: myapp
  placement:
    clusterAffinity:
      clusterNames:
      - cluster-beijing
      - cluster-shanghai
  failover:
    application:
      decisionConditions:
        maxUnavailable: 50%
      gracePeriodSeconds: 180
      purgeMode: Graceful

Data Sync Strategy

Data Type	Sync Method	RPO
Database	Master-slave replication / CDC	Seconds
Object storage	Cross-region replication	Seconds
Configuration	GitOps sync	Minutes
Cache	Per-cluster independent	No sync (can be lost)

Recovery Drills

Drill Process

1. Simulate primary cluster failure in non-production environment
2. Verify DNS switch time
3. Verify standby cluster database promotion time
4. Verify application startup time
5. Verify service recovery time
6. Verify data consistency
7. Verify failback procedure

Chaos Engineering

# Use Chaos Mesh to simulate cluster failure
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: kill-api-pods
  namespace: production
spec:
  action: pod-kill
  mode: all
  selector:
    namespaces:
    - production
    labelSelectors:
      app: api-server
  scheduler:
    cron: "@every 1h"

Drill Checklist

DNS switch completes within 60 seconds
Standby cluster application starts within 3 minutes
Database failover completes within 1 minute
RTO < 5 minutes
RPO < 1 minute
No data loss
Failback procedure works
Monitoring alerts trigger correctly

Multi-Cluster Observability

Unified Monitoring Architecture

Per-cluster Prometheus → Thanos / VictoriaMetrics → Grafana

# Thanos Receive configuration
apiVersion: monitoring.thanos.io/v1alpha1
kind: ThanosReceive
metadata:
  name: thanos-receive
spec:
  replicas: 3
  tsdbVolume:
    storageClass: fast-ssd
    size: 100Gi
  tsdbRetention: 15d
  configReloader:
    enabled: true

Cross-Cluster Logging

Per-cluster Fluentbit → Central Loki / Elasticsearch

Multi-Cluster Monitoring Dashboard

# Grafana datasource configuration
apiVersion: 1
datasources:
- name: Thanos
  type: prometheus
  url: http://thanos-query.monitoring:9090
  isDefault: true
  jsonData:
    timeInterval: "30s"

Key multi-cluster monitoring metrics:

Metric	Meaning
Total Pods per cluster	Cluster scale
Node resource utilization per cluster	Capacity planning
Cross-cluster service latency	DR failover indicator
Pending Pods per cluster	Scaling signal
Argo CD sync status	Deployment health

Summary

Multi-cluster management is one of the most complex areas in the K8s ecosystem. Key takeaways:

Don’t adopt multi-cluster prematurely: If a single cluster can handle the load, don’t go multi-cluster. Multi-cluster management costs 3-5x that of a single cluster.
Choose based on needs: Use Cluster API for cluster lifecycle management, Karmada for cross-cluster orchestration, Argo CD for GitOps deployment.
GitOps is the deployment standard: In multi-cluster environments, manual deployment is unsustainable. Argo CD + Kustomize is a proven combination.
DR requires drills: An untested DR plan equals no DR. Conduct full failover drills at least quarterly.
Unified observability: Multi-cluster monitoring data must be aggregated to a single pane, otherwise troubleshooting wastes time switching between clusters.
Data sync is critical: Application switching is easy; data sync is hard. Database replication and object storage cross-region replication need advance planning.
DNS is the first line of defense: Global DNS load balancing is the simplest traffic switching mechanism. Set TTL short (60 seconds) for faster switching.

Multi-cluster is not a silver bullet—it trades higher complexity for better fault isolation and scalability. Before deciding on multi-cluster, confirm that a single cluster truly can’t meet your needs.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

KubeFed Archive Notice — GitHub, referenced for KubeFed Archive Notice

Overview#

Why Multi-Cluster#

Single Cluster Bottlenecks#

Typical Multi-Cluster Scenarios#

Multi-Cluster Architecture Patterns#

Pattern 1: Hub-Spoke#

Pattern 2: Federation#

Pattern 3: Mesh#

Mainstream Multi-Cluster Management Tools#

Tool Overview#

KubeFed (Archived)#

Cluster API#

Karmada#

OCM (Open Cluster Management)#

Argo CD ApplicationSet#

Tool Selection Guide#

Cross-Cluster Service Discovery#

Option 1: Global DNS#

Option 2: Service Mesh Multi-Cluster#

Option 3: Submariner#

Comparison#

Unified CI/CD#

GitOps Multi-Cluster Deployment#

Configuration Diff Management#

Disaster Recovery Failover#

Multi-Region DR Architecture#

RTO/RPO Design#

Failover Procedure#

Automatic Failover#

Data Sync Strategy#

Recovery Drills#

Drill Process#

Chaos Engineering#

Drill Checklist#

Multi-Cluster Observability#

Unified Monitoring Architecture#

Cross-Cluster Logging#

Multi-Cluster Monitoring Dashboard#

Summary#

References & Acknowledgments#