Kubernetes Operator Development Guide

Overview

A Kubernetes Operator is a pattern that packages the operational knowledge of a specific application (like deployment, scaling, backup, recovery) into automated controller software. Instead of manually editing YAML or running kubectl commands, Operators monitor custom resources and automatically reconcile the cluster toward the desired state.

This guide provides a deep dive into Operator development—from CRD design principles and Controller reconciliation loops to operator-sdk/kubebuilder hands-on development, testing, and release.

Based on Kubernetes v1.30, operator-sdk v1.37, kubebuilder v4. Use either tool; both produce equivalent results.

Operator Core Concepts

What Is an Operator

An Operator = Custom Resource Definition (CRD) + Controller.

┌─────────────────────────────────────────────────────────────┐
│                      Operator                                 │
│  ┌────────────────────┐    ┌────────────────────────────┐   │
│  │   Custom Resource   │    │       Controller            │   │
│  │   Definition (CRD)  │    │   ┌─────────────────────┐  │   │
│  │                      │    │   │   Watch CR changes   │  │   │   │
│  │   apiVersion: app/v1 │    │   └──────────┬──────────┘  │   │
│  │   kind: MyApp         │    │              ▼              │   │
│  │   spec:               │    │   ┌─────────────────────┐  │   │
│  │     replicas: 3       │    │   │   Compare & Diff     │  │   │
│  │     image: nginx      │    │   └──────────┬──────────┘  │   │
│  │     port: 80          │    │              ▼              │   │
│  │   status:             │    │   ┌─────────────────────┐  │   │
│  │     readyReplicas: 3  │    │   │   Reconcile (Act)    │  │   │
│  └────────────────────┘    │   └─────────────────────┘  │   │
│                              └────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘

Relationship with Built-in Controllers

K8s built-in controllers (Deployment, StatefulSet, etc.) are generic—they handle common patterns but can’t cover application-specific operational logic. Operators extend this pattern to any application:

Feature	Built-in Controller	Operator
Resource type	Built-in (Deployment, etc.)	Custom (CRD)
Domain logic	Generic	Application-specific
Operational knowledge	None	Encodes expert knowledge
Automation level	Basic lifecycle	Full lifecycle + domain ops
Example	Deploy Nginx	Deploy and manage PostgreSQL HA cluster

When to Use an Operator

Scenario	Use Operator?	Reason
Stateless web app	No	Deployment + HPA suffices
Database HA cluster	Yes	Requires complex failover, backup logic
Message queue cluster	Yes	Partition rebalancing, order constraints
ML training pipeline	Yes	Needs GPU scheduling, checkpoint management
Custom config management	Maybe	Consider ConfigMap + init container first
Multi-service app stack	Yes	Cross-service orchestration

Operator Capability Levels

The Operator Capability Model defines five levels:

Level	Name	Description	Example
1	Basic Install	Deploy and configure app	Create Deployment, Service
2	Seamless Upgrades	Managed upgrades	Rolling update, schema migration
3	Full Lifecycle	Backup, recovery, scaling	Periodic backup, restore
4	Deep Insights	Metrics, alerts	Prometheus metrics, health checks
5	Auto Pilot	Auto-tuning, auto-healing	Auto-scaling, auto-failover

CRD Design

CRD Structure

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: myapps.app.example.com
spec:
  group: app.example.com
  names:
    kind: MyApp
    listKind: MyAppList
    singular: myapp
    plural: myapps
    shortNames:
    - ma
  scope: Namespaced              # Namespaced or Cluster
  versions:
  - name: v1alpha1
    served: true
    storage: true                # Only one version can be storage=true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            required:
            - replicas
            - image
            properties:
              replicas:
                type: integer
                minimum: 1
                maximum: 100
                default: 1
              image:
                type: string
              port:
                type: integer
                minimum: 1
                maximum: 65535
                default: 80
              resources:
                type: object
                properties:
                  requests:
                    type: object
                    properties:
                      cpu:
                        type: string
                      memory:
                        type: string
              env:
                type: array
                items:
                  type: object
                  properties:
                    name:
                      type: string
                    value:
                      type: string
          status:
            type: object
            properties:
              readyReplicas:
                type: integer
              phase:
                type: string
                enum:
                - Pending
                - Running
                - Failed
              conditions:
                type: array
                items:
                  type: object
                  properties:
                    type:
                      type: string
                    status:
                      type: string
                      enum: ["True", "False", "Unknown"]
                    lastTransitionTime:
                      type: string
                      format: date-time
                    reason:
                      type: string
                    message:
                      type: string
    subresources:
      status: {}                 # Enable /status subresource
      scale:                     # Enable /scale subresource
        specReplicasPath: .spec.replicas
        statusReplicasPath: .status.readyReplicas

Design Principles

Declarative, not imperative: Users declare desired state, not steps to execute.
idempotent: Re-applying the same spec should have no side effects.
status reflects reality: status fields must accurately reflect actual cluster state.
Don’t duplicate built-in resources: Don’t create another Deployment; extend or compose.
Version from the start: Start with v1alpha1, not v1.

API Versioning Strategy

Version	Description	Stability
`v1alpha1`	Initial development	May break
`v1beta1`	Pre-release, likely stable	Minor changes
`v1`	Stable GA	No breaking changes

Conversion between versions is handled by a Conversion Webhook.

Controller Principles

Reconciliation Loop

The Controller’s core pattern is the Reconciliation Loop (Reconcile loop):

┌─────────────────────────────────────────────────────────────┐
│                     Reconcile Loop                            │
│                                                              │
│   ┌──────────────┐                                           │
│   │  Watch events │ ← CR changes, managed resource changes,  │
│   │  (Watch)      │   periodic resync                        │
│   └──────┬───────┘                                           │
│          ▼                                                   │
│   ┌──────────────┐                                           │
│   │  Read CR spec │ ← Desired state                          │
│   └──────┬───────┘                                           │
│          ▼                                                   │
│   ┌──────────────┐                                           │
│   │  Read cluster │ ← Actual state                           │
│   │  resources    │                                          │
│   └──────┬───────┘                                           │
│          ▼                                                   │
│   ┌──────────────┐                                           │
│   │  Diff & Decide│                                          │
│   │  what to do   │                                          │
│   └──────┬───────┘                                           │
│          ▼                                                   │
│   ┌──────────────┐                                           │
│  │  Act: Create/ │ → Create/Update/Delete resources         │
│  │  Update/Delete│                                          │
│   └──────┬───────┘                                           │
│          ▼                                                   │
│   ┌──────────────┐                                           │
│   │  Update status│ → Write actual state back to CR status  │
│   └──────────────┘                                           │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Reconcile Function

The Reconcile function is the Controller’s heart:

// Reconcile is part of the main kubernetes reconciliation loop
func (r *MyAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    logger := log.FromContext(ctx)

    // 1. Get the CR instance
    var myApp appv1.MyApp
    if err := r.Get(ctx, req.NamespacedName, &myApp); err != nil {
        if errors.IsNotFound(err) {
            return ctrl.Result{}, nil    // CR deleted, nothing to do
        }
        return ctrl.Result{}, err
    }

    // 2. Check if deletion is in progress
    if myApp.DeletionTimestamp.IsZero() {
        // Not being deleted — ensure finalizer exists
        if !controllerutil.ContainsFinalizer(&myApp, "myapp.finalizer.app.example.com") {
            controllerutil.AddFinalizer(&myApp, "myapp.finalizer.app.example.com")
            if err := r.Update(ctx, &myApp); err != nil {
                return ctrl.Result{}, err
            }
        }
    } else {
        // Being deleted — run cleanup logic
        return r.reconcileDelete(ctx, &myApp)
    }

    // 3. Reconcile to desired state
    if err := r.reconcileDeployment(ctx, &myApp); err != nil {
        return ctrl.Result{}, err
    }

    if err := r.reconcileService(ctx, &myApp); err != nil {
        return ctrl.Result{}, err
    }

    // 4. Update status
    if err := r.updateStatus(ctx, &myApp); err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
}

Reconcile Key Principles

Don’t pass req to helper functions: Pass CR instance explicitly.
Don’t return too early: Every branch should return a Result.
Always update status: Reflect actual state in CR status.
Handle not-found gracefully: Resource may have been deleted.
Use RequeueAfter for periodic checks: For polling external state.

Return Value	Meaning
`ctrl.Result{}, nil`	Done, no requeue
`ctrl.Result{Requeue: true}, nil`	Requeue immediately
`ctrl.Result{RequeueAfter: 30s}, nil`	Requeue after 30s
`ctrl.Result{}, err`	Error, requeue with backoff

Level-Driven vs Edge-Triggered

K8s controllers are level-driven, not edge-triggered:

Edge-triggered: Act only on events (e.g., when CR changes, do X)
Level-driven:   Always compare desired state vs actual state, fix any drift

This means even without events, the Controller periodically reconciles to catch any drift caused by external changes.

Development with operator-sdk

Project Initialization

# Install operator-sdk
curl -LO https://github.com/operator-framework/operator-sdk/releases/download/v1.37.0/operator-sdk_linux_amd64
chmod +x operator-sdk_linux_amd64
sudo mv operator-sdk_linux_amd64 /usr/local/bin/operator-sdk

# Create project
mkdir myapp-operator
cd myapp-operator
operator-sdk init \
  --domain example.com \
  --repo github.com/example/myapp-operator

# Create API (CRD + Controller)
operator-sdk create api \
  --group app \
  --version v1alpha1 \
  --kind MyApp \
  --resource \
  --controller

Project Structure

myapp-operator/
├── api/
│   └── v1alpha1/
│       ├── myapp_types.go          # CRD type definitions
│       ├── groupversion_info.go    # GroupVersion info
│       ├── zz_generated.deepcopy.go # Auto-generated DeepCopy methods
│       └── zz_generated.openapi.go  # Auto-generated OpenAPI schemas
├── cmd/
│   └── main.go                      # Controller Manager entry point
├── config/
│   ├── crd/                         # CRD manifests
│   ├── default/                     # Default config
│   ├── manager/                     # Controller Manager deployment
│   ├── manifests/                   # Kustomize manifests
│   ├── prometheus/                  # ServiceMonitor
│   ├── rbac/                        # RBAC permissions
│   └── samples/                     # CR samples
├── internal/
│   └── controller/
│       ├── myapp_controller.go      # Controller implementation
│       └── myapp_controller_test.go  # Controller tests
├── Dockerfile
├── Makefile
├── go.mod
└── PROJECT

Type Definitions

// api/v1alpha1/myapp_types.go
package v1alpha1

import (
    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// MyAppSpec defines the desired state of MyApp
type MyAppSpec struct {
    // +kubebuilder:validation:Minimum=1
    // +kubebuilder:validation:Maximum=100
    // +kubebuilder:default=1
    Replicas int32 `json:"replicas"`

    // +kubebuilder:validation:Required
    Image string `json:"image"`

    // +kubebuilder:validation:Minimum=1
    // +kubebuilder:validation:Maximum=65535
    // +kubebuilder:default=80
    Port int32 `json:"port,omitempty"`

    Resources corev1.ResourceRequirements `json:"resources,omitempty"`

    // +kubebuilder:validation:Enum=RollingUpdate;Recreate
    // +kubebuilder:default=RollingUpdate
    Strategy string `json:"strategy,omitempty"`
}

// MyAppStatus defines the observed state of MyApp
type MyAppStatus struct {
    ReadyReplicas int32 `json:"readyReplicas"`

    // +kubebuilder:validation:Enum=Pending;Running;Failed
    Phase string `json:"phase,omitempty"`

    Conditions []metav1.Condition `json:"conditions,omitempty"`
}

//+kubebuilder:object:root=true
//+kubebuilder:subresource:status
//+kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.readyReplicas

type MyApp struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   MyAppSpec   `json:"spec,omitempty"`
    Status MyAppStatus `json:"status,omitempty"`
}

//+kubebuilder:object:root=true

type MyAppList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []MyApp `json:"items"`
}

func init() {
    SchemeBuilder.Register(&MyApp{}, &MyAppList{})
}

Controller Implementation

// internal/controller/myapp_controller.go
package controller

import (
    "context"
    "fmt"
    "time"

    appv1 "github.com/example/myapp-operator/api/v1alpha1"
    "github.com/go-logr/logr"
    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/api/errors"
    "k8s.io/apimachinery/pkg/api/resource"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/util/intstr"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
    "sigs.k8s.io/controller-runtime/pkg/log"
)

const finalizerName = "myapp.finalizer.app.example.com"

type MyAppReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=app.example.com,resources=myapps,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=app.example.com,resources=myapps/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=app.example.com,resources=myapps/finalizers,verbs=update
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups="",resources=services,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups="",resources=events,verbs=create;patch

func (r *MyAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    logger := log.FromContext(ctx)

    // 1. Get MyApp instance
    var myApp appv1.MyApp
    if err := r.Get(ctx, req.NamespacedName, &myApp); err != nil {
        if errors.IsNotFound(err) {
            logger.Info("MyApp resource not found, ignoring")
            return ctrl.Result{}, nil
        }
        logger.Error(err, "failed to get MyApp")
        return ctrl.Result{}, err
    }

    // 2. Handle finalizer
    if myApp.DeletionTimestamp.IsZero() {
        if !controllerutil.ContainsFinalizer(&myApp, finalizerName) {
            controllerutil.AddFinalizer(&myApp, finalizerName)
            if err := r.Update(ctx, &myApp); err != nil {
                return ctrl.Result{}, err
            }
            return ctrl.Result{Requeue: true}, nil
        }
    } else {
        // Being deleted — run cleanup
        if controllerutil.ContainsFinalizer(&myApp, finalizerName) {
            if err := r.cleanupResources(ctx, &myApp); err != nil {
                return ctrl.Result{}, err
            }
            controllerutil.RemoveFinalizer(&myApp, finalizerName)
            if err := r.Update(ctx, &myApp); err != nil {
                return ctrl.Result{}, err
            }
        }
        return ctrl.Result{}, nil
    }

    // 3. Reconcile Deployment
    if err := r.reconcileDeployment(ctx, &myApp); err != nil {
        logger.Error(err, "failed to reconcile Deployment")
        r.setCondition(&myApp, "Ready", "False", "DeploymentReconcileFailed", err.Error())
        _ = r.updateStatus(ctx, &myApp)
        return ctrl.Result{RequeueAfter: 10 * time.Second}, nil
    }

    // 4. Reconcile Service
    if err := r.reconcileService(ctx, &myApp); err != nil {
        logger.Error(err, "failed to reconcile Service")
        return ctrl.Result{}, err
    }

    // 5. Update status
    if err := r.updateStatus(ctx, &myApp); err != nil {
        logger.Error(err, "failed to update status")
        return ctrl.Result{}, err
    }

    return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
}

func (r *MyAppReconciler) reconcileDeployment(ctx context.Context, myApp *appv1.MyApp) error {
    logger := log.FromContext(ctx)

    var deployment appsv1.Deployment
    err := r.Get(ctx, client.ObjectKey{
        Name:      myApp.Name,
        Namespace: myApp.Namespace,
    }, &deployment)

    if errors.IsNotFound(err) {
        // Create new Deployment
        newDeploy := r.buildDeployment(myApp)
        if err := controllerutil.SetControllerReference(myApp, newDeploy, r.Scheme); err != nil {
            return err
        }
        logger.Info("Creating Deployment", "name", newDeploy.Name)
        return r.Create(ctx, newDeploy)
    }
    if err != nil {
        return err
    }

    // Update existing Deployment
    updated := r.buildDeployment(myApp)
    deployment.Spec = updated.Spec
    logger.Info("Updating Deployment", "name", deployment.Name)
    return r.Update(ctx, &deployment)
}

func (r *MyAppReconciler) buildDeployment(myApp *appv1.MyApp) *appsv1.Deployment {
    replicas := myApp.Spec.Replicas
    return &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      myApp.Name,
            Namespace: myApp.Namespace,
            Labels: map[string]string{
                "app.kubernetes.io/name":     "myapp",
                "app.kubernetes.io/instance": myApp.Name,
            },
        },
        Spec: appsv1.DeploymentSpec{
            Replicas: &replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: map[string]string{
                    "app.kubernetes.io/name":     "myapp",
                    "app.kubernetes.io/instance": myApp.Name,
                },
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: map[string]string{
                        "app.kubernetes.io/name":     "myapp",
                        "app.kubernetes.io/instance": myApp.Name,
                    },
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{
                        {
                            Name:  "app",
                            Image: myApp.Spec.Image,
                            Ports: []corev1.ContainerPort{
                                {
                                    ContainerPort: myApp.Spec.Port,
                                    Protocol:      corev1.ProtocolTCP,
                                },
                            },
                            Resources: myApp.Spec.Resources,
                        },
                    },
                },
            },
        },
    }
}

func (r *MyAppReconciler) reconcileService(ctx context.Context, myApp *appv1.MyApp) error {
    var svc corev1.Service
    err := r.Get(ctx, client.ObjectKey{
        Name:      myApp.Name,
        Namespace: myApp.Namespace,
    }, &svc)

    if errors.IsNotFound(err) {
        newSvc := r.buildService(myApp)
        if err := controllerutil.SetControllerReference(myApp, newSvc, r.Scheme); err != nil {
            return err
        }
        return r.Create(ctx, newSvc)
    }
    if err != nil {
        return err
    }

    // Update Service ports if changed
    updated := r.buildService(myApp)
    svc.Spec.Ports = updated.Spec.Ports
    return r.Update(ctx, &svc)
}

func (r *MyAppReconciler) buildService(myApp *appv1.MyApp) *corev1.Service {
    return &corev1.Service{
        ObjectMeta: metav1.ObjectMeta{
            Name:      myApp.Name,
            Namespace: myApp.Namespace,
            Labels: map[string]string{
                "app.kubernetes.io/name":     "myapp",
                "app.kubernetes.io/instance": myApp.Name,
            },
        },
        Spec: corev1.ServiceSpec{
            Selector: map[string]string{
                "app.kubernetes.io/name":     "myapp",
                "app.kubernetes.io/instance": myApp.Name,
            },
            Ports: []corev1.ServicePort{
                {
                    Port:       myApp.Spec.Port,
                    TargetPort: intstr.FromInt(int(myApp.Spec.Port)),
                    Protocol:   corev1.ProtocolTCP,
                },
            },
            Type: corev1.ServiceTypeClusterIP,
        },
    }
}

func (r *MyAppReconciler) updateStatus(ctx context.Context, myApp *appv1.MyApp) error {
    // Read Deployment to get ready replicas
    var deploy appsv1.Deployment
    err := r.Get(ctx, client.ObjectKey{
        Name:      myApp.Name,
        Namespace: myApp.Namespace,
    }, &deploy)
    if err != nil && !errors.IsNotFound(err) {
        return err
    }

    readyReplicas := int32(0)
    if err == nil {
        readyReplicas = deploy.Status.ReadyReplicas
    }

    // Update status only if changed
    if myApp.Status.ReadyReplicas != readyReplicas {
        myApp.Status.ReadyReplicas = readyReplicas
        if readyReplicas == myApp.Spec.Replicas {
            myApp.Status.Phase = "Running"
            r.setCondition(myApp, "Ready", "True", "AllReplicasReady",
                fmt.Sprintf("All %d replicas are ready", readyReplicas))
        } else if readyReplicas > 0 {
            myApp.Status.Phase = "Pending"
            r.setCondition(myApp, "Ready", "False", "NotAllReplicasReady",
                fmt.Sprintf("%d/%d replicas ready", readyReplicas, myApp.Spec.Replicas))
        } else {
            myApp.Status.Phase = "Pending"
            r.setCondition(myApp, "Ready", "False", "NoReplicasReady",
                "No replicas are ready")
        }
        return r.Status().Update(ctx, myApp)
    }
    return nil
}

func (r *MyAppReconciler) setCondition(
    myApp *appv1.MyApp,
    conditionType string,
    status string,
    reason string,
    message string,
) {
    metav1.SetStatusCondition(&myApp.Status.Conditions, metav1.Condition{
        Type:               conditionType,
        Status:             metav1.ConditionStatus(status),
        Reason:             reason,
        Message:            message,
        LastTransitionTime: metav1.Now(),
    })
}

func (r *MyAppReconciler) cleanupResources(ctx context.Context, myApp *appv1.MyApp) error {
    // Delete Deployment
    var deploy appsv1.Deployment
    if err := r.Delete(ctx, &deploy, &client.DeleteOptions{
        Preconditions: &metav1.Preconditions{},
    }); err != nil && !errors.IsNotFound(err) {
        return err
    }
    return nil
}

// SetupWithManager sets up the controller with the Manager
func (r *MyAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&appv1.MyApp{}).
        Owns(&appsv1.Deployment{}).
        Owns(&corev1.Service{}).
        Complete(r)
}

main.go Configuration

// cmd/main.go
package main

import (
    "flag"
    "os"

    appv1 "github.com/example/myapp-operator/api/v1alpha1"
    "github.com/example/myapp-operator/internal/controller"
    "k8s.io/apimachinery/pkg/runtime"
    utilruntime "k8s.io/apimachinery/pkg/util/runtime"
    clientgoscheme "k8s.io/client-go/kubernetes/scheme"
    _ "k8s.io/client-go/plugin/pkg/client/auth"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/healthz"
    "sigs.k8s.io/controller-runtime/pkg/log/zap"
    metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
)

var scheme = runtime.NewScheme()

func init() {
    utilruntime.Must(clientgoscheme.AddToScheme(scheme))
    utilruntime.Must(appv1.AddToScheme(scheme))
}

func main() {
    var metricsAddr string
    var enableLeaderElection bool
    var probeAddr string
    flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "Metrics address")
    flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "Probe address")
    flag.BoolVar(&enableLeaderElection, "leader-elect", false,
        "Enable leader election for controller manager")
    opts := zap.Options{
        Development: true,
    }
    opts.BindFlags(flag.CommandLine)
    flag.Parse()

    ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))

    mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
        Scheme:                 scheme,
        Metrics:                metricsserver.Options{BindAddress: metricsAddr},
        HealthProbeBindAddress: probeAddr,
        LeaderElection:         enableLeaderElection,
        LeaderElectionID:       "myapp-operator.app.example.com",
    })
    if err != nil {
        setupLog.Error(err, "unable to start manager")
        os.Exit(1)
    }

    if err := (&controller.MyAppReconciler{
        Client: mgr.GetClient(),
        Scheme: mgr.GetScheme(),
    }).SetupWithManager(mgr); err != nil {
        setupLog.Error(err, "unable to create controller", "controller", "MyApp")
        os.Exit(1)
    }

    if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
        setupLog.Error(err, "unable to set up health check")
        os.Exit(1)
    }
    if err := mgr.AddReadyzCheck("readyz", healthz.Ping); err != nil {
        setupLog.Error(err, "unable to set up ready check")
        os.Exit(1)
    }

    setupLog.Info("starting manager")
    if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
        setupLog.Error(err, "problem running manager")
        os.Exit(1)
    }
}

Finalizer

What Is a Finalizer

A Finalizer is a special marker that prevents resource deletion until cleanup completes:

1. User deletes CR
2. K8s sets DeletionTimestamp but does NOT delete resource
3. Controller sees DeletionTimestamp, runs cleanup logic
4. Controller removes finalizer
5. K8s deletes resource

Finalizer Implementation

const finalizerName = "myapp.finalizer.app.example.com"

func (r *MyAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // ...

    if myApp.DeletionTimestamp.IsZero() {
        // Not being deleted — ensure finalizer exists
        if !controllerutil.ContainsFinalizer(&myApp, finalizerName) {
            controllerutil.AddFinalizer(&myApp, finalizerName)
            if err := r.Update(ctx, &myApp); err != nil {
                return ctrl.Result{}, err
            }
        }
    } else {
        // Being deleted — run cleanup
        if controllerutil.ContainsFinalizer(&myApp, finalizerName) {
            // Run cleanup logic (e.g., delete external resources)
            if err := r.cleanupExternalResources(ctx, &myApp); err != nil {
                return ctrl.Result{}, err
            }

            // Remove finalizer
            controllerutil.RemoveFinalizer(&myApp, finalizerName)
            if err := r.Update(ctx, &myApp); err != nil {
                return ctrl.Result{}, err
            }
        }
        return ctrl.Result{}, nil
    }

    // ... normal reconciliation
}

func (r *MyAppReconciler) cleanupExternalResources(ctx context.Context, myApp *appv1.MyApp) error {
    // Clean up external resources (e.g., cloud load balancers, DNS records, databases)
    // ...
    return nil
}

Note: If finalizer removal fails, the CR gets stuck in Terminating state. Ensure cleanup logic is robust.

Leader Election

Why Leader Election

In production, the Controller Manager runs with multiple replicas for HA. If all replicas reconcile simultaneously, conflicts occur. Leader election ensures only one active controller:

3 replicas → 1 Leader (active) + 2 Standby (idle)
Leader fails → One Standby becomes new Leader

Enabling Leader Election

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
    // ...
    LeaderElection:   true,
    LeaderElectionID: "myapp-operator.app.example.com",
    // Lease parameters
    LeaseDuration: ptr.To(15 * time.Second),  // Lease duration
    RenewDeadline:  ptr.To(10 * time.Second),  // Renew interval
    RetryPeriod:    ptr.To(2 * time.Second),   // Retry interval
})

# config/manager/manager.yaml
spec:
  replicas: 3    # Run 3 replicas for HA
  template:
    spec:
      containers:
      - command:
        - /manager
        args:
        - --leader-elect    # Enable leader election
        - --leader-election-id=myapp-operator.app.example.com

Testing

Unit Testing

// internal/controller/myapp_controller_test.go
package controller

import (
    "context"
    "testing"

    appv1 "github.com/example/myapp-operator/api/v1alpha1"
    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/api/resource"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/types"
    "k8s.io/client-go/kubernetes/scheme"
    "sigs.k8s.io/controller-runtime/pkg/client/fake"
    "sigs.k8s.io/controller-runtime/pkg/reconcile"
)

func TestReconcile_Create(t *testing.T) {
    // Create a fake MyApp CR
    myApp := &appv1.MyApp{
        ObjectMeta: metav1.ObjectMeta{
            Name:      "test-app",
            Namespace: "default",
        },
        Spec: appv1.MyAppSpec{
            Replicas: 3,
            Image:    "nginx:1.25",
            Port:     80,
            Resources: corev1.ResourceRequirements{
                Requests: corev1.ResourceList{
                    corev1.ResourceCPU:    resource.MustParse("100m"),
                    corev1.ResourceMemory: resource.MustParse("128Mi"),
                },
            },
        },
    }

    // Build fake client
    cl := fake.NewClientBuilder().
        WithScheme(scheme.Scheme).
        WithObjects(myApp).
        WithStatusSubresource(&appv1.MyApp{}).
        Build()

    r := &MyAppReconciler{
        Client: cl,
        Scheme: scheme.Scheme,
    }

    // Call Reconcile
    req := reconcile.Request{
        NamespacedName: types.NamespacedName{
            Name:      "test-app",
            Namespace: "default",
        },
    }
    _, err := r.Reconcile(context.Background(), req)
    if err != nil {
        t.Fatalf("reconcile failed: %v", err)
    }

    // Verify Deployment was created
    var deploy appsv1.Deployment
    err = cl.Get(context.Background(), req.NamespacedName, &deploy)
    if err != nil {
        t.Fatalf("failed to get Deployment: %v", err)
    }
    if *deploy.Spec.Replicas != 3 {
        t.Errorf("expected replicas=3, got %d", *deploy.Spec.Replicas)
    }

    // Verify Service was created
    var svc corev1.Service
    err = cl.Get(context.Background(), req.NamespacedName, &svc)
    if err != nil {
        t.Fatalf("failed to get Service: %v", err)
    }
}

Envtest Integration Testing

# Install envtest (sets up a real etcd + kube-apiserver)
make envtest

# Run integration tests
make test

// internal/controller/myapp_controller_test.go
func TestReconcile_WithEnvtest(t *testing.T) {
    // Start test environment
    testEnv := &envtest.Environment{
        CRDDirectoryPaths:     []string{filepath.Join("..", "..", "config", "crd", "bases")},
        ErrorIfCRDPathMissing: true,
    }
    cfg, _ := testEnv.Start()
    defer testEnv.Stop()

    // Create Manager
    scheme := runtime.NewScheme()
    _ = appv1.AddToScheme(scheme)
    _ = clientgoscheme.AddToScheme(scheme)

    k8sClient, _ := client.New(cfg, client.Options{Scheme: scheme})

    // Create CR
    myApp := &appv1.MyApp{...}
    _ = k8sClient.Create(context.Background(), myApp)

    // Run Reconcile
    // ...
}

Build and Deploy

Build the Image

# Generate manifests
make manifests

# Generate code (DeepCopy etc.)
make generate

# Build and push image
make docker-build docker-push IMG=harbor.example.com/myapp-operator:v0.1.0

Deploy to Cluster

# Install CRD
make install

# Deploy Controller Manager
make deploy IMG=harbor.example.com/myapp-operator:v0.1.0

# Verify
kubectl get pods -n myapp-operator-system
kubectl get crd myapps.app.example.com

Create a Custom Resource

# config/samples/app_v1alpha1_myapp.yaml
apiVersion: app.example.com/v1alpha1
kind: MyApp
metadata:
  name: myapp-sample
spec:
  replicas: 3
  image: nginx:1.25-alpine
  port: 80
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 200m
      memory: 256Mi

kubectl apply -f config/samples/app_v1alpha1_myapp.yaml

# Verify
kubectl get myapp myapp-sample -o yaml
kubectl get deployment myapp-sample
kubectl get svc myapp-sample

Observability

Metrics

import (
    "sigs.k8s.io/controller-runtime/pkg/metrics"
    "github.com/prometheus/client_golang/prometheus"
)

var (
    reconcileTotal = prometheus.NewCounter(
        prometheus.CounterOpts{
            Name: "myapp_reconcile_total",
            Help: "Total number of reconciliations",
        },
    )
    reconcileErrors = prometheus.NewCounter(
        prometheus.CounterOpts{
            Name: "myapp_reconcile_errors_total",
            Help: "Total number of reconciliation errors",
        },
    )
    reconcileDuration = prometheus.NewHistogram(
        prometheus.HistogramOpts{
            Name:    "myapp_reconcile_duration_seconds",
            Help:    "Reconciliation duration in seconds",
            Buckets: prometheus.DefBuckets,
        },
    )
)

func init() {
    metrics.Registry.MustRegister(reconcileTotal)
    metrics.Registry.MustRegister(reconcileErrors)
    metrics.Registry.MustRegister(reconcileDuration)
}

Events

import "k8s.io/client-go/tools/record"

type MyAppReconciler struct {
    client.Client
    Scheme   *runtime.Scheme
    Recorder record.EventRecorder
}

// In Reconcile:
r.Recorder.Eventf(&myApp, corev1.EventTypeNormal, "Created",
    "Deployment %s created", myApp.Name)

r.Recorder.Eventf(&myApp, corev1.EventTypeWarning, "Failed",
    "Failed to create Deployment: %v", err)

ServiceMonitor

# config/prometheus/monitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp-operator-metrics
  labels:
    app: myapp-operator
spec:
  selector:
    matchLabels:
      app: myapp-operator
  endpoints:
  - port: metrics
    path: /metrics
    interval: 30s

Common Patterns

Owner References

// Set owner reference so child resources are garbage-collected with parent
if err := controllerutil.SetControllerReference(myApp, deployment, r.Scheme); err != nil {
    return err
}

Status Subresource

// Update status separately from spec
// This prevents accidentally overwriting spec when updating status
if err := r.Status().Update(ctx, myApp); err != nil {
    return err
}

Optimistic Locking

// Avoid conflicts with concurrent updates
patch := client.MergeFrom(myApp.DeepCopy())
myApp.Status.ReadyReplicas = readyReplicas
if err := r.Status().Patch(ctx, myApp, patch); err != nil {
    return err
}

Event Filtering

// Only reconcile when relevant fields change
// Use predicates to filter events
func (r *MyAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&appv1.MyApp{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
        Owns(&appsv1.Deployment{}).
        Owns(&corev1.Service{}).
        Complete(r)
}

Common Pitfalls

Pitfall	Impact	Solution
Forgetting Finalizer	External resources leak	Add finalizer on creation
Status update overwrites spec	Configuration lost	Use `.Status().Update()`
No error handling in Reconcile	Silent failures	Log all errors
Non-idempotent operations	Duplicated resources	Check before create
Watching too many resources	Performance issues	Filter events
Ignoring context cancellation	Goroutine leak	Pass context
Overwriting OwnerReferences	Garbage collection fails	Use SetControllerReference

Summary

Operator development is a powerful pattern for automating K8s application management. Key takeaways:

CRD design is critical: Well-designed API is more important than Controller implementation. Follow declarative, idempotent, and versioned principles.
Reconcile loop is the core: Understand the level-driven reconciliation pattern. Don’t think in terms of events; think in terms of desired vs actual state.
Always use Finalizers: External resource cleanup is essential to prevent leaks.
Leader Election for HA: Production deployments must use leader election to avoid multi-active conflicts.
Test thoroughly: Use envtest for integration testing to catch real API behavior.
Observability from the start: Metrics, events, and logs are essential for production debugging.
Use operator-sdk/kubebuilder: Don’t write Controller from scratch—use scaffolding tools to reduce boilerplate.
Start simple, iterate: Begin with Level 1 (Basic Install) and add capabilities incrementally.

Operators are not a silver bullet—they’re appropriate for stateful, complex applications with significant operational knowledge. For simple stateless apps, Deployment + HPA is sufficient.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

Operator 模式文档 — Kubernetes Official, referenced for Operator 模式文档
Operator Capability Model — Operatorhub, referenced for Operator Capability Model

Overview#

Operator Core Concepts#

What Is an Operator#

Relationship with Built-in Controllers#

When to Use an Operator#

Operator Capability Levels#

CRD Design#

CRD Structure#

Design Principles#

API Versioning Strategy#

Controller Principles#

Reconciliation Loop#

Reconcile Function#

Reconcile Key Principles#

Level-Driven vs Edge-Triggered#

Development with operator-sdk#

Project Initialization#

Project Structure#

Type Definitions#

Controller Implementation#

main.go Configuration#

Finalizer#

What Is a Finalizer#

Finalizer Implementation#

Leader Election#

Why Leader Election#

Enabling Leader Election#

Testing#

Unit Testing#

Envtest Integration Testing#

Build and Deploy#

Build the Image#

Deploy to Cluster#

Create a Custom Resource#

Observability#

Metrics#

Events#

ServiceMonitor#

Common Patterns#

Owner References#

Status Subresource#

Optimistic Locking#

Event Filtering#

Common Pitfalls#

Summary#

References & Acknowledgments#