Container Persistent Storage Solution Selection

K8s Storage System: Three-Layer Abstraction

The Kubernetes storage system decouples storage consumers from providers through three layers of abstraction. This is the core concept for understanding container persistent storage. According to the K8s storage documentation, the three-layer structure is:

┌──────────────────────────────────────────────────────┐
│                   Pod (Consumer)                      │
│                 volumeMounts → volumes                │
├──────────────────────────────────────────────────────┤
│              PVC (Claim) — User requests storage      │
│           "I need 10Gi RWO storage"                   │
├──────────────────────────────────────────────────────┤
│           StorageClass (Dynamic Provisioning)          │
│         "Use ceph-rbd driver, reclaim: Retain"       │
├──────────────────────────────────────────────────────┤
│              PV (Physical Resource) — Actual storage  │
│          "10.0.0.5:/data/pvc-xxx (NFS)"              │
└──────────────────────────────────────────────────────┘

PV (PersistentVolume)

PV is a cluster-level resource representing an abstraction of physical storage. PVs can be created manually by administrators or automatically provisioned through StorageClass:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nfs-data
spec:
  capacity:
    storage: 50Gi
  accessModes:
  - ReadWriteMany           # Multi-node read-write
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 10.0.0.5
    path: /data/k8s-share
  storageClassName: ""      # Static PV doesn't bind to StorageClass

PVC (PersistentVolumeClaim)

PVC is a namespace-level storage claim. Users request storage through PVCs, and the system automatically matches PVs that meet the requirements:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data-pvc
  namespace: production
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: ceph-rbd
  resources:
    requests:
      storage: 20Gi

StorageClass (Dynamic Provisioning)

StorageClass is the core of dynamic storage provisioning, binding CSI drivers and parameter templates to create PVs on demand:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd
provisioner: rbd.csi.ceph.com
reclaimPolicy: Retain
allowVolumeExpansion: true   # Support online expansion
parameters:
  clusterID: ceph-cluster-1
  pool: k8s-rbd-pool
  imageFormat: "2"
  imageFeatures: layering
  csi.storage.k8s.io/provisioner-secret-name: ceph-secret
  csi.storage.k8s.io/provisioner-secret-namespace: ceph-system

AccessMode Explained

AccessMode	Abbreviation	Meaning	Typical Use Case
ReadWriteOnce	RWO	Single-node read-write	MySQL, PostgreSQL
ReadOnlyMany	ROX	Multi-node read-only	Config files, static assets
ReadWriteMany	RWX	Multi-node read-write	NFS, CephFS, shared data
ReadWriteOncePod	RWOP	Single-Pod read-write	K8s 1.22+, Pod-level precision

CSI Driver Mechanism

CSI (Container Storage Interface) is the standard interface between K8s and storage backends, replacing the earlier in-tree storage plugins. According to the CSI specification, CSI drivers interact with the K8s control plane through Sidecar components.

CSI Architecture

┌───────────────┐    gRPC     ┌──────────────┐
│  K8s Control   │◄──────────►│ CSI Sidecar  │
│  Plane (PV/PVC)│            │ (provisioner │
└───────────────┘            │  attacher)   │
                             └──────┬───────┘
                                    │ gRPC
                             ┌──────▼───────┐
                             │  CSI Driver  │
                             │  (Storage    │
                             │   Backend)   │
                             └──────────────┘

The CSI driver works through three Sidecar components:

external-provisioner: Watches PVC events, calls the CSI driver to create/delete storage volumes
external-attacher: Manages VolumeAttachment, attaches/detaches storage to Nodes
external-snapshotter: Supports storage snapshot functionality

Common CSI Drivers

NFS CSI

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-csi
provisioner: nfs.csi.k8s.io
parameters:
  server: 10.0.0.5
  share: /data/k8s-share
  subDir: ${pvc.metadata.namespace}/${pvc.name}
reclaimPolicy: Delete
volumeBindingMode: Immediate

NFS is suitable for shared file scenarios. Its advantage is native RWX support; its disadvantage is that network performance depends on the NFS server.

Ceph RBD CSI

Ceph RBD provides block storage, suitable for high-IO scenarios like databases:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd-fast
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: ceph-cluster-1
  pool: ssd-pool              # Use SSD pool
  imageFormat: "2"
  imageFeatures: layering
  csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Retain
allowVolumeExpansion: true
mountOptions:
  - discard                   # Enable TRIM

Local Path Provisioner

Rancher Local Path Provisioner uses Node local disks, suitable for low-latency, high-throughput scenarios:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path
provisioner: rancher.io/local-path
volumeBindingMode: WaitForFirstConsumer  # Delay binding until Pod is scheduled
reclaimPolicy: Delete

WaitForFirstConsumer is a critical setting for Local Volumes—it ensures the PV is created only after a Pod is scheduled to a specific Node, avoiding scheduling conflicts.

Storage Selection Matrix

Different storage solutions have trade-offs in performance, cost, and portability:

Storage Type	Performance	Cost	Portability	RWX	Use Case
Local Path	Very high	Very low	Poor	No	Cache, temp data, single-replica databases
NFS	Medium	Low	Medium	Yes	Shared files, config sync
Ceph RBD	High	Medium	Medium	No	Databases, block devices
CephFS	Medium-high	Medium	Medium	Yes	Shared storage, big data
Cloud disk (EBS/PD)	High	High	Poor	No	Cloud databases
Longhorn	Medium-high	Low	Medium	No	K8s-native distributed storage

Selection Decision Tree

Need RWX (multiple Pods read-write simultaneously)?
├─ Yes → NFS (simple) / CephFS (high performance)
└─ No → Need high IOPS?
         ├─ Yes → Local Path (single node) / Ceph RBD SSD (distributed)
         └─ No → Cost sensitive?
                  ├─ Yes → Local Path / Longhorn
                  └─ No → Ceph RBD / Cloud disk

Stateful Application Deployment Considerations

MySQL Deployment

MySQL is a typical stateful application requiring special attention to storage configuration:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql-headless
  replicas: 1               # Master-slave requires Operator management
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"
          limits:
            memory: "4Gi"
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
        - name: config
          mountPath: /etc/mysql/conf.d
      volumes:
      - name: config
        configMap:
          name: mysql-config
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: ceph-rbd
      resources:
        requests:
          storage: 50Gi

Key considerations:

Data safety: Use reclaimPolicy: Retain to prevent data loss from accidental PVC deletion
Resource limits: Memory limit should not be less than innodb_buffer_pool_size + 1GB
Scheduling affinity: Use nodeSelector or podAntiAffinity to pin to high-performance nodes
Backup strategy: Use Velero or storage snapshots for regular backups

Redis Deployment

Redis needs to distinguish between caching and persistence scenarios:

# Redis persistence mode requires AOF enabled
volumeClaimTemplates:
- metadata:
    name: redis-data
  spec:
    accessModes: ["ReadWriteOnce"]
    storageClassName: local-path    # Redis is IO-sensitive, prefer Local
    resources:
      requests:
        storage: 10Gi

Elasticsearch Deployment

ES has extremely high storage IO requirements, with each node needing an independent PV:

# ES Pod uses initContainer to optimize system parameters
initContainers:
- name: sysctl
  image: busybox
  command: ["sysctl", "-w", "vm.max_map_count=262144"]
  securityContext:
    privileged: true
containers:
- name: elasticsearch
  image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
  env:
  - name: ES_JAVA_OPTS
    value: "-Xms2g -Xmx2g"    # Heap memory = 50% of container memory
  volumeMounts:
  - name: data
    mountPath: /usr/share/elasticsearch/data

Storage Troubleshooting

PVC Stuck in Pending

# Check PVC events
kubectl describe pvc app-data-pvc -n production

# Common causes to investigate
# 1. StorageClass doesn't exist
kubectl get sc

# 2. No available PV (static provisioning)
kubectl get pv

# 3. StorageClass provisioner not running
kubectl get pods -n kube-system | grep csi

# 4. Storage backend connection failure (check CSI logs)
kubectl logs -n ceph-system ceph-csi-rbd-provisioner-xxx

PV Mount Failure

# Check Pod events for mount errors
kubectl describe pod <pod-name>

# Common errors and actions
# "MountVolume.MountDevice failed" → Storage backend unreachable
# "Unable to attach or mount volumes" → CSI Node plugin issue on the Node

# Log in to the target Node and check
# Check block devices
lsblk
# Check mount points
mount | grep <pv-name>
# Check CSI Node plugin
crictl ps | grep csi

Storage Expansion Failure

# Confirm StorageClass supports expansion
kubectl get sc ceph-rbd -o jsonpath='{.allowVolumeExpansion}'
# Output should be true

# Perform expansion
kubectl patch pvc app-data-pvc -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'

# Check expansion status
kubectl get pvc app-data-pvc -o jsonpath='{.status.conditions}'

Common Troubleshooting Quick Reference

Symptom	Possible Cause	Diagnostic Command
PVC Pending	SC doesn’t exist / Provisioner not running	`kubectl describe pvc`
Pod ContainerCreating	PV mount failure	`kubectl describe pod`
Pod event “disk pressure”	Node disk full	`kubectl describe node`
High IO latency	Poor storage backend performance	`iostat -x 1`
PVC expansion no response	SC doesn’t have expansion enabled	`kubectl get sc`

Summary

The core of container persistent storage lies in understanding the three-layer abstraction, making informed selections, and preventing failures. The three-layer abstraction (PV/PVC/StorageClass) decouples consumption from provisioning, and CSI standardizes the driver interface. When selecting storage, you need to balance performance, cost, and portability—Local Path for ultimate performance at the cost of portability, Ceph RBD for a balanced distributed block storage solution, and NFS as an economical choice for shared storage.

Key principles for production: data safety first (Retain policy), monitor storage health, and maintain a solid backup strategy. Regularly verifying backup recoverability is more important than the storage selection itself.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

K8s storage documentation — Kubernetes Official, referenced for K8s storage documentation
CSI specification — Kubernetes-csi, referenced for CSI specification
Rancher Local Path Provisioner — GitHub, referenced for Rancher Local Path Provisioner

K8s Storage System: Three-Layer Abstraction#

PV (PersistentVolume)#

PVC (PersistentVolumeClaim)#

StorageClass (Dynamic Provisioning)#

AccessMode Explained#

CSI Driver Mechanism#

CSI Architecture#

Common CSI Drivers#

NFS CSI#

Ceph RBD CSI#

Local Path Provisioner#

Storage Selection Matrix#

Selection Decision Tree#

Stateful Application Deployment Considerations#

MySQL Deployment#

Redis Deployment#

Elasticsearch Deployment#

Storage Troubleshooting#

PVC Stuck in Pending#

PV Mount Failure#

Storage Expansion Failure#

Common Troubleshooting Quick Reference#

Summary#

References & Acknowledgments#