K8s Security Scanning and CIS Benchmark Compliance: A Complete Hardening Guide from kube-bench to Production

Overview

You manage a K8s cluster where the API Server has --anonymous-auth=true, etcd certificates are world-readable at 644, and the kubelet --read-only-port=10255 is still open. Each of these looks like a “minor issue” on its own, but an attacker who gets one foothold can pivot through the entire cluster. This isn’t hypothetical — the CNCF 2025 annual survey showed that over 90% of production K8s clusters have at least one CIS Benchmark-level configuration deficiency.

CIS (Center for Internet Security) Benchmark is an industry-recognized security configuration baseline, and Kubernetes has its own dedicated version. kube-bench is the tool that runs this baseline check — you run it, and it tells you which configurations don’t meet security standards and how to fix them.

But that’s just the first step. Running a report isn’t enough. You need to know how to remediate, how to monitor continuously, and how to embed scanning into your CI/CD pipeline. This article covers the complete path from installing kube-bench, running your first scan, interpreting reports, fixing common issues, to integrating Trivy for image vulnerability scanning, kube-hunter for penetration simulation, and automating security scanning into GitOps workflows.

What Is the CIS Kubernetes Benchmark

The CIS Benchmark is essentially a “security configuration checklist.” CIS brings together security experts to document how a system (OS, database, cloud platform, K8s, etc.) should be configured to be considered secure, with each rule having a clear check method and remediation recommendation.

The K8s CIS Benchmark organizes checks into four layers:

Each check has a severity level: L1 is a baseline requirement that any environment should meet; L2 is a stricter requirement for security-sensitive environments.

kube-bench turns this checklist into an automated tool. It reads K8s component configuration files, startup parameters, and certificate permissions, checks them against CIS Benchmark rules one by one, and outputs PASS/FAIL/WARN results.

Installing kube-bench

Option 1: Run Directly on the Node (Recommended for First Use)

# Download the latest version (as of July 2026, latest is v0.10.0)
curl -L https://github.com/aquasecurity/kube-bench/releases/download/v0.10.0/kube-bench_0.10.0_linux_amd64.tar.gz -o kube-bench.tar.gz
tar -zxvf kube-bench.tar.gz
sudo mv kube-bench /usr/local/bin/

# Verify
kube-bench version

For ARM64 environments (like Apple Silicon or AWS Graviton), replace linux_amd64 with linux_arm64.

Option 2: Run as a Job Inside the Cluster

apiVersion: batch/v1
kind: Job
metadata:
  name: kube-bench
  namespace: kube-system
spec:
  template:
    spec:
      hostPID: true
      containers:
        - name: kube-bench
          image: aquasec/kube-bench:0.10.0
          command: ["kube-bench"]
          args: ["--benchmark", "cis-1.10"]
          volumeMounts:
            - name: var-lib-etcd
              mountPath: /var/lib/etcd
              readOnly: true
            - name: etc-kubernetes
              mountPath: /etc/kubernetes
              readOnly: true
            - name: usr-bin
              mountPath: /usr/local/mount-from-host/bin
              readOnly: true
            - name: var-lib-kubelet
              mountPath: /var/lib/kubelet
              readOnly: true
            - name: etc-systemd
              mountPath: /etc/systemd
              readOnly: true
      restartPolicy: Never
      volumes:
        - name: var-lib-etcd
          hostPath:
            path: /var/lib/etcd
        - name: etc-kubernetes
          hostPath:
            path: /etc/kubernetes
        - name: usr-bin
          hostPath:
            path: /usr/bin
        - name: var-lib-kubelet
          hostPath:
            path: /var/lib/kubelet
        - name: etc-systemd
          hostPath:
            path: /etc/systemd

A few key points to explain:

• hostPID: true: kube-bench needs to read host process info (like kube-apiserver startup parameters). Without the PID namespace, it can only check static config files and miss runtime-effective parameters.
• Multiple hostPath mounts: kube-bench reads etcd data directory permissions, kubelet config, Kubernetes manifest files, etc., which live in different host paths.
• --benchmark cis-1.10: Explicitly specify the benchmark version. kube-bench supports auto-detecting K8s version and matching the corresponding benchmark, but in production, I recommend specifying the version explicitly. Auto-detection relies on K8s minor version inference — if your cluster is 1.30.2, auto-detect treats it as 1.30, but the CIS Benchmark version mapping might not cover 1.30 yet, resulting in outdated rules being used. Explicit version prevents missed checks.

Benchmark Version Reference

Check your K8s version before running, then pick the matching benchmark. Don’t blindly use auto-detection.

Your First Scan

# Run on a master node, check control plane
kube-bench --benchmark cis-1.10 --targets master,node

# Check only a specific component
kube-bench --benchmark cis-1.10 --targets etcd

# Output JSON format report (for programmatic processing)
kube-bench --benchmark cis-1.10 --targets master,node --output json | jq . > kube-bench-report.json

Interpreting Scan Reports

kube-bench output looks like this:

[FAIL] 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated)
[FAIL] 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated)
[PASS] 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated)
[WARN] 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated)

• PASS: Configuration meets requirements, no action needed.
• FAIL: Configuration doesn’t meet security requirements, needs fixing. This is what you focus on.
• WARN: kube-bench can’t determine this automatically, requires manual confirmation. Common for checks that depend on your specific business scenario.

Each FAIL comes with a remediation section that tells you exactly how to fix it. For example:

Remediation:
Edit the API Server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml
on the master node and set the below parameter.
--authorization-mode=Node,RBAC

Just follow the instructions. But don’t blindly fix everything — some checks don’t apply in certain scenarios. For managed K8s (EKS/GKE/AKS), the control plane is managed by the cloud provider, and many checks require platform-level changes you can’t make.

Hands-on: Top 10 Most Common FAIL Items and Fixes

I’ve run kube-bench scans on dozens of clusters. Here are the 10 most frequently occurring FAIL items, with fix methods.

1. API Server Anonymous Auth Not Disabled

# Check current config
ps -ef | grep kube-apiserver | grep anonymous-auth

# Fix: Edit /etc/kubernetes/manifests/kube-apiserver.yaml
# Add the parameter
- --anonymous-auth=false

This parameter controls whether unauthenticated users can access the API Server. When enabled, anyone can access endpoints like /api/v1/namespaces/default/pods. Might seem harmless on an internal network, but if your API Server is exposed to the internet (don’t do that), you’re handing out free access.

Note: Disabling anonymous auth affects health check probes. If your LB health check hits /healthz without auth, use --anonymous-auth=true and restrict access to health check paths only, or combine with --authorization-mode=Node,RBAC and Node auth. It’s a tradeoff.

2. etcd Client Certificate Auth Not Enabled

# Check
ps -ef | grep etcd | grep client-cert-auth

# Fix: Edit etcd configuration
- --client-cert-auth=true
- --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
- --cert-file=/etc/kubernetes/pki/etcd/server.crt
- --key-file=/etc/kubernetes/pki/etcd/server.key

etcd stores the entire cluster state — ConfigMaps, Secrets, ServiceAccount tokens. If etcd accepts connections without client certificates, anyone with network access can read all your secrets.

3. kubelet Read-Only Port Not Closed

# Check
ps -ef | grep kubelet | grep read-only-port

# Fix
# In kubelet config file /var/lib/kubelet/config.yaml
readOnlyPort: 0

Port 10255 (read-only port) allows unauthenticated access to kubelet’s info endpoint, exposing Pod lists, resource usage, and environment variables. It was enabled by default in older K8s versions, and many people forget to close it.

4. RBAC Not Enabled

# Check
ps -ef | grep kube-apiserver | grep authorization-mode

# Fix: Ensure authorization-mode includes RBAC
- --authorization-mode=Node,RBAC

Not enabling RBAC means everyone has cluster-admin permissions. This is unacceptable in any production environment. Note that AlwaysAllow must not appear in authorization-mode, or RBAC is effectively disabled.

5. Audit Logging Not Enabled

# /etc/kubernetes/manifests/kube-apiserver.yaml
- --audit-log-path=/var/log/kubernetes/audit/audit.log
- --audit-log-maxage=30
- --audit-log-maxbackup=10
- --audit-log-maxsize=100
- --audit-policy-file=/etc/kubernetes/audit-policy.yaml

Audit logs record “who did what to which resource at what time.” When a security incident happens, without audit logs you’re flying blind. Write an audit policy file:

# /etc/kubernetes/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log all Secret operations
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets"]
  # Log all auth operations
  - level: Metadata
    resources:
      - group: ""
        resources: ["serviceaccounts", "rolebindings", "clusterrolebindings"]
  # Log all write operations
  - level: Request
    verbs: ["create", "update", "patch", "delete"]
  # Everything else: metadata only
  - level: Metadata

6. Pod Security Standards Not Configured

K8s 1.25+ deprecated PodSecurityPolicy (PSP) in favor of Pod Security Admission (PSA). PSA controls Pod security levels via namespace labels:

# Label namespace to enforce restricted level
kubectl label namespace production \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn=restricted

Three levels from relaxed to strict: privileged > baseline > restricted. For production, I recommend at least baseline, with restricted for core business namespaces.

The restricted level prohibits: privileged containers, host network/PID/IPC, root user, unsafe capabilities, etc. Switching might break some Pods initially — start with warn labels to see alerts, then tighten gradually.

7. etcd Data Directory Permissions Too Loose

# Check
ls -la /var/lib/etcd/

# Fix
chown -R etcd:etcd /var/lib/etcd/
chmod 700 /var/lib/etcd/

CIS requires etcd data directory permissions to be 700. Loose permissions mean any system user can read etcd data files.

8. Kubernetes Certificate File Permissions

# Check
ls -la /etc/kubernetes/pki/

# Fix: CA cert 644, CA private key 600
chmod 644 /etc/kubernetes/pki/ca.crt
chmod 600 /etc/kubernetes/pki/ca.key
# Component certs 644, private keys 600
chmod 644 /etc/kubernetes/pki/apiserver.crt
chmod 600 /etc/kubernetes/pki/apiserver.key

9. kubelet Anonymous Auth Not Disabled

# /var/lib/kubelet/config.yaml
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true

Same principle as API Server anonymous auth. kubelet is directly exposed on the node network — enabling anonymous auth is running naked.

10. TLS Encryption Not Configured

# Ensure API Server has TLS configured
- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

It’s 2026. Clusters not running HTTPS should be phased out.

Trivy: Image-Level Security Scanning

kube-bench checks “is the cluster configuration secure,” while Trivy checks “are the images you’re running secure.” Two dimensions, both essential.

Installation

# Option 1: Binary
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin/

# Option 2: apt (Debian/Ubuntu)
sudo apt-get install trivy

# Option 3: Docker
docker run --rm aquasec/trivy:latest image nginx:latest

Scanning Images

# Scan a single image
trivy image nginx:1.27

# Filter by severity
trivy image --severity HIGH,CRITICAL nginx:1.27

# Output only vulnerable packages
trivy image --severity CRITICAL --format json nginx:1.27 | jq '.Results[0].Vulnerabilities[] | {VulnerabilityID, PkgName, InstalledVersion, FixedVersion}'

# Generate HTML report
trivy image --format template --template "@html.tpl" -o report.html nginx:1.27

Scanning All Images in a K8s Cluster

# List all images used by running Pods
kubectl get pods --all-namespaces -o jsonpath="{range .items[*]}{range .spec.containers[*]}{.image}{'\n'}{end}{end}" | sort -u > images.txt

# Batch scan
while read image; do
  echo "=== Scanning $image ==="
  trivy image --severity HIGH,CRITICAL "$image"
done < images.txt

This script has a practical issue: when a cluster has many images, it’s painfully slow. Use --skip-db-update to skip vulnerability DB updates (update beforehand), or use Trivy Server mode for shared caching.

Trivy KBOM: Kubernetes Bill of Materials

Trivy introduced KBOM (Kubernetes Bill of Materials) in 2024, which scans the entire cluster’s component inventory — similar to SBOM for applications. Useful for asset inventory and security auditing:

# Scan cluster KBOM
trivy k8s cluster --format kubebom -o kbom.json

# Scan cluster misconfigurations (similar to kube-bench but integrated in Trivy)
trivy k8s cluster --scanners misconfig

# Scan cluster for vulnerabilities
trivy k8s cluster --scanners vuln

KBOM shows you what’s actually installed in the cluster: K8s version, component versions, runtime versions, installed Operators and their versions. When a vulnerability hits, check this inventory against CVE announcements to quickly determine if you’re affected.

kube-hunter: Attacker’s Perspective on Your Cluster

kube-bench is the “defender’s” perspective — checking whether configurations meet standards. kube-hunter is the “attacker’s” perspective — simulating how an attacker could exploit your cluster.

# Install
pip install kube-hunter

# Scan from outside the cluster (simulate external attacker)
kube-hunter --remote <cluster-ip>

# Scan from a node (simulate lateral movement after getting a node)
kube-hunter --pod

# Scan from inside the cluster
kubectl run kube-hunter --rm -it --image=aquasec/kube-hunter:latest -- --pod

kube-hunter output example:

Vulnerability                | Severity    | Description
-----------------------------|-------------|--------------------------------------
CAP_NET_RAW enabled          | low         | Pods have CAP_NET_RAW capability
Dashboard exposed            | high        | Kubernetes Dashboard is exposed
Metrics Server insecure      | medium      | Metrics Server allows anonymous access

These findings overlap with kube-bench results, but kube-hunter’s advantage is telling you “if this vulnerability is exploited, here’s the impact.” For the same etcd unauthorized access, kube-bench just marks FAIL, while kube-hunter tells you “an attacker can read all Secrets including ServiceAccount tokens, then escalate to cluster admin.”

My experience in practice: kube-hunter’s false positive control isn’t as strict as kube-bench. Some reported “vulnerabilities” may not be reachable given your network architecture. For example, it reports “API Server port 10250 accessible without auth,” but if your nodes are in a VPC private subnet, it’s not reachable from outside. Judge reachability yourself — don’t panic.

Integrating Security Scanning into CI/CD Pipelines

Running a manual scan is easy. The hard part is making it continuous. Security scanning only prevents recurring issues when it becomes part of the pipeline.

GitOps Pattern: Auto-Scan Images on PR Merge

# .github/workflows/security-scan.yml
name: Container Image Security Scan
on:
  pull_request:
    paths:
      - 'deploy/**'  # Only trigger on deployment config changes

jobs:
  trivy-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Extract images from manifests
        id: images
        run: |
          IMAGES=$(grep -ohP 'image:\s*\K.*' deploy/*.yaml | sort -u)
          echo "images<<EOF" >> $GITHUB_OUTPUT
          echo "$IMAGES" >> $GITHUB_OUTPUT
          echo "EOF" >> $GITHUB_OUTPUT          

      - name: Run Trivy scan
        run: |
          for image in ${{ steps.images.outputs.images }}; do
            echo "=== Scanning $image ==="
            trivy image --severity CRITICAL --exit-code 1 "$image" || {
              echo "::error::Critical vulnerabilities found in $image"
              exit 1
            }
          done          

This pipeline automatically scans changed images when a PR modifies deployment configs, blocking merge on CRITICAL vulnerabilities.

Scheduled Scanning: CronJob for kube-bench

apiVersion: batch/v1
kind: CronJob
metadata:
  name: kube-bench-scan
  namespace: kube-system
spec:
  schedule: "0 2 * * 1"  # Every Monday at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          hostPID: true
          containers:
            - name: kube-bench
              image: aquasec/kube-bench:0.10.0
              command:
                - /bin/sh
                - -c
                - |
                  kube-bench --benchmark cis-1.10 --targets master,node \
                    --output json > /tmp/report.json
                  # Push report to monitoring system or Slack
                  curl -X POST "$SLACK_WEBHOOK" \
                    -H "Content-Type: application/json" \
                    -d "{\"text\":\"Weekly kube-bench scan completed. $(jq '[.Controls[].Checks[] | select(.State == \"FAIL\")] | length') FAIL items.\"}"                  
              env:
                - name: SLACK_WEBHOOK
                  valueFrom:
                    secretKeyRef:
                      name: slack-webhook
                      key: url
              volumeMounts:
                - name: etc-kubernetes
                  mountPath: /etc/kubernetes
                  readOnly: true
                - name: var-lib-kubelet
                  mountPath: /var/lib/kubelet
                  readOnly: true
          restartPolicy: OnFailure
          volumes:
            - name: etc-kubernetes
              hostPath:
                path: /etc/kubernetes
            - name: var-lib-kubelet
              hostPath:
                path: /var/lib/kubelet

Integrating with Prometheus Monitoring

To see compliance metrics on Grafana, build a custom exporter from kube-bench JSON output:

#!/usr/bin/env python3
"""kube-bench results exporter for Prometheus"""
import json
import subprocess
from http.server import HTTPServer, BaseHTTPRequestHandler

class MetricsHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != '/metrics':
            self.send_error(404)
            return

        result = subprocess.run(
            ['kube-bench', '--benchmark', 'cis-1.10',
             '--targets', 'master,node', '--output', 'json'],
            capture_output=True, text=True
        )
        data = json.loads(result.stdout)

        metrics = []
        total_pass = 0
        total_fail = 0
        total_warn = 0

        for control in data.get('Controls', []):
            for check in control.get('Checks', []):
                status = 1 if check['State'] == 'PASS' else 0
                metrics.append(
                    f'kube_bench_check{{id="{check["ID"]}",'
                    f'title="{check["Text"]}",'
                    f'state="{check["State"]}"}} {status}'
                )
                if check['State'] == 'PASS':
                    total_pass += 1
                elif check['State'] == 'FAIL':
                    total_fail += 1
                else:
                    total_warn += 1

        total = total_pass + total_fail + total_warn
        compliance = (total_pass / total * 100) if total > 0 else 0

        metrics.append(f'kube_bench_total_checks {total}')
        metrics.append(f'kube_bench_pass_total {total_pass}')
        metrics.append(f'kube_bench_fail_total {total_fail}')
        metrics.append(f'kube_bench_warn_total {total_warn}')
        metrics.append(f'kube_bench_compliance_percent {compliance}')

        self.send_response(200)
        self.send_header('Content-Type', 'text/plain')
        self.end_headers()
        self.wfile.write('\n'.join(metrics).encode())

if __name__ == '__main__':
    HTTPServer(('0.0.0.0', 9099), MetricsHandler).serve_forever()

Once running, Prometheus scrapes /metrics, and you build a Grafana dashboard showing compliance rate and FAIL trends:

# Compliance rate
kube_bench_compliance_percent

# FAIL count trend
kube_bench_fail_total

# Check status by item
kube_bench_check{state="FAIL"}

Security Hardening Decision Matrix

Not every FAIL needs immediate fixing. Security hardening requires prioritization — just like bug fixing, there are P0s and P3s.

Example of using this matrix:

• API Server anonymous auth enabled + cluster exposed to internet → P0, fix tonight.
• etcd client cert auth not enabled + etcd on isolated network segment → P1, fix this week.
• kubelet read-only port open + nodes on internal network → P2, next sprint.
• etcd data directory at 755 → P3, fixing won’t break anything, but not fixing won’t get you exploited short-term.

Don’t be intimidated by a report full of FAILs. I’ve seen scan results with 80 FAILs out of 200 checks, but only 3-5 were truly urgent. Fix the rest gradually.

Special Handling for Managed K8s (EKS/GKE/AKS)

If you’re using cloud-managed K8s, you can’t touch the control plane, and many kube-bench checks will show N/A or WARN. That’s not your problem — the cloud provider handles control plane security.

But several dimensions are your responsibility:

EKS has a dedicated kube-bench benchmark (--benchmark eks-1.8.0), GKE has --benchmark gke-1.8.0. These versions remove checks that don’t apply to managed environments and add cloud platform-specific checks.

In practice, EKS users should note: AWS published a security bulletin in July 2026 indicating a surge in Linux kernel CVE findings on Ubuntu 22.04 node images, caused by Canonical reclassifying a large number of kernel CVE statuses. Many reported vulnerabilities don’t have upstream fixes yet, and simply upgrading node versions won’t resolve them. Don’t panic in this situation — check actual CVE exploitability first, don’t let the scanner report lead you by the nose.

Open Source Security Tool Comparison

Beyond kube-bench and Trivy, there are many tools in the K8s security space. Here’s a comparison by use case:

My recommended combination:

1. kube-bench: Run baseline checks weekly
2. Trivy: CI/CD pipeline image scanning + periodic cluster-wide scans
3. OPA Gatekeeper or Kyverno: Admission control, block unsafe configs from entering cluster
4. Falco: Runtime monitoring, detect anomalous behavior

These four layers cover the main aspects of K8s security: configuration security, image security, admission control, and runtime security.

OPA Gatekeeper: Blocking Unsafe Configs at Admission Layer

Scanning alone isn’t enough. You fix a FAIL today, someone submits a PR tomorrow that adds it back. Gatekeeper intercepts unsafe configurations at the API Server admission control layer — non-compliant Pods/Deployments simply can’t be created.

# ConstraintTemplate to disallow privileged containers
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sdisallowprivileged
spec:
  crd:
    spec:
      names:
        kind: K8sDisallowPrivileged
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sdisallowprivileged

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          container.securityContext.privileged == true
          msg := sprintf("Privileged container %v is not allowed", [container.name])
        }

        violation[{"msg": msg}] {
          container := input.review.object.spec.initContainers[_]
          container.securityContext.privileged == true
          msg := sprintf("Privileged init container %v is not allowed", [container.name])
        }        
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sDisallowPrivileged
metadata:
  name: no-privileged-containers
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    excludedNamespaces: ["kube-system"]

In practice, I recommend starting with warn mode (alert only, no blocking) for observation, collecting which Pods would be blocked. Once you understand the impact, switch to deny mode. Going straight to deny will likely break running workloads.

Falco: Runtime Security Monitoring

All the tools above are “static” checks — checking configs, images, and admission. But attacks happen at runtime. Someone exploits a vulnerability, gets a Pod shell, and starts lateral movement — static scanning won’t catch that. Falco fills this gap.

Falco monitors system calls at the kernel level (via eBPF or kernel module) and matches anomalous behavior against rules:

# Falco rule example: Detect shell execution in container
- rule: Shell Spawned in Container
  desc: Detect shell spawned in container (possible attack)
  condition: >
    container.id != "" and
    proc.name in (bash, sh, zsh, ksh)    
  output: >
    Shell spawned in container
    (user=%user.name container_id=%container.id
    container_name=%container.name
    shell=%proc.name cmdline=%proc.cmdline)    
  priority: WARNING
  tags: [container, shell, mitre_execution]

Falco rules cover common attack patterns: shell execution in containers, abnormal filesystem read/write, network connections to suspicious IPs, privilege escalation. Alerts can be pushed to Slack/PagerDuty/ELK.

Deploy Falco in DaemonSet mode, running on every node:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set driver.kind=ebpf \
  --set falcosidekick.enabled=true \
  --set falcosidekick.config.slack.webhookurl=$SLACK_WEBHOOK

eBPF mode doesn’t require a kernel module but needs kernel version 5.2+. For older kernels, use the falcoctl module loading approach.

A Complete Hardening Workflow

Tying it all together, the standard workflow for hardening a cluster from scratch:

Step 1: Run kube-bench scan → Get FAIL list
Step 2: Prioritize by decision matrix
Step 3: Fix P0/P1 items (API Server params, etcd auth, RBAC, audit logs)
Step 4: Trivy scan all running images → Fix CRITICAL vulnerability images
Step 5: Deploy PSA labels → Restrict Pod security levels
Step 6: Deploy Gatekeeper → Block unsafe configs (warn mode first)
Step 7: Deploy Falco → Runtime monitoring
Step 8: Add kube-bench + Trivy scanning to CronJob + CI/CD
Step 9: Build Grafana compliance dashboard → Continuous monitoring

This isn’t a one-day job. For small-to-medium clusters (under 50 nodes), steps 1-8 take about 1-2 weeks. Larger clusters take more time, mainly on steps 5-6 for workload adaptation — PSA and Gatekeeper may affect existing workloads, requiring namespace-by-namespace evaluation and migration.

Common Pitfalls

Pitfall 1: kube-bench shows tons of FAILs on kubeadm clusters

kubeadm-deployed clusters do have many CIS Benchmark FAILs by default. Don’t panic — kubeadm prioritizes usability over security hardening. Most FAILs can be fixed by following the remediation instructions with minimal impact. But always back up the yaml files in /etc/kubernetes/manifests/ before making changes — a wrong edit can crash the API Server.

Pitfall 2: API Server won’t start after changing parameters

If you get an API Server startup parameter wrong (like a wrong cert path), the Pod will CrashLoopBackOff and the entire cluster becomes unusable. Solution: edit /etc/kubernetes/manifests/kube-apiserver.yaml directly on the node — kubelet auto-detects changes and restarts the Pod. Back up before editing, and watch journalctl -u kubelet -f to confirm the Pod starts normally.

Pitfall 3: kube-bench produces no results on managed clusters

EKS/GKE/AKS control planes are unreachable, and the hostPath mounts kube-bench expects may not exist on managed nodes. Use the corresponding --benchmark eks-1.8.0 or --benchmark gke-1.8.0, and ensure the containerd/docker config file paths are correct on the nodes.

Pitfall 4: Trivy fails in air-gapped environments

Trivy needs to download its vulnerability database (trivy-db). Air-gapped environments will hang at the download step. Solution:

# Download the DB in a networked environment
trivy --download-db-only

# Copy the DB to the air-gapped environment
# Default path: ~/.cache/trivy/db/

# Scan with --skip-db-update in air-gapped environment
trivy image --skip-db-update nginx:1.27

Pitfall 5: Falco rules produce too many false positives

Falco’s default rules are aggressive, especially for in-container process execution and network connection rules. Run at notice level for a week to establish a baseline, then whitelist high-false-positive rules. Don’t set all rules to critical from day one — alert fatigue will make the team numb.

Summary

K8s security isn’t solvable with a single tool. It’s a layered defense system:

1. Configuration layer: kube-bench runs CIS Benchmark to ensure baseline compliance
2. Image layer: Trivy scans image vulnerabilities and misconfigurations
3. Admission layer: Gatekeeper/Kyverno intercepts unsafe configs before the API Server
4. Runtime: Falco monitors anomalous behavior
5. Continuous: Automate scanning into CI/CD and scheduled tasks

Practical advice: Don’t be scared by a pile of FAILs on your first kube-bench run. Prioritize by the risk matrix, start with P0s. Managed cluster users: use the corresponding benchmark version, don’t run the generic cis benchmark and panic at a wall of N/A results. Image scanning must go into the pipeline — manual scanning is basically not scanning, because new images built tomorrow will bring new vulnerabilities.

Before switching Gatekeeper and PSA to deny mode, run in warn mode first. Going straight to blocking will likely break running business Pods. Security hardening should be progressive, not a一刀切 (one-size-fits-all cut).

Final honest truth: security is never “done.” CVEs come out daily, K8s releases every three months, and a 100% compliant cluster today may sprout new FAILs next month. The key is building a continuous scanning mechanism that catches issues at introduction time, rather than scrambling during a security audit.

References & Acknowledgments

The following resources were referenced during the writing of this article. Thanks to the original authors for their contributions:

1. kube-bench GitHub Repository — Aqua Security, CIS Kubernetes Benchmark automated checking tool and configuration documentation
2. CIS Kubernetes Benchmark Official Documentation — Center for Internet Security, K8s security configuration baseline standard
3. Trivy GitHub Repository — Aqua Security, container image and K8s cluster vulnerability scanner, KBOM feature reference
4. kube-hunter GitHub Repository — Aqua Security, K8s penetration testing tool
5. Security and Hardening of Kubernetes in Public Clouds: A Comparative Study of EKS, AKS, and GKE — IEEE, comparative analysis of kube-bench on EKS/AKS/GKE
6. Kubernetes Security Guide 2025 — Cloud Security Alliance Greater China, K8s attack-defense analysis based on the ATT&CK model
7. AKS Security Bulletins — Microsoft Azure, Ubuntu 22.04 node image kernel CVE classification changes
8. Comparative Analysis of Lightweight Kubernetes Distributions for Edge Computing — Springer, kube-bench security compliance comparison across k3s/k0s/KubeEdge/OpenYurt