Overview

Kubernetes Service provides Layer 4 load balancing, but in production, most web applications need Layer 7 routing capabilities: domain-based virtual hosting, path-based routing, TLS termination, and canary deployments. Ingress is K8s’s abstraction for Layer 7 routing, and the Ingress Controller is the concrete implementation of this abstraction.

Choosing an Ingress Controller is not a small decision—it sits at the entry point of all external traffic. A wrong choice or misconfiguration can impact the availability of the entire cluster’s services. This article compares mainstream Ingress Controllers and provides production configuration practices.

Based on Kubernetes v1.30. Reference: Kubernetes Ingress Documentation

Ingress Principles

Data Flow Path

Client → Load Balancer (Cloud LB/MetalLB) → Ingress Controller Pod → Service → Pod
                                           Ingress Resource
                                           (routing rules)

An Ingress Controller is essentially a Pod running in the cluster (typically a Deployment or DaemonSet) that:

  1. Watches Ingress resource changes in the K8s API
  2. Translates Ingress rules into its own configuration (e.g., nginx.conf)
  3. Hot-reloads configuration, processes external requests, and routes to corresponding Services

Ingress Resource Structure

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - api.example.com
    secretName: api-tls
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /v1
        pathType: Prefix
        backend:
          service:
            name: api-v1-service
            port:
              number: 8080
      - path: /v2
        pathType: Prefix
        backend:
          service:
            name: api-v2-service
            port:
              number: 8080

Three pathType Modes

pathTypeMatching RuleExample
ExactExact match/health only matches /health
PrefixPrefix match/api matches /api, /api/v1, /api/v2/users
ImplementationSpecificDetermined by Ingress ControllerNginx uses regex, Traefik uses its own rules

Note: Prefix matching /api also matches /apixxx because it’s prefix matching, not path segment matching. For path segment matching, use Nginx’s rewrite-target annotation or regex paths.

Mainstream Ingress Controller Comparison

Four Major Solutions

FeatureNginx IngressTraefikHAProxy IngressEnvoy Gateway
EngineNginx/OpenRestyCustom GoHAProxyEnvoy
ConfigAnnotations + ConfigMapCRD + annotationsAnnotations + ConfigMapCRD + xDS
Dynamic configRequires reload (community edition)Hot reloadHot reloadHot reload
PerformanceHighMedium-highVery highVery high
CommunityLargestHighMediumHigh (CNCF)
Learning curveLow (Nginx familiar)LowMediumHigh
EcosystemRichMediumMediumRich (Service Mesh)
ScaleSmall to largeSmall to mediumLargeLarge / Service Mesh

Detailed Analysis

Nginx Ingress Controller

The most widely used Ingress Controller, available in community edition (kubernetes/ingress-nginx) and F5 edition (nginxinc/kubernetes-ingress).

Pros: Largest community, rich documentation, extensive troubleshooting resources; Nginx-based, familiar to ops teams; rich annotations supporting complex routing logic.

Cons: Community edition requires reload for config changes (millisecond-level disruption); too many annotations lead to fragmented config; less elegant support for advanced traffic management (canary, A/B testing).

Traefik

Modern reverse proxy written in Go, with automatic service discovery.

Pros: Hot reload without disruption; native Let’s Encrypt auto-certificate; intuitive Dashboard; powerful CRD (IngressRoute).

Cons: Performance below Nginx/HAProxy; advanced routing depends on CRD, limited compatibility with standard Ingress; community edition has limited features, enterprise edition is paid.

HAProxy Ingress

Based on industrial-grade LB HAProxy.

Pros: Extremely high performance, excellent connection reuse; powerful statistics and reporting; suited for high-concurrency scenarios.

Cons: Smaller community, less documentation than Nginx; different annotation style from Nginx, high migration cost.

Envoy Gateway

Next-generation gateway based on Envoy, a CNCF project.

Pros: Extreme performance and observability; native HTTP/2, gRPC support; seamless integration with Service Mesh (Istio); xDS dynamic config without disruption.

Cons: Steep learning curve; ecosystem still developing; high configuration complexity.

Selection Recommendations

ScenarioRecommendationReason
General web apps, team familiar with NginxNginx IngressMature ecosystem, rich resources
Rapid prototyping / small scale, need auto certsTraefikSimple config, Let’s Encrypt integration
Ultra-high concurrency, performance priorityHAProxy IngressBest performance
Service Mesh / gRPC-intensiveEnvoy GatewayNative support, Istio integration
Multi-team shared clusterNginx IngressMulti-IngressClass isolation

Nginx Ingress Controller Deployment

Installation

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --set controller.replicaCount=2 \
  --set controller.service.type=LoadBalancer \
  --set controller.config.proxy-body-size=50m \
  --set controller.config.proxy-read-timeout=3600 \
  --set controller.config.proxy-send-timeout=3600

Production-Grade Helm Configuration

# values.yaml
controller:
  replicaCount: 2

  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 1000m
      memory: 512Mi

  nodeSelector:
    node-role: ingress

  tolerations:
  - key: "ingress-only"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"

  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: ingress-nginx
        topologyKey: kubernetes.io/hostname

  config:
    proxy-body-size: "50m"
    proxy-read-timeout: "3600"
    proxy-send-timeout: "3600"
    proxy-connect-timeout: "10"
    keep-alive: "120"
    keep-alive-requests: "10000"
    worker-processes: "auto"
    max-worker-connections: "65535"
    enable-brotli: "true"
    brotli-types: "text/xml text/plain text/css application/javascript application/json image/svg+xml"
    use-gzip: "true"
    gzip-types: "text/plain text/css application/javascript application/json"
    ssl-protocols: "TLSv1.2 TLSv1.3"
    ssl-ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256"
    ssl-session-cache: "shared:SSL:10m"
    ssl-session-timeout: "10m"
    log-format-escape-json: "true"
    log-format-upstream: '{"time":"$time_iso8601","remote_addr":"$remote_addr","x_forwarded_for":"$proxy_add_x_forwarded_for","request":"$request","status":$status,"body_bytes_sent":$body_bytes_sent,"request_time":$request_time,"upstream_response_time":"$upstream_response_time","upstream_addr":"$upstream_addr","upstream_status":"$upstream_status","http_referer":"$http_referer","http_user_agent":"$http_user_agent","request_id":"$req_id"}'

  podSecurityContext:
    runAsNonRoot: true
    runAsUser: 101
    fsGroup: 101

  containerSecurityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
      - ALL
      add:
      - NET_BIND_SERVICE

  healthCheckPath: /healthz

  metrics:
    enabled: true
    serviceMonitor:
      enabled: true

  admissionWebhooks:
    enabled: true
    failurePolicy: Fail

defaultBackend:
  enabled: true
  replicaCount: 2

TLS Termination

Certificate Management Options

ApproachUse CaseCharacteristics
Manual TLS SecretInternal services / self-signedSimple, manual renewal
cert-manager + Let’s EncryptPublic domainsAuto issuance and renewal
cert-manager + private CAInternal servicesUnified internal cert management
Cloud provider cert managementCloud K8sIntegrates with cloud LB

cert-manager Auto Certificate

helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true
# Let's Encrypt ClusterIssuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
    - http01:
        ingress:
          class: nginx
# Ingress with auto certificate
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
  - hosts:
    - api.example.com
    secretName: api-tls  # cert-manager will auto-create
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 8080

TLS Version and Cipher Hardening

# ConfigMap TLS hardening
data:
  ssl-protocols: "TLSv1.2 TLSv1.3"           # Disable TLS 1.0/1.1
  ssl-ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384"
  ssl-prefer-server-ciphers: "true"
  ssl-session-cache: "shared:SSL:10m"
  ssl-session-timeout: "10m"
  ssl-session-tickets: "false"                # Disable session tickets
  hsts: "true"                                 # Enable HSTS
  hsts-max-age: "31536000"
  hsts-include-subdomains: "true"
  hsts-preload: "true"

HTTP to HTTPS Redirect

# Global redirect (ConfigMap)
data:
  ssl-redirect: "true"

# Per-Ingress annotation
annotations:
  nginx.ingress.kubernetes.io/ssl-redirect: "true"

Path Routing

Path-Baseded Multi-Service Routing

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - host: api.example.com
    http:
      paths:
      # /api/v1/* -> api-v1-service/*
      - path: /api/v1(/|$)(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: api-v1-service
            port:
              number: 8080
      # /api/v2/* -> api-v2-service/*
      - path: /api/v2(/|$)(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: api-v2-service
            port:
              number: 8080
      # /static/* -> static-service/* (no path rewrite)
      - path: /static/
        pathType: Prefix
        backend:
          service:
            name: static-service
            port:
              number: 80
      # / -> frontend-service
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend-service
            port:
              number: 80

WebSocket Support

annotations:
  nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
  nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
  nginx.ingress.kubernetes.io/upstream-hash-by: "$remote_addr"  # Session affinity

Nginx Ingress Controller supports WebSocket by default—just ensure timeout values are long enough.

gRPC Routing

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grpc-ingress
  annotations:
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
spec:
  tls:
  - hosts:
    - grpc.example.com
    secretName: grpc-tls
  rules:
  - host: grpc.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: grpc-service
            port:
              number: 9090

Canary Deployment

Canary Annotation Approach

Nginx Ingress Controller supports canary deployment via annotations:

# Main Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-main
spec:
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-stable
            port:
              number: 8080
# Canary Ingress — weight-based canary
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"   # 10% traffic to new version
spec:
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-canary
            port:
              number: 8080

Canary Strategy Comparison

StrategyAnnotationUse Case
By weightcanary-weight: "10"10% traffic to new version
By Headercanary-by-header: "x-canary"Route by specific Header value
By Cookiecanary-by-cookie: "canary"Route by specific Cookie
# Header-based canary
annotations:
  nginx.ingress.kubernetes.io/canary: "true"
  nginx.ingress.kubernetes.io/canary-by-header: "x-canary"
  nginx.ingress.kubernetes.io/canary-by-header-value: "true"
# Combined: 10% traffic + specific Header gets full canary
annotations:
  nginx.ingress.kubernetes.io/canary: "true"
  nginx.ingress.kubernetes.io/canary-weight: "10"
  nginx.ingress.kubernetes.io/canary-by-header: "x-canary"
  nginx.ingress.kubernetes.io/canary-by-header-value: "always"

Note: Canary Ingress doesn’t handle requests independently—it adds routing rules on top of the main Ingress. Both Ingresses must use the same host and path.

Performance Tuning

Nginx Worker Parameter Tuning

# ConfigMap tuning
data:
  worker-processes: "auto"             # Auto-match CPU cores
  worker-cpu-affinity: "auto"          # CPU affinity
  max-worker-connections: "65535"      # Max connections per worker
  max-worker-open-files: "65535"       # Max file descriptors per worker
  worker-shutdown-timeout: "240s"      # Graceful shutdown wait time
  worker-processes: "4"                # Manual override (when auto is inaccurate)

Connection and Timeout Tuning

data:
  keep-alive: "120"                    # Client keep-alive seconds
  keep-alive-requests: "10000"         # Max requests per keep-alive
  upstream-keepalive-connections: "320" # Keepalive connections to backend
  upstream-keepalive-timeout: "60"     # Backend keepalive timeout
  upstream-keepalive-requests: "10000" # Backend keepalive requests
  proxy-connect-timeout: "10"          # Backend connection timeout
  proxy-read-timeout: "60"             # Backend read timeout
  proxy-send-timeout: "60"             # Backend send timeout
  proxy-next-upstream: "error timeout" # Retry next backend on failure
  proxy-next-upstream-tries: "3"       # Max 3 retries

Compression Optimization

data:
  use-gzip: "true"
  gzip-types: "text/plain text/css application/javascript application/json application/xml image/svg+xml"
  gzip-min-length: "1024"              # Only compress > 1KB
  gzip-comp-level: "5"                 # Compression level 1-9
  enable-brotli: "true"                # Brotli compression (better than gzip)
  brotli-types: "text/plain text/css application/javascript application/json application/xml image/svg+xml"
  brotli-min-length: "1024"

Buffer Tuning

data:
  proxy-buffer-size: "16k"             # Proxy buffer size
  proxy-buffers: "4 16k"               # Proxy buffer count and size
  proxy-busy-buffers-size: "32k"       # Busy buffer size
  client-header-buffer-size: "4k"      # Client request header buffer
  large-client-header-buffers: "4 16k" # Large request header buffer

Load Balancing Strategies

# Default: round-robin
# Switch via annotation
annotations:
  nginx.ingress.kubernetes.io/load-balance: "least_conn"    # Least connections
  # nginx.ingress.kubernetes.io/load-balance: "ip_hash"     # IP hash (session affinity)
  # nginx.ingress.kubernetes.io/upstream-hash-by: "$request_uri"  # URI hash
StrategyDescriptionUse Case
round_robin (default)Round-robinGeneral
least_connLeast connectionsLong connections, varied request durations
ip_hashClient IP hashSession affinity needed
consistent_hashConsistent hashingCaching scenarios

Observability

Log Configuration

# JSON format logs (recommended)
data:
  log-format-escape-json: "true"
  log-format-upstream: >-
    {"time":"$time_iso8601","remote_addr":"$remote_addr","x_forwarded_for":"$proxy_add_x_forwarded_for","request":"$request","method":"$request_method","uri":"$request_uri","status":$status,"body_bytes_sent":$body_bytes_sent,"request_time":$request_time,"upstream_response_time":"$upstream_response_time","upstream_addr":"$upstream_addr","upstream_status":"$upstream_status","http_referer":"$http_referer","http_user_agent":"$http_user_agent","request_id":"$req_id","host":"$host","server_name":"$server_name"}    

Prometheus Metrics

# Enable metrics
controller:
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
      additionalLabels:
        release: prometheus

Key metrics:

MetricMeaningAlert Threshold
nginx_ingress_controller_requestsTotal requests / QPSBased on capacity planning
nginx_ingress_controller_response_duration_secondsResponse latency P99> 1s
nginx_ingress_controller_nginx_process_connectionsCurrent connections> 80% of max-worker-connections
nginx_ingress_controller_upstream_latency_secondsBackend latency> 500ms
nginx_ingress_controller_errorsError countContinuously increasing

Grafana Dashboard

Recommended official Dashboard IDs: 9614 (Nginx Ingress Controller) and 14314 (detailed Nginx monitoring).

Production Practices

Multi-IngressClass Isolation

# Internal IngressClass
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx-internal
  annotations:
    ingressclass.kubernetes.io/is-default-class: "false"
spec:
  controller: k8s.io/ingress-nginx
  parameters:
    apiGroup: k8s.example.com
    kind: IngressParameters
    name: internal-params
# External IngressClass
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx-external
  annotations:
    ingressclass.kubernetes.io/is-default-class: "true"
spec:
  controller: k8s.io/ingress-nginx

Node Affinity Deployment

# Schedule Ingress Controller to dedicated nodes
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-nginx-controller
spec:
  template:
    spec:
      nodeSelector:
        node-role.kubernetes.io/ingress: "true"
      tolerations:
      - key: "ingress-only"
        operator: "Exists"
        effect: "NoSchedule"
      # Anti-affinity ensures replicas are on different nodes
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app.kubernetes.io/name: ingress-nginx
            topologyKey: kubernetes.io/hostname

Rate Limiting

annotations:
  nginx.ingress.kubernetes.io/limit-connections: "100"      # Max concurrent connections
  nginx.ingress.kubernetes.io/limit-rps: "50"               # Requests per second
  nginx.ingress.kubernetes.io/limit-burst: "100"            # Burst requests
  nginx.ingress.kubernetes.io/limit-retry-after-time: "60"  # 429 response Retry-After

Basic Authentication

# Create htpasswd Secret
kubectl create secret generic basic-auth \
  --from-file=auth \
  -n production

# Ingress annotation
annotations:
  nginx.ingress.kubernetes.io/auth-type: basic
  nginx.ingress.kubernetes.io/auth-secret: basic-auth
  nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"

CORS Configuration

annotations:
  nginx.ingress.kubernetes.io/enable-cors: "true"
  nginx.ingress.kubernetes.io/cors-allow-origin: "https://example.com,https://app.example.com"
  nginx.ingress.kubernetes.io/cors-allow-methods: "GET, POST, PUT, DELETE, OPTIONS"
  nginx.ingress.kubernetes.io/cors-allow-headers: "DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization"
  nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
  nginx.ingress.kubernetes.io/cors-max-age: "86400"

Common Troubleshooting

502 Bad Gateway

# Cause: Backend Service not ready or Pod unreachable
# Troubleshooting steps:
kubectl get endpoints <service-name> -n <namespace>  # Check Endpoints
kubectl get pods -l app=<app-name> -n <namespace>    # Check Pod status
kubectl logs <ingress-pod> -n ingress-nginx | grep <domain>  # Check Ingress logs

504 Gateway Timeout

# Cause: Backend response timeout
# Solution: Adjust timeout config
annotations:
  nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
  nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"

Configuration Not Taking Effect

# Check if Ingress is recognized
kubectl get ingress -A
kubectl describe ingress <name> -n <namespace>

# Check IngressClass match
kubectl get ingressclass

# Check if Nginx config is generated
kubectl exec -it <ingress-pod> -n ingress-nginx -- cat /etc/nginx/nginx.conf | grep <domain>

# Force reload (not recommended, debugging only)
kubectl exec -it <ingress-pod> -n ingress-nginx -- /nginx-ingress-controller --publish-service 2>&1 | head

SSL Certificate Not Working

# Check if certificate Secret exists
kubectl get secret <tls-secret-name> -n <namespace>
kubectl get certificate -n <namespace>  # cert-manager

# Check certificate content
kubectl get secret <tls-secret-name> -n <namespace> -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout

Summary

The Ingress Controller is the traffic entry point for a K8s cluster—its selection and configuration directly impact service availability. Key takeaways:

  1. Choose based on scenario: Nginx Ingress is the safest choice with a mature ecosystem and rich resources. Only consider alternatives when there are clear requirements (extreme performance, Service Mesh integration).
  2. TLS must be hardened: Disable TLS 1.0/1.1, enable HSTS, use cert-manager for automatic certificate management.
  3. Use Canary annotations for canary deployment: Nginx Ingress’s canary annotations meet most canary needs. For complex traffic management, consider Istio/Envoy.
  4. Focus on key performance parameters: max-worker-connections, upstream-keepalive-connections, and keep-alive are the three most critical performance parameters.
  5. Observability is essential: JSON logs + Prometheus metrics are the standard. A gateway without observability is flying blind.
  6. Multi-replica + anti-affinity: Ingress Controller should have at least 2 replicas on different nodes to avoid single point of failure.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

  1. Kubernetes Ingress Documentation — Kubernetes Official, referenced for Kubernetes Ingress Documentation