Overview
Kubernetes Service provides Layer 4 load balancing, but in production, most web applications need Layer 7 routing capabilities: domain-based virtual hosting, path-based routing, TLS termination, and canary deployments. Ingress is K8s’s abstraction for Layer 7 routing, and the Ingress Controller is the concrete implementation of this abstraction.
Choosing an Ingress Controller is not a small decision—it sits at the entry point of all external traffic. A wrong choice or misconfiguration can impact the availability of the entire cluster’s services. This article compares mainstream Ingress Controllers and provides production configuration practices.
Based on Kubernetes v1.30. Reference: Kubernetes Ingress Documentation
Ingress Principles
Data Flow Path
Client → Load Balancer (Cloud LB/MetalLB) → Ingress Controller Pod → Service → Pod
↑
Ingress Resource
(routing rules)
An Ingress Controller is essentially a Pod running in the cluster (typically a Deployment or DaemonSet) that:
- Watches Ingress resource changes in the K8s API
- Translates Ingress rules into its own configuration (e.g., nginx.conf)
- Hot-reloads configuration, processes external requests, and routes to corresponding Services
Ingress Resource Structure
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp-ingress
namespace: production
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
ingressClassName: nginx
tls:
- hosts:
- api.example.com
secretName: api-tls
rules:
- host: api.example.com
http:
paths:
- path: /v1
pathType: Prefix
backend:
service:
name: api-v1-service
port:
number: 8080
- path: /v2
pathType: Prefix
backend:
service:
name: api-v2-service
port:
number: 8080
Three pathType Modes
| pathType | Matching Rule | Example |
|---|---|---|
Exact | Exact match | /health only matches /health |
Prefix | Prefix match | /api matches /api, /api/v1, /api/v2/users |
ImplementationSpecific | Determined by Ingress Controller | Nginx uses regex, Traefik uses its own rules |
Note:
Prefixmatching/apialso matches/apixxxbecause it’s prefix matching, not path segment matching. For path segment matching, use Nginx’srewrite-targetannotation or regex paths.
Mainstream Ingress Controller Comparison
Four Major Solutions
| Feature | Nginx Ingress | Traefik | HAProxy Ingress | Envoy Gateway |
|---|---|---|---|---|
| Engine | Nginx/OpenResty | Custom Go | HAProxy | Envoy |
| Config | Annotations + ConfigMap | CRD + annotations | Annotations + ConfigMap | CRD + xDS |
| Dynamic config | Requires reload (community edition) | Hot reload | Hot reload | Hot reload |
| Performance | High | Medium-high | Very high | Very high |
| Community | Largest | High | Medium | High (CNCF) |
| Learning curve | Low (Nginx familiar) | Low | Medium | High |
| Ecosystem | Rich | Medium | Medium | Rich (Service Mesh) |
| Scale | Small to large | Small to medium | Large | Large / Service Mesh |
Detailed Analysis
Nginx Ingress Controller
The most widely used Ingress Controller, available in community edition (kubernetes/ingress-nginx) and F5 edition (nginxinc/kubernetes-ingress).
Pros: Largest community, rich documentation, extensive troubleshooting resources; Nginx-based, familiar to ops teams; rich annotations supporting complex routing logic.
Cons: Community edition requires reload for config changes (millisecond-level disruption); too many annotations lead to fragmented config; less elegant support for advanced traffic management (canary, A/B testing).
Traefik
Modern reverse proxy written in Go, with automatic service discovery.
Pros: Hot reload without disruption; native Let’s Encrypt auto-certificate; intuitive Dashboard; powerful CRD (IngressRoute).
Cons: Performance below Nginx/HAProxy; advanced routing depends on CRD, limited compatibility with standard Ingress; community edition has limited features, enterprise edition is paid.
HAProxy Ingress
Based on industrial-grade LB HAProxy.
Pros: Extremely high performance, excellent connection reuse; powerful statistics and reporting; suited for high-concurrency scenarios.
Cons: Smaller community, less documentation than Nginx; different annotation style from Nginx, high migration cost.
Envoy Gateway
Next-generation gateway based on Envoy, a CNCF project.
Pros: Extreme performance and observability; native HTTP/2, gRPC support; seamless integration with Service Mesh (Istio); xDS dynamic config without disruption.
Cons: Steep learning curve; ecosystem still developing; high configuration complexity.
Selection Recommendations
| Scenario | Recommendation | Reason |
|---|---|---|
| General web apps, team familiar with Nginx | Nginx Ingress | Mature ecosystem, rich resources |
| Rapid prototyping / small scale, need auto certs | Traefik | Simple config, Let’s Encrypt integration |
| Ultra-high concurrency, performance priority | HAProxy Ingress | Best performance |
| Service Mesh / gRPC-intensive | Envoy Gateway | Native support, Istio integration |
| Multi-team shared cluster | Nginx Ingress | Multi-IngressClass isolation |
Nginx Ingress Controller Deployment
Installation
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--create-namespace \
--set controller.replicaCount=2 \
--set controller.service.type=LoadBalancer \
--set controller.config.proxy-body-size=50m \
--set controller.config.proxy-read-timeout=3600 \
--set controller.config.proxy-send-timeout=3600
Production-Grade Helm Configuration
# values.yaml
controller:
replicaCount: 2
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 1000m
memory: 512Mi
nodeSelector:
node-role: ingress
tolerations:
- key: "ingress-only"
operator: "Equal"
value: "true"
effect: "NoSchedule"
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
topologyKey: kubernetes.io/hostname
config:
proxy-body-size: "50m"
proxy-read-timeout: "3600"
proxy-send-timeout: "3600"
proxy-connect-timeout: "10"
keep-alive: "120"
keep-alive-requests: "10000"
worker-processes: "auto"
max-worker-connections: "65535"
enable-brotli: "true"
brotli-types: "text/xml text/plain text/css application/javascript application/json image/svg+xml"
use-gzip: "true"
gzip-types: "text/plain text/css application/javascript application/json"
ssl-protocols: "TLSv1.2 TLSv1.3"
ssl-ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256"
ssl-session-cache: "shared:SSL:10m"
ssl-session-timeout: "10m"
log-format-escape-json: "true"
log-format-upstream: '{"time":"$time_iso8601","remote_addr":"$remote_addr","x_forwarded_for":"$proxy_add_x_forwarded_for","request":"$request","status":$status,"body_bytes_sent":$body_bytes_sent,"request_time":$request_time,"upstream_response_time":"$upstream_response_time","upstream_addr":"$upstream_addr","upstream_status":"$upstream_status","http_referer":"$http_referer","http_user_agent":"$http_user_agent","request_id":"$req_id"}'
podSecurityContext:
runAsNonRoot: true
runAsUser: 101
fsGroup: 101
containerSecurityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
healthCheckPath: /healthz
metrics:
enabled: true
serviceMonitor:
enabled: true
admissionWebhooks:
enabled: true
failurePolicy: Fail
defaultBackend:
enabled: true
replicaCount: 2
TLS Termination
Certificate Management Options
| Approach | Use Case | Characteristics |
|---|---|---|
| Manual TLS Secret | Internal services / self-signed | Simple, manual renewal |
| cert-manager + Let’s Encrypt | Public domains | Auto issuance and renewal |
| cert-manager + private CA | Internal services | Unified internal cert management |
| Cloud provider cert management | Cloud K8s | Integrates with cloud LB |
cert-manager Auto Certificate
helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--set installCRDs=true
# Let's Encrypt ClusterIssuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: admin@example.com
privateKeySecretRef:
name: letsencrypt-prod-key
solvers:
- http01:
ingress:
class: nginx
# Ingress with auto certificate
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: api-ingress
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
tls:
- hosts:
- api.example.com
secretName: api-tls # cert-manager will auto-create
rules:
- host: api.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api-service
port:
number: 8080
TLS Version and Cipher Hardening
# ConfigMap TLS hardening
data:
ssl-protocols: "TLSv1.2 TLSv1.3" # Disable TLS 1.0/1.1
ssl-ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384"
ssl-prefer-server-ciphers: "true"
ssl-session-cache: "shared:SSL:10m"
ssl-session-timeout: "10m"
ssl-session-tickets: "false" # Disable session tickets
hsts: "true" # Enable HSTS
hsts-max-age: "31536000"
hsts-include-subdomains: "true"
hsts-preload: "true"
HTTP to HTTPS Redirect
# Global redirect (ConfigMap)
data:
ssl-redirect: "true"
# Per-Ingress annotation
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
Path Routing
Path-Baseded Multi-Service Routing
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp-ingress
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- host: api.example.com
http:
paths:
# /api/v1/* -> api-v1-service/*
- path: /api/v1(/|$)(.*)
pathType: ImplementationSpecific
backend:
service:
name: api-v1-service
port:
number: 8080
# /api/v2/* -> api-v2-service/*
- path: /api/v2(/|$)(.*)
pathType: ImplementationSpecific
backend:
service:
name: api-v2-service
port:
number: 8080
# /static/* -> static-service/* (no path rewrite)
- path: /static/
pathType: Prefix
backend:
service:
name: static-service
port:
number: 80
# / -> frontend-service
- path: /
pathType: Prefix
backend:
service:
name: frontend-service
port:
number: 80
WebSocket Support
annotations:
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
nginx.ingress.kubernetes.io/upstream-hash-by: "$remote_addr" # Session affinity
Nginx Ingress Controller supports WebSocket by default—just ensure timeout values are long enough.
gRPC Routing
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: grpc-ingress
annotations:
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
spec:
tls:
- hosts:
- grpc.example.com
secretName: grpc-tls
rules:
- host: grpc.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: grpc-service
port:
number: 9090
Canary Deployment
Canary Annotation Approach
Nginx Ingress Controller supports canary deployment via annotations:
# Main Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: api-main
spec:
rules:
- host: api.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api-stable
port:
number: 8080
# Canary Ingress — weight-based canary
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: api-canary
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "10" # 10% traffic to new version
spec:
rules:
- host: api.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api-canary
port:
number: 8080
Canary Strategy Comparison
| Strategy | Annotation | Use Case |
|---|---|---|
| By weight | canary-weight: "10" | 10% traffic to new version |
| By Header | canary-by-header: "x-canary" | Route by specific Header value |
| By Cookie | canary-by-cookie: "canary" | Route by specific Cookie |
# Header-based canary
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "x-canary"
nginx.ingress.kubernetes.io/canary-by-header-value: "true"
# Combined: 10% traffic + specific Header gets full canary
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "10"
nginx.ingress.kubernetes.io/canary-by-header: "x-canary"
nginx.ingress.kubernetes.io/canary-by-header-value: "always"
Note: Canary Ingress doesn’t handle requests independently—it adds routing rules on top of the main Ingress. Both Ingresses must use the same host and path.
Performance Tuning
Nginx Worker Parameter Tuning
# ConfigMap tuning
data:
worker-processes: "auto" # Auto-match CPU cores
worker-cpu-affinity: "auto" # CPU affinity
max-worker-connections: "65535" # Max connections per worker
max-worker-open-files: "65535" # Max file descriptors per worker
worker-shutdown-timeout: "240s" # Graceful shutdown wait time
worker-processes: "4" # Manual override (when auto is inaccurate)
Connection and Timeout Tuning
data:
keep-alive: "120" # Client keep-alive seconds
keep-alive-requests: "10000" # Max requests per keep-alive
upstream-keepalive-connections: "320" # Keepalive connections to backend
upstream-keepalive-timeout: "60" # Backend keepalive timeout
upstream-keepalive-requests: "10000" # Backend keepalive requests
proxy-connect-timeout: "10" # Backend connection timeout
proxy-read-timeout: "60" # Backend read timeout
proxy-send-timeout: "60" # Backend send timeout
proxy-next-upstream: "error timeout" # Retry next backend on failure
proxy-next-upstream-tries: "3" # Max 3 retries
Compression Optimization
data:
use-gzip: "true"
gzip-types: "text/plain text/css application/javascript application/json application/xml image/svg+xml"
gzip-min-length: "1024" # Only compress > 1KB
gzip-comp-level: "5" # Compression level 1-9
enable-brotli: "true" # Brotli compression (better than gzip)
brotli-types: "text/plain text/css application/javascript application/json application/xml image/svg+xml"
brotli-min-length: "1024"
Buffer Tuning
data:
proxy-buffer-size: "16k" # Proxy buffer size
proxy-buffers: "4 16k" # Proxy buffer count and size
proxy-busy-buffers-size: "32k" # Busy buffer size
client-header-buffer-size: "4k" # Client request header buffer
large-client-header-buffers: "4 16k" # Large request header buffer
Load Balancing Strategies
# Default: round-robin
# Switch via annotation
annotations:
nginx.ingress.kubernetes.io/load-balance: "least_conn" # Least connections
# nginx.ingress.kubernetes.io/load-balance: "ip_hash" # IP hash (session affinity)
# nginx.ingress.kubernetes.io/upstream-hash-by: "$request_uri" # URI hash
| Strategy | Description | Use Case |
|---|---|---|
round_robin (default) | Round-robin | General |
least_conn | Least connections | Long connections, varied request durations |
ip_hash | Client IP hash | Session affinity needed |
consistent_hash | Consistent hashing | Caching scenarios |
Observability
Log Configuration
# JSON format logs (recommended)
data:
log-format-escape-json: "true"
log-format-upstream: >-
{"time":"$time_iso8601","remote_addr":"$remote_addr","x_forwarded_for":"$proxy_add_x_forwarded_for","request":"$request","method":"$request_method","uri":"$request_uri","status":$status,"body_bytes_sent":$body_bytes_sent,"request_time":$request_time,"upstream_response_time":"$upstream_response_time","upstream_addr":"$upstream_addr","upstream_status":"$upstream_status","http_referer":"$http_referer","http_user_agent":"$http_user_agent","request_id":"$req_id","host":"$host","server_name":"$server_name"}
Prometheus Metrics
# Enable metrics
controller:
metrics:
enabled: true
serviceMonitor:
enabled: true
additionalLabels:
release: prometheus
Key metrics:
| Metric | Meaning | Alert Threshold |
|---|---|---|
nginx_ingress_controller_requests | Total requests / QPS | Based on capacity planning |
nginx_ingress_controller_response_duration_seconds | Response latency P99 | > 1s |
nginx_ingress_controller_nginx_process_connections | Current connections | > 80% of max-worker-connections |
nginx_ingress_controller_upstream_latency_seconds | Backend latency | > 500ms |
nginx_ingress_controller_errors | Error count | Continuously increasing |
Grafana Dashboard
Recommended official Dashboard IDs: 9614 (Nginx Ingress Controller) and 14314 (detailed Nginx monitoring).
Production Practices
Multi-IngressClass Isolation
# Internal IngressClass
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
name: nginx-internal
annotations:
ingressclass.kubernetes.io/is-default-class: "false"
spec:
controller: k8s.io/ingress-nginx
parameters:
apiGroup: k8s.example.com
kind: IngressParameters
name: internal-params
# External IngressClass
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
name: nginx-external
annotations:
ingressclass.kubernetes.io/is-default-class: "true"
spec:
controller: k8s.io/ingress-nginx
Node Affinity Deployment
# Schedule Ingress Controller to dedicated nodes
apiVersion: apps/v1
kind: Deployment
metadata:
name: ingress-nginx-controller
spec:
template:
spec:
nodeSelector:
node-role.kubernetes.io/ingress: "true"
tolerations:
- key: "ingress-only"
operator: "Exists"
effect: "NoSchedule"
# Anti-affinity ensures replicas are on different nodes
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
topologyKey: kubernetes.io/hostname
Rate Limiting
annotations:
nginx.ingress.kubernetes.io/limit-connections: "100" # Max concurrent connections
nginx.ingress.kubernetes.io/limit-rps: "50" # Requests per second
nginx.ingress.kubernetes.io/limit-burst: "100" # Burst requests
nginx.ingress.kubernetes.io/limit-retry-after-time: "60" # 429 response Retry-After
Basic Authentication
# Create htpasswd Secret
kubectl create secret generic basic-auth \
--from-file=auth \
-n production
# Ingress annotation
annotations:
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: basic-auth
nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
CORS Configuration
annotations:
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-origin: "https://example.com,https://app.example.com"
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, POST, PUT, DELETE, OPTIONS"
nginx.ingress.kubernetes.io/cors-allow-headers: "DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization"
nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
nginx.ingress.kubernetes.io/cors-max-age: "86400"
Common Troubleshooting
502 Bad Gateway
# Cause: Backend Service not ready or Pod unreachable
# Troubleshooting steps:
kubectl get endpoints <service-name> -n <namespace> # Check Endpoints
kubectl get pods -l app=<app-name> -n <namespace> # Check Pod status
kubectl logs <ingress-pod> -n ingress-nginx | grep <domain> # Check Ingress logs
504 Gateway Timeout
# Cause: Backend response timeout
# Solution: Adjust timeout config
annotations:
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
Configuration Not Taking Effect
# Check if Ingress is recognized
kubectl get ingress -A
kubectl describe ingress <name> -n <namespace>
# Check IngressClass match
kubectl get ingressclass
# Check if Nginx config is generated
kubectl exec -it <ingress-pod> -n ingress-nginx -- cat /etc/nginx/nginx.conf | grep <domain>
# Force reload (not recommended, debugging only)
kubectl exec -it <ingress-pod> -n ingress-nginx -- /nginx-ingress-controller --publish-service 2>&1 | head
SSL Certificate Not Working
# Check if certificate Secret exists
kubectl get secret <tls-secret-name> -n <namespace>
kubectl get certificate -n <namespace> # cert-manager
# Check certificate content
kubectl get secret <tls-secret-name> -n <namespace> -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout
Summary
The Ingress Controller is the traffic entry point for a K8s cluster—its selection and configuration directly impact service availability. Key takeaways:
- Choose based on scenario: Nginx Ingress is the safest choice with a mature ecosystem and rich resources. Only consider alternatives when there are clear requirements (extreme performance, Service Mesh integration).
- TLS must be hardened: Disable TLS 1.0/1.1, enable HSTS, use cert-manager for automatic certificate management.
- Use Canary annotations for canary deployment: Nginx Ingress’s canary annotations meet most canary needs. For complex traffic management, consider Istio/Envoy.
- Focus on key performance parameters:
max-worker-connections,upstream-keepalive-connections, andkeep-aliveare the three most critical performance parameters. - Observability is essential: JSON logs + Prometheus metrics are the standard. A gateway without observability is flying blind.
- Multi-replica + anti-affinity: Ingress Controller should have at least 2 replicas on different nodes to avoid single point of failure.
References & Acknowledgments
This article referenced the following materials during writing. We thank the original authors for their contributions:
- Kubernetes Ingress Documentation — Kubernetes Official, referenced for Kubernetes Ingress Documentation