Pods Stuck Pending After IP Pool Exhaustion: 5 Production Decisions for TKE Network Selection and Node Pool Governance

Overview

2 AM, phone buzzing with alerts. I open it up — 40+ pods in the TKE cluster suddenly enter Pending state, business APIs timing out across the board. A quick kubectl describe pod reveals the scheduling failure reason, crystal clear: Insufficient tke.cloud.tencent.com/eni-ip.

This wasn’t some exotic technical puzzle. The root cause was straightforward: under VPC-CNI mode, the auxiliary IPs bound to the elastic network interfaces (ENIs) on the nodes were exhausted. But the investigation and fix took 3 hours — because nobody expected IPs to run out, there was no monitoring metric for it, and node pool autoscaling couldn’t provision new nodes either (new nodes couldn’t bind IPs either — the subnet IPs were all allocated).

The biggest lesson from this incident: TKE network model selection isn’t about choosing performance parameters, it’s about choosing your failure mode. The difference between GlobalRouter and VPC-CNI goes far beyond “10% performance improvement.” They differ fundamentally in IP management, scaling behavior, monitoring metrics, and failure propagation paths. This article starts from that real incident and breaks down 5 production decisions I’ve personally navigated, to help you avoid these pitfalls at the selection stage.

If you’re using or planning to adopt Tencent Cloud TKE (Tencent Kubernetes Engine for Containers), whether migrating from self-hosted K8s or building fresh, network model selection and node pool governance are the first hurdle you can’t skip. Get it wrong, and the cost of remediation far exceeds the half-day of extra thinking you’d spend at selection time.

TKE’s Three Network Models: You’re Not Choosing Performance, You’re Choosing Failure Modes

Let me explain what each model is, then explain why the choice is fundamentally about “choosing your failure mode.”

GlobalRouter: Abundant Addresses but an Extra Hop

GlobalRouter is TKE’s container networking solution built on Tencent Cloud’s VPC global routing capability. The mechanism is straightforward: each worker node gets assigned a CIDR block (typically /24), all pods on that node pull IPs from this block, and the node acts as a router, using VPC’s underlying routing policies for pod-to-pod communication.

# View the container CIDR assigned to a node
kubectl get node <node-name> -o jsonpath='{.spec.providerID}'
# Under GlobalRouter mode, each node has its own container CIDR

Its advantages are clear: the container subnet doesn’t overlap with the VPC subnet, so address space is abundant — a single /24 gives you 254 pods. It scales well, integrates seamlessly with standard K8s features, and while pod IPs change on restart or migration, the Service abstraction shields you from this.

The downside is that pod packets go through the node’s bridge device for forwarding, adding an extra hop. Latency is roughly 10% higher than VPC-CNI. Also, pod IPs aren’t real VPC IPs, making VPC-level network policies and traffic mirroring inconvenient.

VPC-CNI: Better Performance but IP Constrained by ENI Quotas

VPC-CNI is built on the CNI specification and VPC elastic network interfaces (ENIs). Pods get real IPs from the VPC subnet directly. Packets don’t go through a node bridge, no VxLAN tunnel encapsulation is needed, and performance, observability, rate limiting, and isolation are all better. Tencent Cloud officially recommends VPC-CNI as the default network solution.

But there’s no free lunch. Under VPC-CNI, pod IPs come from auxiliary IPs bound to ENIs on the node, and how many IPs an ENI can bind depends on the CVM instance specification. An SA2.LARGE8 (2-core 8GB) can bind at most 30 auxiliary IPs; an SA2.2XLARGE32 (8-core 32GB) can bind at most 60. If the subnet doesn’t have enough IPs, even a high-spec instance is stuck.

This means under VPC-CNI, pod density is limited by min(ENI IP quota for instance spec, remaining subnet IPs), not by the node’s CPU/memory. I’ve seen teams request large 16-core 64GB instances, only to find VPC-CNI can only bind 60 IPs, capping pod count at 60 with resource utilization below 40%.

Cilium-Overlay: The Third Option

In 2026, TKE also introduced Cilium-Overlay, an eBPF-based networking solution that balances performance and address abundance. However, there aren’t enough production cases yet — this article focuses on GlobalRouter and VPC-CNI selection. Cilium-Overlay is discussed as an alternative in Decision 5.

Why It’s “Choosing Your Failure Mode”

Choose GlobalRouter and you’ll likely never hit IP exhaustion, but you accept the 10% performance loss and extra bridge-forwarding latency. Choose VPC-CNI and performance improves, but the IP pool becomes a new failure domain — and a hidden one, since most teams don’t add ENI IP utilization to their monitoring dashboards.

That’s what “choosing your failure mode” means: both options will fail, just in different places. The selection process is about choosing the failure type you can monitor, troubleshoot, and accept.

Decision 1: GlobalRouter vs VPC-CNI — Don’t Be Misled by “10% Performance Improvement”

My Painful Experience

During TKE cluster selection for a mobility project, the team saw VPC-CNI’s “approximately 10% performance improvement over GlobalRouter” and went all-in on VPC-CNI as the default network mode. Two months after launch, during a traffic peak autoscaling event, pods on new nodes stayed Pending, and the alert chat exploded.

In hindsight, the problem was that selection only looked at performance differences, not IP management constraints. The mobility project used many small instances (4-core 8GB), and under VPC-CNI each could only bind 30 auxiliary IPs. With each microservice pod reserving an IP, 30 IPs ran out fast.

Selection Decision Framework

My current selection criteria:

Default to GlobalRouter, unless there’s a hard requirement for VPC-CNI. Reasons:

1. GlobalRouter has abundant addresses — you won’t get stuck on scaling due to IP issues. For most businesses, 10% network performance difference isn’t the bottleneck — your app latency bottleneck is in database queries and external API calls, not the extra forwarding hop
2. GlobalRouter has the best compatibility with standard K8s features — community docs and troubleshooting guides apply directly
3. VPC-CNI’s IP management adds operational complexity requiring extra monitoring metrics and capacity planning

When to choose VPC-CNI:

• Need pod fixed IPs (IP whitelists, legacy middleware integration, log collection by IP)
• Need client real source IP (VPC-CNI direct mode lets CLB forward directly to pods, no NAT)
• Extreme latency requirements (trading systems, real-time compute)
• Need VPC-level network policies for traffic control

When to mix:

The best practice for most scenarios is GlobalRouter as default + VPC-CNI enabled on demand. Create the cluster with GlobalRouter, then enable VPC-CNI support in the cluster’s basic info page as needed. Only workloads that explicitly need fixed IPs or direct load balancing should specify the k8s.v1.cni.cncf.io/networks: tke-vpc-cni annotation to use VPC-CNI; all other pods default to GlobalRouter.

# Enable VPC-CNI for a specific workload (mixed mode)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  template:
    metadata:
      annotations:
        # Specify this pod uses VPC-CNI network mode
        k8s.v1.cni.cncf.io/networks: tke-vpc-cni
    spec:
      containers:
      - name: payment
        image: registry.example.com/payment:v2.1.0

The core principle of this decision: reduce VPC-CNI’s footprint to shrink the blast radius of IP exhaustion. If only 5 services needing fixed IPs use VPC-CNI, IP pool management only applies to those services’ nodes. If the entire cluster uses VPC-CNI, every node is a potential IP exhaustion point.

Comparison with ACK

If you’re also evaluating Alibaba Cloud ACK, the comparison is apt: ACK’s Terway plugin has similar ENI shared and dedicated modes, with identical selection logic. Cloud vendors’ CNI solutions follow the same fundamental approach: either sacrifice performance for address abundance via bridge forwarding, or gain performance with direct VPC attachment but accept IP quota constraints. Don’t get confused by each vendor’s marketing — the underlying logic is the same. (Related: K8s Networking Model: CNI and Service Networking)

Decision 2: VPC-CNI’s IP Pool Isn’t Unlimited — Pre-binding Count Is the First Trap

If you’ve chosen VPC-CNI (whether pure VPC-CNI or mixed mode), IP pool management is something you must understand.

How the IP Pool Works

Under VPC-CNI shared ENI mode, TKE’s IPAMD component maintains an elastically scaling IP pool on each node. The key parameters are the pre-binding counts:

• Minimum pre-binding count (default 5): when bound IPs < pod count + 5, IPAMD actively binds more IPs
• Maximum pre-binding count (default 5): when bound IPs > pod count + 5, IPAMD periodically releases excess IPs (approximately every 2 minutes)

So with 5 pods on a node, IPAMD maintains 10 bound IPs (5 for pods + 5 reserved). When a pod is created, it gets a random IP from the pool; when destroyed, the IP returns to the pool without immediately releasing to the VPC.

Failure Scenario: Pre-binding Too Small Causes Scaling Stalls

The default min/max of 5 seems fine. But if you’re using HPA autoscaling, a burst of pod scaling requests during traffic peaks can drain the pre-bound IP pool. New pods must wait for IPAMD to bind new IPs before scheduling succeeds. Binding ENI IPs calls the cloud API — sub-second latency per call, but batch binding may take 10-30 seconds.

# Check node ENI IP usage
kubectl get node <node-name> -o jsonpath='{.status.allocatable.tke\.cloud\.tencent\.com/eni-ip}'
# Output: 30  ← allocatable ENI IP count

kubectl describe node <node-name> | grep eni-ip
# Output:
# tke.cloud.tencent.com/eni-ip    30          28          2           2
#                                  capacity   allocated   remaining   request
# Above means: total capacity 30, allocated 28, remaining 2

Tuning Recommendations

Adjust pre-binding count based on your workload characteristics:

Configuration method — edit IPAMD config:

# Modify tke-eni-ipamd configuration
kubectl edit deploy tke-eni-ipamd -n kube-system
# Add/modify in args:
#   - --min-ip=10
#   - --max-ip=15

Note: Changing pre-binding count only affects incremental behavior. Already-bound IPs won’t release immediately — IPAMD adjusts to the new range during the next periodic check (~2 minutes).

Must-Have Monitoring Metric

Under VPC-CNI mode, here’s the Prometheus alert I consider mandatory:

# PromQL query for node ENI IP utilization
# Alert at 80%, pods will Pending near 100%
- alert: TKEENIIPExhaustion
  expr: |
    1 - (
      kube_node_status_allocatable{resource="tke_cloud_tencent_com_eni-ip"} 
      / 
      kube_node_status_capacity{resource="tke_cloud_tencent_com_eni-ip"}
    ) > 0.8    
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Node {{ $labels.node }} ENI IP utilization above 80%"
    description: "ENI IP running low, pods may enter Pending soon"

This alert fires 10-30 minutes before IP exhaustion, giving you enough time to scale or release IPs. Don’t wait for pods to go Pending before discovering the problem. (Related: Kubernetes Autoscaling: HPA/VPA/CA Deep Dive)

Decision 3: Node Pool Autoscaling “Silent Failure” — Discovering Resource Shortages Only When Scaling

TKE’s autoscaling is built on Tencent Cloud AS (Auto Scaling) and the community cluster-autoscaler. This chain has a hidden failure mode: everything looks fine normally, but when scaling is actually triggered, it fails.

What Is “Silent Failure”

The node pool configuration looks fine: instance type selected, security group configured, subnet chosen. But when autoscaling triggers, any of these reasons cause failure:

1. Instance type out of stock: CVM instance spec sold out in the availability zone (common during peaks)
2. Subnet IP exhaustion: remaining subnet IPs insufficient for new node ENIs
3. Missing SSH key: associated SSH key deleted or permissions changed
4. Security group changes: associated security group modified causing rule mismatch
5. Insufficient account balance: pay-as-you-go instances fail to create due to low balance

None of these errors surface at configuration time — they only manifest when scaling triggers. Worse, cluster-autoscaler’s scaling failure logs are at a low verbosity level by default, invisible unless you actively check.

TKE Elastic Health Feature

In 2026, TKE launched the “Elastic Health” feature, which proactively detects these risks before scaling is triggered. It defines two core metrics:

• Effective inventory (Xi): maximum nodes a single resource pool can scale to, Xi = min(stock level, remaining subnet IPs)
• Elastic resilience (N): number of resource pools with effective inventory > 0, reflecting scaling option diversity

Health status has three levels:

I recommend enabling elastic health checks for all node pools using autoscaling, with alert notifications configured. Far more efficient than troubleshooting after a failure occurs.

Production Configuration Recommendations

# View node pool autoscaling configuration
tccli tke DescribeClusterNodePools \
  --ClusterId cls-xxxxxxxx \
  --filters.0.Name node-pool-id \
  --filters.0.Values np-xxxxxxxx

# Key configuration checks:
# 1. MultiZoneSubnetPolicy: recommend EQUALITY for multi-AZ distribution
# 2. Alternative instance types: configure at least 2-3 alternatives
# 3. RetryPolicy: recommend IMMEDIATE_RETRY to avoid transient failures

Instance type strategy: configure at least 2-3 alternative types. I recommend “performance tiers” rather than all the same spec:

Primary: SA5.LARGE8 (2-core 8GB) — daily workload
Alternative 1: SA5.XLARGE16 (4-core 16GB) — fallback when primary sold out
Alternative 2: S5.LARGE8 (2-core 8GB) — cross-generation fallback, more stock

This way, even if one spec in one AZ sells out, AS can try others — not a complete scaling failure.

Multi-AZ strategy: configure subnets in 2+ availability zones. TKE’s MultiZoneSubnetPolicy supports PRIORITY (by priority) and EQUALITY (balanced allocation). I recommend EQUALITY — slightly increases cross-AZ latency but dramatically improves scaling success rate.

Decision 4: Mixed Mode Isn’t a Silver Bullet — GlobalRouter + VPC-CNI Boundary Management

Mixed mode (GlobalRouter as default + VPC-CNI enabled on demand) is the recommended approach for most scenarios. But mixing introduces new governance challenges.

Pod Scheduling Network Mode Attribution

In mixed mode, pods default to GlobalRouter. Only pods with the explicit annotation use VPC-CNI:

# Default to GlobalRouter (no annotation needed)
apiVersion: v1
kind: Pod
metadata:
  name: normal-service
spec:
  containers:
  - name: app
    image: nginx:1.25

# Specified to use VPC-CNI
apiVersion: v1
kind: Pod
metadata:
  name: fixed-ip-service
  annotations:
    k8s.v1.cni.cncf.io/networks: tke-vpc-cni
spec:
  containers:
  - name: app
    image: nginx:1.25

Network communication between VPC-CNI fixed-IP pods and GlobalRouter pods works fine — both are in the same VPC, with routing handled by VPC’s underlying layer. But the following differences require attention:

Bandwidth Rate Limiting Differences

Both GlobalRouter and VPC-CNI shared ENI mode support the community bandwidth plugin for pod-level rate limiting, but configuration differs:

# GlobalRouter mode: modify tke-bridge-agent
kubectl edit daemonset tke-bridge-agent -n kube-system
# Add --bandwidth to args to enable the plugin

# VPC-CNI shared ENI mode: modify eniipamd component config
# In component management page, set agent.cniChaining.bandwidth to true

Then specify rate limits via annotations:

metadata:
  annotations:
    kubernetes.io/ingress-bandwidth: "100M"
    kubernetes.io/egress-bandwidth: "50M"

Note: VPC-CNI dedicated ENI mode does not support the bandwidth plugin. If your workload requires dedicated ENIs (e.g., high-performance computing), pod-level rate limiting isn’t available — you’ll need VPC-level traffic control instead.

Pod Security Group Dependencies

VPC-CNI mode supports pod-level security groups (via the SecurityGroupPolicy component), allowing independent security group rules per pod. GlobalRouter mode doesn’t support pod-level security groups — only node security groups.

In mixed mode, if certain VPC-CNI pods need specific security group rules, you must install the SecurityGroupPolicy component in the cluster, then create SecurityGroupPolicy resources:

apiVersion: networking.tke.cloud.tencent.com/v1
kind: SecurityGroupPolicy
metadata:
  name: payment-sg-policy
  namespace: production
spec:
  securityGroups:
  - sg-xxxxxxxx
  podSelector:
    matchLabels:
      app: payment-service

But note: SecurityGroupPolicy currently only works for pods on super nodes. VPC-CNI pods on regular and native nodes still use node security groups. This limitation is frequently overlooked, leading to configured security group rules that don’t take effect.

Mixed Mode Governance Recommendations

I recommend concentrating VPC-CNI pods onto dedicated node pools in mixed mode, rather than mixing them with GlobalRouter pods on the same pool. Two reasons:

1. VPC-CNI pods have different IP consumption patterns than GlobalRouter pods — separate capacity management is clearer
2. Dedicated node pools allow targeted ENI IP monitoring and pre-binding configuration without affecting GlobalRouter nodes

Implementation with nodeSelector or nodeAffinity:

spec:
  nodeSelector:
    network-mode: vpc-cni
  # Label the node pool: kubectl label node <node> network-mode=vpc-cni

Decision 5: Node Type Selection Is More Critical Than Network Selection

TKE offers three node types: regular nodes, native nodes, and super nodes. Many focus on network models during selection and overlook how node types affect operations. In reality, node type determines the upgrade path, failure recovery approach, and cost model.

Three Node Types Compared

My Selection Strategy

Regular nodes: suitable when you need full OS control (custom monitoring agents, special kernel tuning). But heaviest operational burden — CVE fixes require manual drain + node replacement. Generally not recommended for production.

Native nodes: recommended as the primary node type. TKE manages the OS, with unified upgrade and CVE fix processes. Flexible network mode support for both GlobalRouter and VPC-CNI.

Super nodes: ideal for elastic scaling scenarios. Second-level pod scheduling, usage-based billing, no CVM pre-purchase needed. Two limitations: only supports VPC-CNI network mode, and doesn’t support DaemonSet (DaemonSet pods won’t schedule onto super nodes). If your business relies on node-level DaemonSets (log collection agents, monitoring agents), those agents won’t exist on super nodes — you’ll need Sidecar or remote collection alternatives.

CVE Fix Differences

This difference matters during security incidents. Take CVE-2026-31431:

• Regular nodes: wait for CVM public image patch update, then manually drain old node → remove from cluster → create new node (defaults to patched image). The process may take 1-2 hours per node
• Native nodes: wait for TKE to publish patch, same replacement flow but TKE manages the OS version
• Super nodes: wait for TKE patch, then simply rebuild pods (delete and let controller reschedule) — fixes apply automatically

If your cluster has 50 regular nodes needing patches, manual replacement is enormous work. This is why I recommend native or super nodes — let TKE handle OS maintenance while the team focuses on the application layer.

Note on Cilium-Overlay

The Cilium-Overlay mode mentioned earlier is currently supported primarily on native nodes. It uses eBPF to replace traditional bridges, combining GlobalRouter’s address abundance with VPC-CNI’s performance advantages. If your cluster version is recent enough (K8s 1.28+), it’s worth trying. However, I don’t yet have enough production cases to make a recommendation — the mature GlobalRouter + VPC-CNI mixed approach remains the safer bet.

Summary

Back to the 2 AM incident. The root cause was VPC-CNI mode subnet IPs being fully allocated, preventing all nodes from binding new IPs, causing pods to go Pending en masse. The fix was painful — emergency subnet CIDR expansion, waiting for IPAMD to rebind, restoring Pending pods one by one.

After the postmortem, I did three things:

1. Switched the cluster to GlobalRouter + VPC-CNI mixed mode. Only 3 services needing fixed IPs use VPC-CNI; everything else runs on GlobalRouter. The IP exhaustion blast radius shrank from the entire cluster to a dedicated node pool for 3 services
2. Added ENI IP utilization monitoring. Added the tke.cloud.tencent.com/eni-ip allocatable/capacity metric to Prometheus, with an 80% threshold alert
3. Configured elastic health checks and multiple alternative instance types for node pools. No longer relying on a single instance type, avoiding scaling failures from stock shortages

The core logic of the 5 production decisions:

One final thought: cloud-managed services do save a lot of basic ops work, but “managed” doesn’t mean “maintenance-free.” TKE manages the master nodes and control plane, but networking, IP pools, node pools, and security groups are still your responsibility. Think one step further at selection time, and you’ll have one fewer 2 AM wake-up call in production.

References & Acknowledgments

The following resources were referenced during the writing of this article. Thanks to the original authors for their contributions:

1. TKE Container Network Overview — Tencent Cloud, technical principles and comparison of VPC-CNI/GlobalRouter/Cilium-Overlay network solutions
2. VPC-CNI Mode Introduction — Tencent Cloud, VPC-CNI container networking implementation based on CNI and VPC elastic network interfaces
3. Multi-Pod Shared ENI Mode — Tencent Cloud, IP pool management principles, pre-binding count and elastic scaling mechanisms
4. TKE ENI Direct-Connect Pod Network Load Balancing — Tencent Cloud, performance comparison and selection between direct-connect and NodePort forwarding
5. Cluster Selection Recommendations — Tencent Cloud, official selection guidance for GlobalRouter and VPC-CNI
6. Fixed IP Usage — Tencent Cloud, enabling and using VPC-CNI fixed IP mode
7. Elastic Health — Tencent Cloud, node pool autoscaling risk assessment and proactive operations capability
8. Pod Bandwidth Rate Limiting on TKE — Tencent Cloud, bandwidth rate limiting configuration under GlobalRouter and VPC-CNI modes
9. Pod Security Groups — Tencent Cloud, TKE pod-level security group policies usage and limitations
10. CVE-2026-31431 Fix Instructions — Tencent Cloud, CVE fix method differences across node types