VPC Peering Became Your Bottleneck: Cloud-Native Network Architecture Selection and 5 Production-Grade Decisions

Overview

At 2 AM, the alert group for a ride-hailing project exploded. The order service P99 latency spiked from 80ms to 600ms, and client-side timeouts were spreading. After a full round of troubleshooting, the culprit turned out to be VPC Peering — the cross-VPC call volume between two core business VPCs had saturated the concurrent connection limit of a single Peering link during peak hours. Meanwhile, 3 out of 12 new Peering connections created via Full Mesh for another recently launched VPC failed due to CIDR overlap, causing routing table chaos that sent some traffic through suboptimal paths.

This isn’t an isolated case. Once your cloud VPC count exceeds 5, the maintenance cost of Full Mesh Peering starts growing exponentially. Beyond 10, the operations team is basically firefighting.

I once led the K8s migration of 120+ microservices at a new-energy logistics platform. Before the migration, we invested significant effort in network architecture review. I noticed a pattern: most team’s network failures don’t originate at the application layer — they’re baked in during the network architecture selection phase. VPC Peering looks simple and direct, but it has three fundamental flaws: no transitive routing, Full Mesh connection growth of N*(N-1)/2, and no centralized route policy management. By the time you discover these problems, the architecture is already locked in, and the cost of rework far exceeds the two days of upfront decision-making you skipped.

This article doesn’t cover “what is a VPC.” I assume you’re already using cloud, managing VPCs, and have felt the pain of multi-VPC interconnection. What I’ll break down: when to use VPC Peering, when to switch to Transit Gateway / CEN Transit Router, how to do hybrid networking, and how to plan CIDR blocks to avoid “network goes live, then everything needs rework.” Each decision point comes with real production data and hard-learned lessons — not generic “both have pros and cons.”

Three Fatal Flaws of VPC Peering

VPC Peering is a point-to-point network connection between two VPCs. Traffic flows over the cloud provider’s backbone network. Same-region peering is free with the lowest latency. Sounds perfect. But it has three unavoidable problems in production.

Flaw 1: No Transitive Routing

This is the most critical one. Suppose you have VPC-A, VPC-B, and VPC-C. A peers with B, and B peers with C. You’d think A could reach C through B? No. AWS documentation explicitly states: VPC Peering does not support transitive routing — two VPCs without direct peering cannot communicate through a third VPC.

What does this mean? To make N VPCs mutually reachable, you need pairwise peering connections — the Full Mesh model. The connection count is N*(N-1)/2. Five VPCs need 10 connections, ten need 45, twenty need 190.

In a real project, I saw 15 VPCs running Full Mesh at an e-commerce platform. Every time a new VPC was added, the ops team had to manually configure 14 Peering connections, including route tables and security group rules on both ends of each. One new VPC launch took 3 days, 2 of which were spent on network configuration alone.

Flaw 2: Scattered Routing, No Centralized Control

In Full Mesh mode, routes for each peering connection must be manually added to both VPCs’ route tables. There’s no unified control plane. This causes three problems:

1. Route drift: During one change, an ops engineer forgot to add the return route on the peer VPC’s route table, resulting in one-way reachability — A could ping B, but B’s response packets couldn’t get back. It took 2 hours to find the missing route table entry.

2. No route policies: You can’t say “VPC-A’s production subnet can access VPC-B’s database subnet, but VPC-A’s test subnet cannot.” Peering operates at the VPC level — there’s no finer-grained routing control. You can only patch it with security group rules.

3. No change audit: Who changed which route and when? No centralized log. When a failure occurs, you can only dig through CloudTrail entry by entry.

Flaw 3: CIDR Overlap Blocks Peering

VPC Peering requires non-overlapping CIDR blocks between the two VPCs. Poor initial planning turns this into a ticking time bomb.

Real case: A ride-hailing project initially used 10.0.0.0/16 for both dev and test environments. When they later needed to connect the two for integration testing, Peering couldn’t be established. The solutions: either change one VPC’s CIDR (requires reassigning IPs for all instances, at least 4 hours of downtime), or use a NAT gateway for address translation (adds latency and cost, and source IP changes mean all security policies need rewriting).

They chose the CIDR change. During execution, they discovered that RDS instances in the VPC needed restart after subnet migration. The planned 2-hour window ballooned to 6 hours. The PM was not pleased.

Transit Gateway / CEN Transit Router: The Hub-Spoke Solution

The three flaws of VPC Peering all stem from the same root: point-to-point connections inherently lack a central control plane. Transit Gateway (AWS) and CEN Transit Router (Alibaba Cloud) solve this with the Hub-Spoke model.

In simple terms: all VPCs connect to a central gateway (Hub). VPCs don’t need pairwise connections. The gateway handles route propagation, policy control, and traffic forwarding.

How It Works

Alibaba Cloud documentation defines the Transit Router as: a core forwarding component that acts as a central router, connecting different types of network instances (VPC, VBR, CCN, VPN gateway, etc.). One Transit Router instance is created per region, responsible for intra-region traffic forwarding and cross-region route scheduling.

AWS Transit Gateway describes it similarly: a regional virtual router that routes traffic between VPCs and on-premises networks, forwarding at Layer 3 based on destination IP.

Both share the same core mechanism: Hub-Spoke + route propagation.

Comparison: VPC Peering vs Transit Gateway / Transit Router

This table isn’t about picking “the better one.” Both approaches have their use cases — the key is your VPC count, traffic model, and management requirements.

My Recommendations

• VPC count <= 5, simple traffic model: Use VPC Peering. Free, lowest latency, manageable configuration. Five VPCs = 10 connections, done in an afternoon.
• VPC count 6-15, or multi-environment isolation needs: Start considering Transit Gateway / Transit Router. Ten VPCs in Full Mesh = 45 connections — maintenance cost is already unacceptable.
• VPC count > 15, or multi-account, multi-region, hybrid cloud needs: Transit Gateway / Transit Router is mandatory. Full Mesh at this scale is unmanageable.
• A few VPC pairs with high direct traffic: Hybrid networking. High-traffic pairs use Peering (free, no bandwidth limit), the rest go through the Transit Router. This is Alibaba Cloud’s officially recommended combination approach.

5 Production-Grade Decisions

These 5 decision points were earned through real production incidents — lessons paid for with time and money. Each gives specific scenarios and data, not “it depends” hedging.

Decision 1: When to Migrate from Full Mesh to Hub-Spoke

Conclusion: Start planning migration at 8 VPCs, complete migration by 12.

Why 8? Because 8 VPCs in Full Mesh = 28 connections. At ~15 minutes per connection (route tables + security groups on both ends), a single new VPC requires 7 operations x 15 minutes = ~2 hours of network configuration. That’s before change approval and verification. At 12 VPCs (66 connections), a new VPC takes half a day — the ops team can’t keep up.

Migration strategy: Don’t do a big-bang cutover. Follow this sequence:

1. Create the Transit Gateway / Transit Router instance first
2. Attach non-critical VPCs first (dev, test environments), verify route propagation and connectivity
3. Attach core VPCs one by one, deleting corresponding Peering connections after each
4. After the final batch, clean up all remaining Peering connections

Pitfall: During migration, old and new connection types coexist. If a VPC has both Peering and Transit Gateway connections, the route table will have two paths to the same destination. AWS Transit Gateway and Alibaba Cloud Transit Router both handle this through route priority — but you must ensure the priority is configured correctly, otherwise traffic may follow stale Peering paths.

During a migration at a ride-hailing project, I found that the Peering route metric in one VPC’s route table was lower (higher priority) than the Transit Gateway route. Traffic was still using the old Peering after migration. It took an hour to find the route table priority issue.

Decision 2: CIDR Planning — Don’t Discover Conflicts After Going Live

Conclusion: Spend a day on full-network CIDR planning before creating the first VPC.

CIDR planning is the “foundation” of cloud network architecture. A shaky foundation means everything built on top will be crooked. I’ve seen too many teams say “create the VPC first, figure it out later” — only to discover overlapping CIDRs six months later when trying to interconnect.

Planning principles:

1. Layer by environment: Production 10.1.0.0/16, staging 10.2.0.0/16, testing 10.3.0.0/16. Different environments are naturally isolated. Future Peering or Transit Router connections won’t conflict.

2. Segment by region: North 10.1.0.0/18, East 10.1.64.0/18, South 10.1.128.0/18. Same environment in different regions has non-overlapping VPC CIDRs.

3. Reserve expansion space: Don’t allocate all of 10.0.0.0/8. Leave at least 30% address space for future regions, environments, and business lines.

4. Reserve landing subnets for Transit Gateway: Alibaba Cloud official practice recommends planning two /29 subnets per VPC as TR-attach landing subnets, using different availability zones for HA. This separates business traffic from TR management traffic without consuming business CIDR ranges.

Anti-pattern: A team used 10.0.0.0/16, 10.0.0.0/16, and 10.0.1.0/24 for three VPCs. The second VPC’s CIDR completely overlapped with the first — Peering was impossible. They had to delete and recreate the second VPC, losing 2 days.

CIDR Planning Reference:

Decision 3: Hybrid Networking — Combining Peering and Transit Router

Conclusion: High-traffic VPC pairs use Peering, the rest go through Transit Router. Use a shared services account for Transit Router cost allocation.

This is the approach recommended by Alibaba Cloud in their autonomous driving enterprise X case study, and it’s the combination strategy I use most frequently in production.

Scenario: You have 8 VPCs. VPC-1 and VPC-2 have large data sync requirements (TB-level daily traffic). The rest have low inter-VPC traffic.

Solution:

• VPC-1 and VPC-2 use VPC Peering (free, no bandwidth limit within same region)
• The other 6 VPCs all attach to the Transit Router
• VPC-1 and VPC-2 also attach to the Transit Router for communication with other VPCs
• Deploy the Transit Router in a shared services account, with costs allocated across business lines

Benefits: Large traffic between VPC-1 and VPC-2 flows over Peering with zero data processing fees. Small traffic from other VPCs enjoys centralized management through the Transit Router.

Pitfall: In hybrid networking, VPC-1 has both Peering (to VPC-2) and Transit Router connections (to other VPCs). The route table may have two paths to the same destination CIDR. You need to set correct priorities in the route table:

# Alibaba Cloud: view Transit Router route table
aliyun cen describe-transit-route-table-attributes \
  --transit-route-table-id rtb-xxx \
  --region cn-hangzhou

# AWS: view Transit Gateway route table
aws ec2 search-transit-gateway-routes \
  --transit-gateway-route-table-id tgw-rtb-xxx \
  --filters "Name=type,Values=propagated"

Route priority rule: more specific prefix wins. If the Peering route is 10.2.0.0/16 and the Transit Router route is 10.2.0.0/18, the latter is more specific — traffic goes through the Transit Router. If you want traffic to go through Peering, ensure the Peering route prefix isn’t broader than the Transit Router’s.

During one troubleshooting session, I found that the Transit Router’s auto-propagated route was broadcasting the entire 10.2.0.0/16 range, overriding the manually configured Peering route (same prefix — propagated routes have higher priority than manual ones). I resolved it by adding a more specific blackhole route in the Transit Router — but that’s a trick, not the proper approach. The correct method is configuring prefix filters in the Transit Router’s route policy to prevent propagated routes from covering Peering paths.

Decision 4: Cross-Region Connectivity — Bandwidth Package vs Pay-Per-Traffic

Conclusion: Steady traffic uses bandwidth packages, burst traffic uses pay-per-traffic, hybrid is usually optimal.

Cross-region connectivity is a common cloud networking need — production in Beijing, DR in Shanghai, VPCs in both regions need to communicate. This requires cross-region connections.

Alibaba Cloud CEN cross-region connections offer two billing modes:

AWS Transit Gateway cross-region connections bill by connection hours + data processing fees.

My decision model:

1. Collect 30 days of cross-region traffic data, calculate daily average and peak
2. If daily average > 500GB/day, bandwidth package is more cost-effective (fixed bandwidth, traffic costs capped)
3. If daily average < 100GB/day and highly variable, pay-per-traffic (pay for what you use, no waste)
4. Between 100-500GB/day, calculate total cost for both options, pick the cheaper one

Measured data (Alibaba Cloud, East to North, 2026):

• Pay-per-traffic: 0.8 CNY/GB, daily 200GB → 160 CNY/day → 4,800 CNY/month
• 10Mbps bandwidth package: ~3,500 CNY/month, max ~105GB/day → insufficient
• 50Mbps bandwidth package: ~12,000 CNY/month, max ~527GB/day → underutilized, wasteful
• Hybrid: 20Mbps package (5,000 CNY/month) + excess billed per traffic → avg 6,500 CNY/month, covers 200GB/day steady + occasional 400GB bursts

Conclusion: Hybrid is optimal, saving ~30% cost. But requires rate limiting to prevent burst traffic from saturating the bandwidth package and causing uncontrolled pay-per-traffic charges.

Decision 5: Security and Compliance — VPC Isolation Under Zero Trust

Conclusion: Use Transit Router route policies for network-level isolation, security groups for host-level isolation — two layers of defense.

In China’s MLPS 2.0 (Cybersecurity Classified Protection) compliance scenarios, network isolation between security domains is a hard requirement. VPC Peering can’t do fine-grained route isolation — Peering is all-or-nothing at the VPC level. You can’t control “only VPC-A’s production subnet can access VPC-B’s database subnet” at the routing layer.

Transit Router supports route policies and VRF isolation. AWS Transit Gateway Connect implements on-premises network segmentation through GRE tunnels and VRF, with each VRF corresponding to an independent routing domain.

Alibaba Cloud Transit Router supports route policies based on prefix and source/destination CIDR:

# Alibaba Cloud: create route policy to block test environment from accessing production DB
aliyun cen create-route-policy \
  --cen-id cen-xxx \
  --transit-router-route-table-id rtb-xxx \
  --route-policy-name "block-test-to-prod-db" \
  --route-policy-priority 10 \
  --route-policy-type "drop" \
  --source-cidr "10.3.0.0/18" \
  --destination-cidr "10.1.8.0/22"

# Verify route policy is active
aliyun cen list-route-policy \
  --cen-id cen-xxx \
  --transit-router-route-table-id rtb-xxx

Security Layered Model:

During an MLPS 2.0 audit, the auditor required proof that “the test environment cannot access the production database.” Using the Transit Router’s route policy drop rule + packet capture verification, we passed in one shot. If we’d used Peering + security groups, the auditor would question “what if the security group rule was modified?” Route policies are declarative with auditable change logs — much higher trust.

Architecture Decision Trade-off Analysis

The 5 decision points above are summarized here as a complete trade-off comparison. Not a single “best answer” — you choose based on actual conditions.

Option A: Pure VPC Peering (Full Mesh)

Suitable for: <=5 VPCs, no multi-account, no cross-region, no compliance isolation needs

Pros: Free within same region, lowest latency, simple configuration

Cons: N*(N-1)/2 growth, no centralized routing, no route policies, CIDR overlap blocks peering

Annual cost (5 VPCs, 12 months):

• Network fees: 0 CNY
• Ops effort: ~20 person-days/year (route table maintenance + troubleshooting)
• Risk cost: Medium (30% probability of route drift per year)

Option B: Pure Transit Gateway / Transit Router

Suitable for: >5 VPCs, or multi-account/cross-region/compliance needs

Pros: Linear connection growth, automatic route propagation, route policies and QoS, centralized monitoring and audit

Cons: Instance and data processing fees, +1 hop latency (~1-2ms), learning curve

Annual cost (10 VPCs, 12 months, Alibaba Cloud East):

• Transit Router instance fee: ~0.5 CNY/hour x 24 x 365 = 4,380 CNY/year
• Data processing fee: ~0.143 CNY/GB x daily 500GB x 365 = 26,098 CNY/year
• Ops effort: ~5 person-days/year
• Risk cost: Low (centralized control reduces route drift probability)

Option C: Hybrid Networking (Peering + Transit Router)

Suitable for: A few VPC pairs with high traffic, rest with low traffic

Pros: High-traffic pairs use Peering to save money, rest enjoy centralized control through Transit Router, maximum flexibility

Cons: Route priority management complexity, maintaining two connection types, troubleshooting requires checking both path types

Annual cost (10 VPCs, 2 high-traffic pairs, 12 months):

• Peering fees: 0 CNY (free within same region)
• Transit Router fees: ~30,000 CNY/year
• Savings from high-traffic pairs avoiding data processing fees: ~15,000 CNY/year
• Net cost: ~15,000 CNY/year
• Ops effort: ~8 person-days/year (more than pure Transit Router due to dual management)

Three-Way Comparison

My recommendations:

• Small team / startup phase (<=5 VPCs): Option A. Sufficient, free, don’t over-optimize prematurely.
• Mid-size team / growth phase (6-15 VPCs): Option B. Scalability is the primary consideration — don’t wait until Full Mesh is unmaintainable before migrating.
• Large team / mature phase (>15 VPCs or special traffic models): Option C. High-traffic pairs use Peering to save money, rest use Transit Router for management. But configure route policies properly — don’t let route priority chaos undermine the architecture.

Production Operations

Managing Network Architecture with Terraform

Manually creating Peering and Transit Router connections is unmanageable at scale. IaC with Terraform (Getting Started with Terraform IaC) is mandatory.

# AWS Transit Gateway + VPC Attachment example

# Create Transit Gateway
resource "aws_ec2_transit_gateway" "main" {
  description                     = "Production Transit Gateway"
  default_route_table_association = "enable"
  default_route_table_propagation = "enable"
  dns_support                     = "enable"
  multicast_support               = "disable"

  tags = {
    Name        = "prod-tgw"
    Environment = "production"
    ManagedBy   = "terraform"
  }
}

# VPC Attachment (connect VPC to Transit Gateway)
resource "aws_ec2_transit_gateway_vpc_attachment" "vpc_a" {
  transit_gateway_id = aws_ec2_transit_gateway.main.id
  vpc_id             = var.vpc_a_id
  subnet_ids         = var.vpc_a_subnet_ids
  dns_support        = "enable"

  tags = {
    Name = "vpc-a-tgw-attachment"
  }
}

# Cross-region Transit Gateway Peering
resource "aws_ec2_transit_gateway_peering_attachment" "cross_region" {
  transit_gateway_id      = aws_ec2_transit_gateway.main.id
  peer_transit_gateway_id = var.peer_tgw_id
  peer_region             = var.peer_region

  tags = {
    Name = "cross-region-tgw-peering"
  }
}

# Accept cross-region peering (executed in peer region)
resource "aws_ec2_transit_gateway_peering_attachment_accepter" "cross_region" {
  transit_gateway_attachment_id = aws_ec2_transit_gateway_peering_attachment.cross_region.id

  tags = {
    Name = "cross-region-tgw-peering-accepter"
  }
}

# Alibaba Cloud: create CEN Transit Router connection with Terraform
# Requires aliyun/terraform-provider-alicloud

# Define CEN instance
resource "alicloud_cen_instance" "main" {
  name        = "prod-cen"
  description = "Production CEN for multi-VPC networking"
}

# Create Transit Router
resource "alicloud_cen_transit_router" "main" {
  cidr        = "172.17.0.0/16"
  cen_id      = alicloud_cen_instance.main.id
  description = "Production Transit Router"
}

# Attach VPC to Transit Router
resource "alicloud_cen_transit_router_vpc_attachment" "vpc_a" {
  transit_router_id                    = alicloud_cen_transit_router.main.id
  cen_id                               = alicloud_cen_instance.main.id
  vpc_id                               = var.vpc_a_id
  zone                                 = var.vpc_a_zone
  vpc_owner_id                         = "account-id"
  transit_router_attachment_name       = "vpc-a-tr-attachment"
  transit_router_attachment_description = "VPC A to Transit Router"

  # Specify landing subnets
  vpc_id   = var.vpc_a_id
  zone     = var.vpc_a_zone
}

Benefits of Terraform management:

• Network changes are auditable (terraform plan previews, terraform apply executes)
• Environments are reproducible (one codebase creates identical dev/staging architecture)
• Changes are rollbackable (terraform state records all resource states)

Want to dive deeper into Terraform State management? Check out Related: Lost Your State File?.

Route Policy Configuration Example

Below is a production route policy configuration for environment isolation:

#!/usr/bin/env python3
"""
VPC route policy automated checker.
Inspects Transit Gateway route tables for policy violations.
"""

import boto3
import json
from datetime import datetime

class TransitGatewayRouteChecker:
    def __init__(self, region="cn-hangzhou"):
        self.ec2 = boto3.client("ec2", region_name=region)
        # Violation rule definitions
        self.violation_rules = [
            # Test environment must not access production database
            {"source": "10.3.0.0/18", "dest": "10.1.8.0/22", "action": "should_not_exist"},
            # Dev environment must not access any production subnet
            {"source": "10.3.64.0/18", "dest": "10.1.0.0/18", "action": "should_not_exist"},
        ]

    def get_tgw_route_tables(self, tgw_id):
        """Get all Transit Gateway route tables."""
        response = self.ec2.describe_transit_gateway_route_tables(
            Filters=[{"Name": "transit-gateway-id", "Values": [tgw_id]}]
        )
        return response.get("TransitGatewayRouteTables", [])

    def get_routes(self, route_table_id):
        """Get all routes in a route table."""
        response = self.ec2.search_transit_gateway_routes(
            TransitGatewayRouteTableId=route_table_id,
            Filters=[{"Name": "state", "Values": ["active", "pending"]}],
        )
        return response.get("Routes", [])

    def check_violations(self, tgw_id):
        """Check for policy-violating routes."""
        route_tables = self.get_tgw_route_tables(tgw_id)
        violations = []

        for rt in route_tables:
            routes = self.get_routes(rt["TransitGatewayRouteTableId"])
            for route in routes:
                for rule in self.violation_rules:
                    if (route.get("DestinationCidr") == rule["dest"] and
                        route.get("Source")):
                        violations.append({
                            "route_table": rt["TransitGatewayRouteTableId"],
                            "destination": route.get("DestinationCidr"),
                            "rule": rule,
                            "timestamp": datetime.now().isoformat(),
                        })

        return violations

    def report(self, tgw_id):
        """Generate a check report."""
        violations = self.check_violations(tgw_id)
        if violations:
            print(f"[ALERT] Found {len(violations)} violating routes:")
            for v in violations:
                print(f"  - Route table {v['route_table']}: "
                      f"dest={v['destination']} "
                      f"violates rule={v['rule']}")
            # Send alert
            self._send_alert(violations)
        else:
            print("[OK] Route policy check passed, no violations found")

    def _send_alert(self, violations):
        """Send alert to Feishu/DingTalk (simplified example)."""
        # In production, integrate with Feishu webhook
        print(f"  -> Alert sent, {len(violations)} violations")


if __name__ == "__main__":
    checker = TransitGatewayRouteChecker(region="cn-hangzhou")
    checker.report("tgw-xxx")

Network Connectivity Verification

After migration or changes, verify network connectivity. Don’t just test ping — ICMP reachability doesn’t mean TCP works, and TCP working doesn’t mean the application layer is fine.

#!/bin/bash
# VPC connectivity verification script
# Usage: ./vpc_connectivity_check.sh <source_ip> <target_ip> <target_port>

SOURCE_IP=$1
TARGET_IP=$2
TARGET_PORT=${3:-80}

echo "=== VPC Connectivity Verification ==="
echo "Source: $SOURCE_IP"
echo "Target: $TARGET_IP:$TARGET_PORT"
echo ""

# Layer 3: ICMP connectivity
echo -n "[L3] ICMP ping... "
if ping -c 3 -W 2 "$TARGET_IP" > /dev/null 2>&1; then
    echo "OK"
else
    echo "FAILED"
    echo "  -> Check if route table has return route"
    echo "  -> Check if security group allows ICMP"
    exit 1
fi

# Layer 4: TCP port connectivity
echo -n "[L4] TCP $TARGET_PORT... "
if timeout 3 bash -c "echo > /dev/tcp/$TARGET_IP/$TARGET_PORT" 2>/dev/null; then
    echo "OK"
else
    echo "FAILED"
    echo "  -> Check if security group inbound allows $TARGET_PORT"
    echo "  -> Check target instance firewall (iptables/firewalld)"
    exit 1
fi

# Layer 7: HTTP response
echo -n "[L7] HTTP request... "
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 3 \
    "http://$TARGET_IP:$TARGET_PORT/" 2>/dev/null)
if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 500 ]; then
    echo "OK ($HTTP_CODE)"
else
    echo "FAILED ($HTTP_CODE)"
    echo "  -> Check if application service is running"
    echo "  -> Check application logs"
    exit 1
fi

echo ""
echo "=== All checks passed ==="

Why layered verification matters: After one migration, ping succeeded, and the ops engineer reported “network is fine.” But when business traffic ran, it timed out. Investigation revealed the route table only had ICMP return routes — TCP routes pointed to the old Peering. Layered verification catches this “looks connected but isn’t” problem.

Monitoring and Alerting

VPC network monitoring should focus on these core metrics:

For monitoring system design, refer to the experience in Related: Alerting Strategy Design. Alert delivery in 5 seconds, fault localization from 30 minutes to 8 minutes — this system applies equally to network monitoring.

Prometheus Metrics for VPC Network Monitoring

Using Prometheus + CloudWatch Exporter (or Alibaba Cloud Monitor Exporter) enables unified cross-VPC metric collection. Below are core PromQL alert rules:

# Prometheus alert rules: VPC network anomalies
groups:
  - name: vpc-network-alerts
    rules:
      # Peering connection packet loss exceeds threshold
      - alert: VpcPeeringPacketLoss
        expr: |
          rate(aws_vpc_peering_packets_drop_total[5m])
          / rate(aws_vpc_peering_packets_total[5m]) > 0.001          
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "VPC Peering packet loss exceeds threshold"
          description: "Peering {{ $labels.peering_id }} loss rate: {{ $value }}"

      # Transit Gateway packet drops
      - alert: TgwPacketDrop
        expr: rate(aws_tgw_packets_drop_total[5m]) > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Transit Gateway dropping packets"
          description: "TGW {{ $labels.tgw_id }} dropping packets for over 5 minutes"

      # Cross-VPC latency spike
      - alert: CrossVpcLatencySpike
        expr: |
          histogram_quantile(0.99,
            rate(vpc_cross_latency_seconds_bucket[5m])) > 0.1          
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "Cross-VPC P99 latency exceeds 100ms"
          description: "{{ $labels.source }} -> {{ $labels.target }}: {{ $value }}s"

Metric collection latency issue: CloudWatch metrics have a default 1-5 minute delay. For network incidents, a 5-minute delay means users have already reported the issue before you see metric anomalies. The solution is deploying custom probes on critical links — an instance that pings the target VPC every 10 seconds, writing results to Prometheus. This reduces latency to 10-15 seconds.

At a ride-hailing project, this approach cut cross-VPC communication failure detection from 5 minutes (CloudWatch delay) to 15 seconds (custom probing), buying 4 minutes and 45 seconds for fault localization. Don’t underestimate those minutes — for a platform processing a million orders daily, a 4-minute detection window means ~2,800 orders potentially affected.

Common Failure Scenarios and Troubleshooting

Here are three real failure scenarios with root causes and fixes.

Scenario 1: Partial VPC unreachability after migration

Symptom: After VPC-A migrated to Transit Gateway, VPC-A could reach VPC-B, but VPC-B couldn’t reach VPC-A.

Root cause: VPC-A’s route table had a return route pointing to the old Peering (deleted during TGW migration), so VPC-A’s response packets couldn’t route to VPC-B.

# Check VPC-A route table for return routes
aws ec2 describe-route-tables \
  --filters "Name=vpc-id,Values=vpc-a-id" \
  --query 'RouteTables[0].Routes[?DestinationCidrBlock==`10.2.0.0/16`]'
# Output: GatewayId points to deleted pcx-xxx

# Fix: point return route to Transit Gateway
aws ec2 replace-route \
  --route-table-id rtb-xxx \
  --destination-cidr-block 10.2.0.0/16 \
  --transit-gateway-id tgw-xxx

Scenario 2: Peering and TGW route conflict

Symptom: VPC-A and VPC-B have both Peering and TGW connections. Traffic intermittently takes the wrong path.

Root cause: TGW route propagation automatically added a 10.2.0.0/16 route, conflicting with the manually configured Peering route (same prefix). AWS rule: propagated routes take priority over manual routes.

Fix: Configure prefix filters in the TGW route table to prevent 10.2.0.0/16 from propagating through TGW, forcing traffic through Peering.

Scenario 3: CIDR overlap blocks Peering creation

Symptom: New VPC used 10.1.0.0/16, overlapping with existing VPC’s 10.1.0.0/18. Peering creation failed.

Root cause: CIDR containment. Although not identical, 10.1.0.0/18 is a subnet of 10.1.0.0/16. AWS Peering requires no CIDR overlap including containment.

Fix: Change the new VPC’s CIDR to 10.5.0.0/16, non-overlapping with all existing VPCs. The lesson: CIDR planning must be done upfront — you can’t discover conflicts at VPC creation time.

Troubleshooting Checklist

When cross-VPC communication fails, troubleshoot in this order:

# Step 1: Check VPC Peering connection status
aws ec2 describe-vpc-peering-connections \
  --filters "Name=requester-vpc-info.vpc-id,Values=vpc-xxx" \
  --query 'VpcPeeringConnections[*].[Id,Status.Code]' \
  --output table

# Step 2: Check route table for Peering routes
aws ec2 describe-route-tables \
  --filters "Name=vpc-id,Values=vpc-xxx" \
  --query 'RouteTables[*].Routes[?GatewayId!=`null` && contains(GatewayId, `pcx`)]' \
  --output table

# Step 3: Check security group rules
aws ec2 describe-security-groups \
  --filters "Name=vpc-id,Values=vpc-xxx" \
  --query 'SecurityGroups[*].IpPermissions[*].[IpProtocol,FromPort,IpRanges[*].CidrIp]' \
  --output table

# Step 4: Check Transit Gateway route propagation
aws ec2 search-transit-gateway-routes \
  --transit-gateway-route-table-id tgw-rtb-xxx \
  --filters "Name=state,Values=active" \
  --output table

# Step 5: Check network ACLs
aws ec2 describe-network-acls \
  --filters "Name=vpc-id,Values=vpc-xxx" \
  --query 'NetworkAcls[*].Entries[?Egress==`false`]' \
  --output table

Troubleshooting mantra: 1) connection status, 2) route table, 3) security groups, 4) TGW routes, 5) NACLs. 80% of VPC network issues are in the first three steps.

Disaster Recovery and Rollback

Network Architecture DR Design

Transit Router adds a central node compared to Peering, but this node has its own HA design. AWS Transit Gateway deploys elastic network interfaces in each availability zone. Alibaba Cloud Transit Router requires subnets in at least two availability zones. However, single-Region failure risk remains.

Cross-Region DR approach:

1. Deploy one Transit Gateway / Transit Router in each Region
2. Connect the two gateways via cross-region peering
3. Each VPC attaches to its local Region’s gateway
4. Cross-Region traffic routes through gateway peering
5. DNS-level traffic switching — failover to DR Region on primary Region failure

Rollback plan:

If issues arise after migrating from Peering to Transit Gateway:

1. Ensure old Peering connections aren’t deleted (retain for 30 days during migration)
2. Restore Peering route priority in route tables
3. Verify traffic flows through Peering again
4. Delete Transit Gateway Attachment (doesn’t affect Peering)

Key point: Don’t delete old Peering connections during migration. Retain a 30-day rollback window. Only clean up old connections after confirming the new architecture is stable for 30 days. This experience comes from Related: Change Management - Canary Release and Rollback — the key to change management isn’t “can the change succeed” but “can you roll back when it fails.”

Summary

Recapping the 5 production-grade decisions:

1. Migration timing: Start planning Full Mesh to Hub-Spoke migration at 8 VPCs, complete by 12. Don’t wait until it’s unmaintainable — the change risk is already too high by then.

2. CIDR planning: Spend a day on full-network CIDR planning before creating the first VPC. Segment by environment, sub-segment by region, reserve 30% expansion space, allocate TR landing subnets. Poor foundation means constant rework.

3. Hybrid networking: High-traffic pairs use Peering (free, no limits), rest go through Transit Router. Use a shared services account for Transit Router cost allocation. Configure route priorities properly — don’t let propagated routes override Peering paths.

4. Cross-region billing: Steady traffic uses bandwidth packages, burst traffic uses pay-per-traffic. Analyze 30 days of historical data to decide — hybrid is usually optimal. Configure rate limiting to prevent burst traffic from running uncontrolled.

5. Security isolation: Transit Router route policies for L3 isolation, security groups for L4 control, WAF for L7 protection. Two layers of defense are more reliable than security groups alone. MLPS auditors trust declarative route policies, not ad-hoc security group rules.

The core of cloud-native network architecture selection isn’t picking “the best solution” — it’s finding the optimal answer for your current VPC count, traffic model, management needs, and cost constraints. Options A through C aren’t mutually exclusive — they’re an upgrade path as your business scales. The key is knowing when to switch — don’t wait until the architecture is locked in to regret not making the decision earlier.

One final opinion on multi-cloud networking: if you’re not forced into multi-cloud (compliance requirements, client requirements), don’t proactively pursue it. Cross-cloud network interconnection complexity is 3x that of single-cloud, with a wider failure domain. Getting your single-cloud network architecture in order is far more pragmatic than running between multiple clouds.

References & Acknowledgments

The following resources were referenced during the writing of this article. Thanks to the original authors for their contributions:

1. VPC peering - AWS Whitepapers — AWS, official VPC Peering documentation, explicitly states no transitive routing
2. Amazon Transit Gateway — AWS, Transit Gateway official product page, Hub-Spoke model
3. Amazon Transit Gateway pricing — AWS, Transit Gateway billing rules
4. What is Cloud Enterprise Network? — Alibaba Cloud, CEN Transit Router core concepts
5. Multi-VPC interconnection with VPC Peering and Transit Router — Alibaba Cloud, hybrid networking solution and Peering vs Transit Router comparison
6. CEN Transit Router enterprise cloud networking scenarios — Alibaba Cloud, Transit Router landing subnet planning practice
7. Cross-region connections — Alibaba Cloud, bandwidth package vs pay-per-traffic comparison
8. Segmenting hybrid networks with AWS Transit Gateway connect — AWS, segmenting hybrid networks with TGW Connect and VRF
9. Multi-VPC network design in the same region — Alibaba Cloud, Well-Architected framework for multi-VPC design
10. Enterprise SD-WAN and public cloud integration: multi-cloud access architecture design and pitfalls — Tencent Cloud Developer Community, route conflicts and QoS pitfalls in multi-cloud access scenarios