I was migrating traffic between subnets and noticed that instances in subnet-A could reach instances in subnet-B, but the reverse wasn’t working. At first, I thought it was a security group issue, but after checking the rules, I realized the problem was in my route tables. One subnet was still using the main route table instead of a custom one, and another had a more specific route that was overriding my intended traffic path. In this post, I’ll walk through exactly what causes this and how to fix it.

The Problem

Traffic flows one direction but not the other, or takes an unexpected path through your network. An application in subnet-A can reach its database in subnet-B, but health checks from subnet-B timing out. Or traffic destined for a Transit Gateway is being blackholed instead of forwarded.

Here’s what you might see when testing connectivity:

Instance A (10.0.1.100) -> Instance B (10.0.2.100): SUCCESS
Instance B (10.0.2.100) -> Instance A (10.0.1.100): TIMEOUT
Problem Description
Asymmetric Routing Traffic flows one direction but not the reverse
Blackholed Routes Traffic destined for a destination gets dropped with no error
Unexpected Route Precedence More specific routes override general routes (longest prefix match)
TGW Routes Not Propagating Transit Gateway routes missing from route tables

Why Does This Happen?

  • Subnet associated with main route table instead of custom: By default, new subnets are associated with the main route table. If you create custom route tables but forget to associate a subnet, that subnet uses main. The main route table often has different or missing routes compared to your custom tables, causing asymmetric routing.

  • More specific route entries override the intended route: AWS uses longest prefix match for routing. A route to 10.0.0.0/24 is more specific than 10.0.0.0/16. If you have both and the /24 route points to a NAT instance that’s down, traffic to 10.0.0.0/24 will fail even though 10.0.0.0/16 exists.

  • Missing local route for VPC CIDR: Every route table should have a local route allowing traffic within the VPC CIDR. If this is missing or deleted, all internal communication breaks. (This is rare but can happen with API mistakes.)

  • Route pointing to a stopped or terminated NAT instance: You might be using a NAT instance instead of NAT Gateway. If that instance is stopped, all traffic through it fails. Routes silently discard traffic if the target is unavailable.

  • Transit Gateway routes not propagated: TGW routes must be explicitly enabled on each route table. Without route propagation, TGW attachments appear “connected” but routes never appear in the route table, causing traffic to be blackholed.

The Fix

First, identify which route table is associated with your subnet:

aws ec2 describe-route-tables \
  --filters "Name=association.subnet-id,Values=subnet-0a1b2c3d4e5f6g7h8" \
  --query 'RouteTables[*].[RouteTableId,Tags[?Key==`Name`].Value|[0],Associations[*].SubnetId]' \
  --output table

Check all routes in that table:

aws ec2 describe-route-tables \
  --route-table-ids rtb-0a1b2c3d4e5f6g7h8 \
  --query 'RouteTables[0].Routes[].[DestinationCidrBlock,State,GatewayId,NatGatewayId,TransitGatewayId,InstanceId]' \
  --output table

Look for routes with blackhole state — these are routes pointing nowhere:

aws ec2 describe-route-tables \
  --route-table-ids rtb-0a1b2c3d4e5f6g7h8 \
  --query 'RouteTables[0].Routes[?State==`blackhole`]'

Delete blackholed routes:

aws ec2 delete-route \
  --route-table-id rtb-0a1b2c3d4e5f6g7h8 \
  --destination-cidr-block 10.0.0.0/24

Fix Subnet Route Table Association

If a subnet is using the main route table and needs a custom one, create or select a custom route table:

aws ec2 create-route-table \
  --vpc-id vpc-0a1b2c3d4e5f6g7h8 \
  --tag-specifications 'ResourceType=route-table,Tags=[{Key=Name,Value=custom-routes}]'

Associate the subnet with the custom route table:

aws ec2 associate-route-table \
  --subnet-id subnet-0a1b2c3d4e5f6g7h8 \
  --route-table-id rtb-0x1y2z3a4b5c6d7e8

Enable Transit Gateway Route Propagation

If using a Transit Gateway, enable route propagation on the route table:

aws ec2 describe-transit-gateway-route-tables \
  --query 'TransitGatewayRouteTables[0].TransitGatewayRouteTableId' \
  --output text

Enable propagation:

aws ec2 enable-transit-gateway-route-table-propagation \
  --transit-gateway-route-table-id tgw-rtb-0a1b2c3d4e5f6g7h8 \
  --transit-gateway-attachment-id tgw-attach-0a1b2c3d4e5f6g7h8

Replace Routes Pointing to Stopped Instances

If you have routes pointing to a NAT instance that’s stopped, replace them with a NAT Gateway:

aws ec2 allocate-address --domain vpc --output text

aws ec2 create-nat-gateway \
  --subnet-id subnet-0a1b2c3d4e5f6g7h8 \
  --allocation-id eipalloc-0a1b2c3d4e5f6g7h8

Replace the route:

aws ec2 replace-route \
  --route-table-id rtb-0a1b2c3d4e5f6g7h8 \
  --destination-cidr-block 0.0.0.0/0 \
  --nat-gateway-id nat-0a1b2c3d4e5f6g7h8

How to Run This

  1. Identify the route table associated with each subnet.
  2. Check for blackholed routes — delete them.
  3. Verify the longest prefix match rule — ensure more specific routes are intentional.
  4. For Transit Gateway, enable route propagation on each table.
  5. Replace NAT instance routes with NAT Gateway routes.
  6. Test asymmetric paths: ping in both directions.

Is This Safe?

Yes, these changes are safe. Route table updates apply immediately and don’t require instance restarts. Just ensure you understand the intended traffic paths before deleting routes.

Key Takeaway

Route table issues cause silent failures — traffic is dropped without errors. Always check route table associations, look for blackholed routes, and remember longest prefix match. For Transit Gateway, route propagation must be explicitly enabled. Test traffic in both directions to catch asymmetric routing early.

Have questions or ran into a different networking issue? Connect with me on LinkedIn or X.