Australian cloud for business software — built and run in Sydney since 2003

NetLeverage
NetLeverage

AWS VPC routing: finding the block with Reachability Analyzer and flow logs

Technical article
Cloud

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

Use Reachability Analyzer first, because it names the configuration item that stops a path. Then use VPC Flow Logs to confirm what happened on the network interfaces. The first tool reads configuration and sends no packets. The second records traffic but does not tell you which route was used or which rule refused it. Neither result proves the application works.

Commands below come from AWS references except where marked as written for this article. They were not run in a lab for this article. The addresses, IDs and the SQL Server flow are hypothetical.

Why it matters

A timeout in a VPC can come from at least four places: a route table, a network ACL, a security group, or the host itself. Each has different rules, and each is checked at a different point. Changing them in turn until traffic flows leaves rules behind that nobody can explain.

The two tools answer different questions, so the disagreement between them is useful.

Reachability Analyzer VPC Flow Logs
Evidence A model of the network configuration; it does not send packets or analyse the data plane IP traffic records per network interface
Answers Does configuration permit this path, and if not, what blocks it? Did traffic appear at this interface, and was it accepted or rejected?
Delay An analysis can take a few minutes Typically about 5 minutes to CloudWatch Logs and about 10 minutes to S3, on a best-effort basis
Cost $0.10 per analysis in the AWS pricing example Vended log ingestion and archival charges
Weak point Not traffic; feature gaps listed below Excluded traffic, aggregation, no route detail

How a packet is decided

Three separate mechanisms apply, and they fail differently.

Route tables choose the next hop. Each subnet is associated with one route table. If no route table is explicitly associated, the subnet uses the VPC's main route table. Forward and return traffic are routed by different tables when the two ends are in different subnets, so the return direction needs checking too.

Route selection starts with the longest prefix. When destinations are identical, AWS lists this order:

  1. Static routes, such as those to internet gateways and peering connections
  2. Prefix list routes
  3. Propagated routes
  4. Direct Connect BGP routes
  5. VPN static routes
  6. VPN BGP routes

A more specific route always wins, whatever kind it is. If several prefix lists overlap and point at different targets, AWS picks one at random and then keeps using it. That makes overlapping prefix lists a poor design for anything you need to predict.

A route's state is active or blackhole. A blackhole means the target is unavailable, for example a gateway that is no longer attached or a NAT instance that has been terminated.

Network ACLs filter at the subnet edge. Rules are evaluated from the lowest number up, and the first match decides. They are checked when traffic enters and leaves a subnet, not as it moves inside one. Two instances in the same subnet are therefore not filtered by the ACL. A packet matching no numbered rule is denied by the asterisk rule.

Network ACLs are stateless, so response traffic needs its own rule. The port it needs is the client's ephemeral port. AWS gives 32768-61000 for many Linux kernels, 49152-65535 for Windows Server 2008 and later, and 1024-65535 for a NAT gateway, Elastic Load Balancing and Lambda.

Security groups filter at the resource and are stateful. Responses to allowed traffic are allowed regardless of the rules for the opposite direction. Connection tracking has a consequence that catches people during a fix: changing a rule does not immediately interrupt tracked connections. They continue until they time out. Some TCP and UDP flows are not tracked: those permitted for all addresses by a rule, with a matching rule in the other direction that permits all response traffic on any port. They are cut immediately, unless the flow is part of an automatically tracked connection.

Step 1: write down the flow

Define the exact flow before opening either tool: source, destination, protocol and port. This hypothetical example is used throughout.

Item Value
Client App server, 10.20.1.15, eni-0aaa…, subnet A
Server SQL Server, 10.20.2.30, eni-0bbb…, subnet B
Flow TCP to port 1433
Symptom Connection attempts time out

Be careful with a test based on ping. Reachability Analyzer's --protocol accepts only TCP or UDP, so an ICMP result cannot be compared with it. Test the real port.

Step 2: read the route tables yourself

This is cheap and often settles the question. The next two commands were written for this article, using documented filters and output fields.

aws ec2 describe-route-tables \
  --filters Name=association.subnet-id,Values=subnet-0aaa… \
  --query 'RouteTables[].Routes[].[DestinationCidrBlock,DestinationPrefixListId,GatewayId,NatGatewayId,TransitGatewayId,VpcPeeringConnectionId,NetworkInterfaceId,InstanceId,LocalGatewayId,CoreNetworkArn,State,Origin]' \
  --output table

The query lists the common target fields, not every one the output can carry, and it leaves out IPv6 destinations. A row with no destination or target needs the full JSON. An empty result here means the subnet has no explicit association, so look at the VPC's main route table instead. Repeat for subnet B, because subnet B carries the return traffic.

To list every blackhole route in the VPC:

aws ec2 describe-route-tables \
  --filters Name=vpc-id,Values=vpc-0123… Name=route.state,Values=blackhole \
  --query 'RouteTables[].RouteTableId'

The filter matches route tables containing such a route, so read the routes in each table returned.

Step 3: run the analysis

Create a path, then start an analysis. This form is from the CLI reference, with this article's hypothetical IDs.

aws ec2 create-network-insights-path \
  --source eni-0aaa… \
  --destination eni-0bbb… \
  --destination-port 1433 \
  --protocol TCP

aws ec2 start-network-insights-analysis \
  --network-insights-path-id nip-0123…

If you use --filter-at-source you cannot also pass the source IP or destination port options. Put them in the filter instead. To analyse a destination given as an IP address, omit --destination and set DestinationAddress in the filter.

An analysis has a Status of running until it completes. This query was written for this article and uses documented output fields:

aws ec2 describe-network-insights-analyses \
  --network-insights-analysis-ids nia-0123… \
  --query 'NetworkInsightsAnalyses[0].{Status:Status,Found:NetworkPathFound,Codes:Explanations[].ExplanationCode}'

For scope, source and destination must be in the same Region, and in the same VPC or in VPCs connected by peering or a transit gateway. They can be in different accounts of the same AWS Organization, once cross-account analysis has been set up. A resource in another account must be given by ARN. The supported resource list has no on-premises host. The inference is that the analysis can test the path as far as a virtual private gateway or transit gateway, and says nothing about the network beyond it.

Where several paths exist, the tool shows the shortest. To test a specific one, use --filter-in-arns or --filter-out-arns to require or ignore a resource.

Step 4: read the explanation code

When NetworkPathFound is false, Explanations carries the codes. The full list is long. These are the ones a routing or filtering fault usually produces.

Code AWS meaning Where to look
NO_ROUTE_TO_DESTINATION The route table has no applicable route to the destination Route table named in the result
BAD_STATE_ROUTE The route is not in a functional state Blackhole route and its target
CANNOT_ROUTE The route's destination does not match the packet's destination address Prefix or CIDR in the route
MORE_SPECIFIC_ROUTE A more specific route matches instead The longer prefix, and whether it is intended
HIGHER_PRIORITY_ROUTE A higher-priority route has the same destination CIDR Static against propagated routes
VPC_LOCAL_ROUTE_CIDR_RESTRICTION Local routes apply only inside the VPC CIDR The destination address
SUBNET_ACL_RESTRICTION The subnet's network ACL must admit the traffic ACL rule numbers and direction
ENI_SG_RULES_MISMATCH The security group has no rule that applies Group named in the result
SG_HAS_NO_RULES The security group has no inbound or outbound rules Group named in the result
ENI_SOURCE_DEST_CHECK_RESTRICTION The interface rejects traffic not addressed to or from its own private IPs Appliance and routing instances

Two practical notes come with the result. AWS warns that additional components might also be blocking the path, so one fix may only reveal the next block. An explanation can also carry a Direction of ingress or egress, which narrows the rule set to read.

NO_PATH is different in kind. It is a configuration code meaning no path could be found at all. Common causes include optional details that were not met, a temporarily isolated component, a source that cannot initiate traffic, or an unsupported feature such as IPv6. Check the path definition before you change the network.

When NetworkPathFound is true, read ForwardPathComponents and ReturnPathComponents. Components can show the route, ACL rule or security group rule that matched. Confirm that the path uses the hop you intended.

An UNIDIRECTIONAL_PATH_ANALYSIS_ONLY detail code means only the forward path was analysed and the reverse path could still be blocked. Treat that as an unfinished result.

Step 5: confirm with flow logs

Create a flow log on both interfaces, or on the VPC. This form, written for this article from the documented options, captures accepted and rejected traffic on the two interfaces. Replace the bucket with your own.

aws ec2 create-flow-logs \
  --resource-type NetworkInterface \
  --resource-ids eni-0aaa… eni-0bbb… \
  --traffic-type ALL \
  --log-destination-type s3 \
  --log-destination arn:aws:s3:::example-bucket/flow-logs/ \
  --max-aggregation-interval 60 \
  --log-format '${version} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${tcp-flags} ${flow-direction} ${traffic-path} ${log-status}'

Set the format now, because a flow log's configuration and record format cannot be changed afterwards. You delete it and create another. On Nitro-based instances the interval is one minute or less regardless of the setting.

The default format omits tcp-flags, flow-direction and traffic-path. For the SYN and return-path reasoning below, a custom format is worth the effort. CloudWatch Logs Insights discovers a set list of flow log fields, and those three are not on it, so you would need the parse command to query them there.

Then produce a few real connection attempts and wait for delivery. Active status with no records is not yet a fault: AWS says it can take 10 minutes or more, and records appear only when traffic does.

The table below reads the records for the example flow. The first two rows follow AWS's published examples. The others are this article's inference from how the fields are defined.

Records for the flow What it points to Next check
Server ENI: request ACCEPT, response REJECT Outbound network ACL on the server's subnet. A stateful security group would have allowed the response Outbound rule for the client's ephemeral ports
Server ENI: single REJECT for the request Security group on the server, or the inbound ACL on its subnet Compare with the Reachability Analyzer code
Server ENI: request ACCEPT, tcp-flags 2, no response record with 18 Packets arrived and were allowed, but no SYN-ACK was recorded Host firewall, service listener, the port it binds to
Client ENI: outbound ACCEPT, server ENI: nothing Lost between the two interfaces, or the server's log is missing or delayed Routes and any device in between
Client ENI: nothing at all Traffic was not sent, or is of a type flow logs exclude Host resolution and routing, the exclusion list
SKIPDATA or NODATA in log-status Records were skipped, or no traffic in the interval Do not treat absence as proof

Treat ACCEPT carefully. The action field describes acceptance, with security groups, network ACLs and packets arriving after a connection closed given as examples of rejection. AWS does not document it as a delivery confirmation or a routing verdict. An ACCEPT on the client side does not prove the packet arrived.

REJECT does not identify the rule either. The reject-reason field only covers BPA and EC, meaning VPC Block Public Access and VPC encryption controls; other rejections show -. Late packets after a connection closes also count as REJECT, so a small number of them next to healthy traffic does not by itself indicate a fault.

tcp-flags values can be OR-ed within an interval, so 19 means SYN-ACK and FIN on the same line. traffic-path shows the type of egress used, such as 3 for a virtual private gateway. It does not name the route table or the specific route.

For flow logs in CloudWatch Logs using the default format, this Logs Insights query was written for this article, based on AWS's connection example.

fields @timestamp, srcAddr, dstAddr, srcPort, dstPort, protocol, action, bytes
| filter interfaceId = 'eni-0bbb…' and (dstPort = 1433 or srcPort = 1433)
| sort @timestamp desc
| limit 50

Step 6: fix, re-analyse and clean up

Make one change, then run the analysis again on the same path. Retest with real traffic, since the analysis alone says nothing about the application.

When finished, delete the analysis, then the path. Analyses are deleted automatically 120 days after creation. The default quotas are 1,000 paths, 10,000 analyses and 100 concurrent analyses, all adjustable.

aws ec2 delete-network-insights-analysis --network-insights-analysis-id nia-0123…
aws ec2 delete-network-insights-path --network-insights-path-id nip-0123…

Delete temporary flow logs too. Deleting a flow log stops new records but leaves existing data at the destination, so remove that separately when you are finished with it.

When the tools disagree

Reachable, but the connection fails. The configuration permits the path, so look at what the model cannot see. That includes the host firewall, the listener, and target health, which is not considered for load balancers. Connection-tracking exhaustion is another candidate: AWS says that packets are dropped once the instance's maximum is reached, and the conntrack_allowance_exceeded metric shows it.

Not reachable, but flow logs show ACCEPT. After a security group rule was removed, a tracked connection can continue until it times out. New connections are the fair test.

Network Firewall in the path. Stateful and stateless 5-tuple rules are supported, but domain lists, Suricata rules and some rule options are not. AWS's detail codes tell you to verify the result against the firewall's real behaviour when such a rule could have matched.

What to choose and why

These are editorial recommendations.

  • Read the route tables first when the fault is new and a recent change is known.
  • Run Reachability Analyzer before enabling flow logs. It is quick, and it names a component.
  • Enable flow logs when the analysis says reachable, when the fault is intermittent, or when you need evidence of what happened.
  • Log both ends. A record on one interface cannot tell you about the other.
  • Capture ALL, not only REJECT. The accept-then-reject pattern needs both.
  • Use temporary interface-level logs for a specific fault, and a separate, planned VPC-level log for routine monitoring, so diagnosis does not depend on what was already enabled.

Risks and limits

Warning: Network ACL changes apply to every associated subnet. Adding a blocking ACL rule breaks existing connections. If you add and delete rules at the same time and exceed the quota, the deletions happen and the new entries are not added. Make these changes in a planned window, with a way back that does not depend on the path you are changing.

Reachability Analyzer's documented limits include:

  • IPv4 only. A dual-stack resource is analysed on IPv4 alone.
  • Transit gateway Connect attachments and policy tables are not supported.
  • For TCP through a transit gateway route table, only forward traffic is analysed.
  • Gateway Load Balancer endpoint paths exclude the load balancer and its targets.
  • Packet headers can change along the path, for example through NAT.

Flow logs do not capture all traffic. Excluded types include:

  • Traffic to the Amazon DNS server (traffic to your own DNS server is logged)
  • Windows licence activation
  • Instance metadata at 169.254.169.254 and Amazon Time Sync at 169.254.169.123
  • DHCP and ARP
  • Traffic to the reserved address of the default VPC router

Security groups also do not filter Amazon DNS, DHCP, metadata and similar services. An outage in that traffic will not appear as a security group problem in either tool, so diagnose it from the host.

Source and destination fields can differ from the packet when traffic passes through a secondary IP or an intermediate layer. Add pkt-srcaddr and pkt-dstaddr when you need the original addresses. You cannot enable flow logs on a peered VPC unless it is in your account, and transit gateway flow logs are a separate feature.

Timestamps are approximate. The start and end fields can differ from the actual packet time by up to 60 seconds. Do not use them to order events finely across interfaces.

This article was not validated in a lab, and AWS changes these services. Check the linked pages before relying on a limit, code or price.

Sources

Opened 10 November 2026. Sources behind the corrections reopened 10 November 2026.