Networking and Content Delivery · Topic 6 of 6

Troubleshooting Network Connectivity

Find AWS connectivity faults step by step with VPC Flow Logs, Reachability Analyzer, security group and network ACL checks, DNS tests and ELB error codes.

6 min

In a nutshell

Check in order: DNS, routes, security groups, network ACLs, then the host and application.
Security groups are stateful and allow only. Network ACLs are stateless, ordered and can deny.
Flow logs record metadata, not payloads, with ACCEPT or REJECT per flow.

What it is

Most network faults in AWS come from a short list: a missing route, a blocked security group or network ACL, a DNS problem, or an unhealthy target. A good method checks them in order instead of guessing.

AWS also provides tools that look at the configuration or the traffic for you, so you do not have to read every setting by hand.

How it works

A checklist from the source to the target

Name resolution. Does the name resolve to the right address? Check Route 53 records, private hosted zone association and the VPC DNS settings.
Routing. Does the source subnet's route table send traffic toward the destination (IGW, NAT gateway, peering, transit gateway, VPN)? Is there a return route?
Security groups. The destination must allow the inbound port from the source. The source must allow outbound. Security groups are stateful.
Network ACLs. Both inbound and outbound rules must allow the flow, including the ephemeral port range for replies, since they are stateless.
Operating system and application. Is the service listening on the port? Does a host firewall block it?
Addressing. Does a public subnet instance have a public or Elastic IP? Is the destination CIDR overlapping another route?

VPC Flow Logs

VPC Flow Logs capture metadata about IP traffic for a VPC, subnet or network interface. They do not capture packet contents. They can be delivered to CloudWatch Logs, S3 or Firehose.

# AWS CLI: log all traffic for a VPC to CloudWatch Logs
aws ec2 create-flow-logs --resource-type VPC \
  --resource-ids vpc-0123456789abcdef0 --traffic-type ALL \
  --log-destination-type cloud-watch-logs \
  --log-group-name vpc-flow \
  --deliver-logs-permission-arn arn:aws:iam::111122223333:role/flowlogs

A default format record contains these fields in order.

version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status
2 111122223333 eni-0a1b2c3d 203.0.113.12 10.0.1.25 44321 22 6 10 840 1620000000 1620000060 REJECT OK

The protocol number 6 is TCP, and 17 is UDP. REJECT means a security group or network ACL denied the traffic, and ACCEPT means they allowed it.

A useful clue: if the inbound request shows ACCEPT but the reply shows REJECT, a network ACL is probably blocking the reply, because security groups are stateful and would allow it. Flow logs do not tell you which of the two rejected a flow, so combine them with configuration checks.

VPC Reachability Analyzer

Reachability Analyzer analyses your configuration between a source and a destination, such as two network interfaces, an instance and an internet gateway, or a transit gateway attachment. It does not send packets. It reports whether the path is reachable and, if not, names the component that blocks it, such as a route table or security group.

# AWS CLI: define and run a path analysis
aws ec2 create-network-insights-path \
  --source eni-0aaa1111 --destination eni-0bbb2222 \
  --protocol tcp --destination-port 443
aws ec2 start-network-insights-analysis \
  --network-insights-path-id nip-0123456789abcdef0

Network Access Analyzer looks for unintended network access paths, such as an internet reachable database, rather than a single connection.

Other tools

Tool	Use
Traffic Mirroring	Copies traffic from a network interface to an appliance for deep inspection
CloudWatch metrics	Instance network in and out, NAT gateway, load balancer and VPN tunnel state
CloudTrail	Shows who changed a route table or security group
AWS Config	Shows when a network setting changed
Session Manager and standard tools	Run `ping`, `traceroute`, `nslookup` or `curl` from inside an instance

CloudWatch network monitoring services

Three CloudWatch services watch network health continuously, and each publishes metrics you can alarm on.

Service	What it measures
Network Monitor	Probes along hybrid paths between VPC subnets and on premises destinations, reporting packet loss and round trip time
Internet Monitor	Availability and performance of internet traffic between your users' networks and your AWS application, with health events when a network or location degrades
Network Flow Monitor	Health of the traffic flows between your workloads, such as instances and other AWS resources, using agents on the hosts

Create a CloudWatch alarm on their metrics to be told early. Use them to decide whether a slowdown comes from your configuration, the hybrid link or the wider internet, and use flow logs and Reachability Analyzer for the configuration itself.

Load balancer errors

Code	Common meaning
502 Bad Gateway	The target closed the connection or sent an invalid response
503 Service Unavailable	No healthy targets are registered
504 Gateway Timeout	The target did not answer within the load balancer's idle timeout
403 or 4XX from ALB	Request blocked by a rule or WAF, or client error

If targets are unhealthy, check the health check path and port, and that the target security group allows traffic from the load balancer's security group.

Common scenario fixes

Private instance cannot reach the internet: missing route to the NAT gateway, NAT gateway in a private subnet, or a missing Elastic IP.
Cannot reach S3 privately: missing gateway endpoint route or endpoint policy blocks the bucket.
Peering works one way: a return route is missing in the other VPC.
VPN up but no traffic: route propagation off, or on premises firewall rules.
Name does not resolve: private zone not associated, or DNS hostnames off.

Exam tip

Flow logs show what happened to traffic, and Reachability Analyzer shows what the configuration allows. Use flow logs for live symptoms and the analyzer to find the blocking setting.

What to remember for the exam

Check in order: DNS, routes, security groups, network ACLs, then the host and application.
Security groups are stateful and allow only. Network ACLs are stateless, ordered and can deny.
Flow logs record metadata, not payloads, with ACCEPT or REJECT per flow.
ACCEPT inbound with REJECT on the reply suggests a network ACL.
Reachability Analyzer checks configuration and names the blocking component.
Traffic Mirroring copies packets for inspection.
502, 503 and 504 point to bad target responses, no healthy targets and timeouts.
A zonal NAT gateway must be in a public subnet, and private routes must point to it.
Peering and transit routes are needed on both sides.

Common mistakes

Changing many things at once. Alter one setting, test, and note the result.

Checking only the destination security group. The source outbound rules and the network ACLs also matter.

Forgetting ephemeral ports. Replies use high numbered ports that a network ACL must allow.

Reading flow logs as packet captures. They only show metadata.

Assuming the instance is the problem. The service may not listen on the port, or the health check path may be wrong.

Watch out

Flow logs are not real time, and records can appear after a delay. Use them to analyse a pattern, not to watch a connection second by second.

Check yourself

Answer these three questions.

Which tool tells you which component blocks a path without sending traffic?
VPC Reachability Analyzer.
A load balancer returns 503. What is the likely cause?
There are no healthy targets registered, so check target health and the health check settings.
Why can a network ACL cause one way failures when security groups look right?
Network ACLs are stateless, so a missing rule for return traffic blocks replies.

Topic 6 of 6 in Networking and Content Delivery

Test yourself on Networking and Content Delivery

The full AWS SOA-C03 question bank has 850 questions, with timed practice exams and an explanation for every answer.

Subscribe Try Sample Exam

This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.