Security Engineering · Topic 4 of 8
Network Troubleshooting Tools and Method
Use packet capture, tcpdump, port scans, DNS and TLS checks, flow data and logs in a layered method to find where a network security problem lives.
- 5 min
- Troubleshoot with a method: hypothesis, evidence, one change at a time, retest.
- Packet captures need a capture point, such as a mirror port or tap, on the path.
- Encrypted content cannot be read without keys or a decrypting proxy.
What it is
Troubleshooting a network security problem is evidence gathering. You form a hypothesis about where traffic is being blocked, altered or abused, then use a tool that can prove or disprove it.
The right tool depends on the question. Packet captures show what actually crossed a link, scans show what is exposed, and logs and flow records show history.
How it works
A layered method
Work from the bottom up or the top down, but be consistent. Confirm the link and addressing, then routing and filtering, then name resolution and certificates, then the application.
Change one thing at a time and record what you did. Compare against a known good baseline when you have one, since "normal" is the most useful reference.
Capturing traffic
A protocol analyzer decodes captured packets. Wireshark is a graphical analyzer, and tcpdump is a command line capture tool on Linux.
To see traffic, a capture point must be on the path. On switches, a mirror port (also called SPAN, Switched Port Analyzer) copies traffic to a monitoring port, and a network tap is a passive hardware device that copies traffic without loading the switch. Encrypted traffic shows headers and handshakes but not content, unless you hold the keys or inspect at a decrypting proxy.
# Linux shell: capture DNS traffic on interface eth0, no name resolution
sudo tcpdump -i eth0 -nn port 53
# Linux shell: save a capture to a file, then read it back
sudo tcpdump -i eth0 -nn -w capture.pcap host 10.0.0.5
tcpdump -nn -r capture.pcapLook for Transmission Control Protocol (TCP) handshakes that never finish (a synchronize, or SYN, packet with no reply suggests filtering or a dead host), resets from a device in the path, retransmissions, and unexpected destinations.
Seeing what is listening and reachable
| Question | Tool | Example |
|---|---|---|
| What ports is this host listening on? | ss or netstat | ss -tulpn (Linux shell), netstat -ano (Windows Command Prompt) |
| What is exposed from outside? | Port scanner such as nmap | nmap -sV -p 22,80,443 host |
| Where does the path break? | Traceroute | traceroute host (Linux), tracert host (Windows) |
| Is the Domain Name System (DNS) answering correctly? | dig or nslookup | dig example.com, nslookup example.com |
| Is the certificate valid? | OpenSSL client | openssl s_client -connect host:443 -servername host |
Scan only systems you are authorized to test, and be aware that active scanning can disturb fragile devices. For certificate checks, read the issuer, the validity dates, the name on the certificate, and whether the chain is complete.
Flow data, logs and alerts
Flow records, such as NetFlow or Internet Protocol Flow Information Export (IPFIX), summarize who talked to whom, over which ports, and how much. They are small, so they are kept for a long time, and they reveal scanning, exfiltration and unusual connections without the cost of full capture.
Firewall, proxy, DNS, authentication and intrusion detection logs show decisions and alerts. Correlate them by time, which makes synchronized clocks essential. A security information and event management (SIEM) platform brings the sources together.
Network behavior analysis
Baselines of normal traffic let you spot anomalies, such as a server contacting a new country, large uploads at night, or periodic beacons to one destination. Anomaly detection produces false positives, so confirm with other evidence before acting.
Matching symptom to tool
| Symptom | First tool |
|---|---|
| Connection times out | Packet capture at both ends, firewall logs |
| Name does not resolve | dig or nslookup against different resolvers |
| Transport Layer Security (TLS) handshake failure | OpenSSL client, check dates, names, chain, protocol versions |
| Slow or erratic path | Traceroute, flow data, device health |
| Suspected data theft | Flow data, proxy and DNS logs, then targeted capture |
| Unknown exposed service | Port scan, then host check with ss or netstat |
Safe testing
Use a lab or maintenance window for intrusive tests. Capture files may contain credentials or personal data, so store them securely, limit retention, and share them only with those who need them.
What to remember for the exam
- Troubleshoot with a method: hypothesis, evidence, one change at a time, retest.
- Packet captures need a capture point, such as a mirror port or tap, on the path.
- Encrypted content cannot be read without keys or a decrypting proxy.
- A SYN with no reply points to filtering or an unreachable host, and a reset means something actively refused or ended the connection.
- Flow data is a long retention, low cost view of conversations.
ss -tulpnandnetstat -anoshow listening ports and the owning process.openssl s_clientchecks the certificate chain and TLS negotiation.- Synchronized time is needed to correlate logs and captures.
For a question about proving what is crossing a link, packet capture wins. For a question about long term trends and who talked to whom, flow data wins.
Common mistakes
Scanning without permission. Even internal scans need authorization and a plan.
Capturing in the wrong place. If traffic does not pass the capture point, the capture proves nothing.
Trusting one data source. Combine captures, logs and flow records, since each has blind spots.
Leaving captures lying around. They may hold secrets, so protect and delete them.
Active scans and fuzzing against industrial or fragile devices can crash them, so use passive methods on those networks.
Check yourself
Work through these three questions.
Which tool copies traffic to a monitor without affecting the switch?
A network tap, which is a passive device.
What does a SYN packet with no reply suggest?
Traffic is being filtered or dropped, or the destination host or port is not reachable.
Why keep flow records?
They summarize conversations compactly, so they can be stored for long periods to find scanning, exfiltration and unusual connections.
Topic 4 of 8 in Security Engineering
Test yourself on Security Engineering
The full CompTIA SecurityX CAS-005 question bank has 654 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by CompTIA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.