Tunnels · Topic 3 of 3
IP Tunnel Troubleshooting Tools
Troubleshoot Junos GRE and IP-IP tunnels with ping, traceroute, show commands, traceoptions and logs, and find MTU and recursive routing faults.
- 6 min
- Test the outer path first with
ping destination source local-address, then the inner path. ping size N do-not-fragmentfinds the effective MTU.- Traceroute across a tunnel shows the tunnel as one hop.
What it is
A tunnel fault is hard to see because the interface usually shows up. The tools on this page separate the problem into two layers: the outer path between the tunnel endpoints, and the inner path across the tunnel. A fault at either layer breaks the tunnel in a different way.
Work from the outside in. First prove that the two physical addresses can talk, then prove that the inner addresses can talk, then test a real packet size.
How it works
Test the outer path with ping
The outer path must work before the tunnel can. Ping the far tunnel destination from the local tunnel source address.
user@R1> ping 198.51.100.1 source 192.0.2.1 count 3
PING 198.51.100.1 (198.51.100.1): 56 data bytes
64 bytes from 198.51.100.1: icmp_seq=0 ttl=63 time=4.2 ms
If this fails, the problem is routing or a filter on the transport network, not the tunnel. If it works, check the tunnel configuration and the inner path.
Test the inner path
Ping the far end's inner tunnel address. This sends a packet through the tunnel.
user@R1> ping 10.0.0.2 count 3
Success proves that encapsulation, the transit path and decapsulation all work in the direction of the ping and for the reply. Failure with a good outer ping points to a configuration mismatch, a GRE key mismatch, a filter dropping IP protocol 47 or 4, or a missing return route.
Test the MTU
Large packets expose fragmentation problems that small pings hide. Send a larger ping with the Do not Fragment bit set.
user@R1> ping 10.0.0.2 size 1472 do-not-fragment count 3
user@R1> ping 10.0.0.2 size 1448 do-not-fragment count 3
On a 1500 byte link carrying GRE, the largest inner packet is 1476 bytes. A ping of size 1448 gives a 1476 byte packet (1448 data plus 8 ICMP plus 20 IP), and succeeds, while size 1472 gives 1500 bytes and fails. The point where the ping stops working is the effective tunnel MTU.
| Ping size result | Meaning |
|---|---|
| Small works, large fails with do-not-fragment | Tunnel MTU or physical MTU is too small for the packet plus overhead |
| Large works without the do-not-fragment option | Fragmentation is happening, which costs performance |
| All sizes fail | Not an MTU problem; return to the outer path and configuration |
traceroute
traceroute shows the path of a packet and the hops where it is lost.
user@R1> traceroute 10.2.0.1 no-resolve
traceroute to 10.2.0.1 (10.2.0.1), 30 hops max, 40 byte packets
1 10.0.0.2 2.1 ms 1.9 ms 1.8 ms
2 10.2.0.1 2.5 ms 2.3 ms 2.2 ms
The whole tunnel appears as a single hop. The routers in the transit network do not show up, because the TTL of the inner packet is reduced only once at each tunnel end. To see the transit path, trace the outer addresses: traceroute 198.51.100.1 source 192.0.2.1 no-resolve. If the trace goes into the tunnel when you trace the tunnel destination, you have found recursive routing.
Show commands
| Command | What it reveals |
|---|---|
show interfaces terse | Which tunnel interfaces exist and their admin and link state |
show interfaces gr-0/0/10.0 | Outer source, destination and protocol, MTU, and input and output counters |
show interfaces gr-0/0/10.0 extensive | Additional counters and error statistics |
show route 198.51.100.1 | How the tunnel destination is reached; it must not be through the tunnel |
show route 10.2.0.0/24 | Whether traffic for the remote network uses the tunnel |
show route forwarding-table destination 10.2.0.1 | What the forwarding plane will actually do |
show firewall | Counters of filters, if a filter is applied |
show ospf neighbor or show bgp summary | Protocol state across the tunnel |
Read the counters as a diagnosis. If output rises and input stays at zero, packets leave but nothing returns, so look at the far end and the path back. If both stay at zero, no traffic is routed into the tunnel: check the route.
Traceoptions
Tunnel interfaces have no protocol of their own, so tracing means tracing what runs over the tunnel. Trace the routing protocol that forms the adjacency across it.
user@R1# set protocols ospf traceoptions file ospf.log size 1m files 3
user@R1# set protocols ospf traceoptions flag hello detail
user@R1# set protocols ospf traceoptions flag error
user@R1# commit
user@R1> show log ospf.log
The file shows whether Hello packets leave and arrive on gr-0/0/10.0. If hellos are sent but never received, the tunnel path or the far end is faulty. To trace route changes on the router in general, set routing-options traceoptions file rt.log with flag route records route table activity.
Traceoptions use CPU and disk. Remove the traceoptions statement once the fault is found.
Logging
The system log records events such as protocol adjacency changes and interface state changes.
user@R1> show log messages | match "gr-0/0/10|OSPF"
user@R1> monitor start messages
monitor start messages follows the log live, and monitor stop ends it. Repeated adjacency up and down events every few minutes across a tunnel are a classic sign of recursive routing.
Common mistakes
Testing only with a default size ping. A 56 byte ping passes when a 1400 byte packet is dropped.
Skipping the outer test. Without it you cannot tell a transit fault from a tunnel fault.
Expecting traceroute to list transit routers. Inside the tunnel the path is a single hop.
What to remember for the exam
- Test the outer path first with
ping destination source local-address, then the inner path. ping size N do-not-fragmentfinds the effective MTU.- Traceroute across a tunnel shows the tunnel as one hop.
show interfaces gr-0/0/10.0shows outer header values and counters.- Output rising with no input points to the far end or the return path.
show routefor the tunnel destination must not use the tunnel.- Traceoptions apply to the protocol over the tunnel, such as OSPF, and the log is read with
show log. - Repeated adjacency flaps point to recursive routing.
Check yourself
Which test do you run before touching the tunnel configuration?
Ping the far tunnel destination from the local tunnel source to prove the outer path works.
A 56 byte ping through the tunnel works but a file transfer stalls. What do you test next?
Pings with a larger size and do-not-fragment, to find the effective MTU, then lower the tunnel interface MTU.
A routing protocol across the tunnel keeps dropping and reforming every few minutes. What is the likely cause?
Recursive routing, where the tunnel destination is learned through the tunnel itself.
Topic 3 of 3 in Tunnels
Test yourself on Tunnels
The full Juniper JNCIS-ENT JN0-352 question bank has 846 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Juniper. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.