Tunnels · Topic 3 of 3

IP Tunnel Troubleshooting Tools

Troubleshoot Junos GRE and IP-IP tunnels with ping, traceroute, show commands, traceoptions and logs, and find MTU and recursive routing faults.

  • 6 min
In a nutshell
  • Test the outer path first with ping destination source local-address, then the inner path.
  • ping size N do-not-fragment finds the effective MTU.
  • Traceroute across a tunnel shows the tunnel as one hop.

What it is

A tunnel fault is hard to see because the interface usually shows up. The tools on this page separate the problem into two layers: the outer path between the tunnel endpoints, and the inner path across the tunnel. A fault at either layer breaks the tunnel in a different way.

Work from the outside in. First prove that the two physical addresses can talk, then prove that the inner addresses can talk, then test a real packet size.

Outer pathsource to destinationworksTunnel configmirrored endsworksInner pathping across tunnelworksPacket sizeMTU testworksRouting and policy

How it works

Test the outer path with ping

The outer path must work before the tunnel can. Ping the far tunnel destination from the local tunnel source address.

user@R1> ping 198.51.100.1 source 192.0.2.1 count 3
PING 198.51.100.1 (198.51.100.1): 56 data bytes
64 bytes from 198.51.100.1: icmp_seq=0 ttl=63 time=4.2 ms

If this fails, the problem is routing or a filter on the transport network, not the tunnel. If it works, check the tunnel configuration and the inner path.

Test the inner path

Ping the far end's inner tunnel address. This sends a packet through the tunnel.

user@R1> ping 10.0.0.2 count 3

Success proves that encapsulation, the transit path and decapsulation all work in the direction of the ping and for the reply. Failure with a good outer ping points to a configuration mismatch, a GRE key mismatch, a filter dropping IP protocol 47 or 4, or a missing return route.

Test the MTU

Large packets expose fragmentation problems that small pings hide. Send a larger ping with the Do not Fragment bit set.

user@R1> ping 10.0.0.2 size 1472 do-not-fragment count 3
user@R1> ping 10.0.0.2 size 1448 do-not-fragment count 3

On a 1500 byte link carrying GRE, the largest inner packet is 1476 bytes. A ping of size 1448 gives a 1476 byte packet (1448 data plus 8 ICMP plus 20 IP), and succeeds, while size 1472 gives 1500 bytes and fails. The point where the ping stops working is the effective tunnel MTU.

Ping size resultMeaning
Small works, large fails with do-not-fragmentTunnel MTU or physical MTU is too small for the packet plus overhead
Large works without the do-not-fragment optionFragmentation is happening, which costs performance
All sizes failNot an MTU problem; return to the outer path and configuration

traceroute

traceroute shows the path of a packet and the hops where it is lost.

user@R1> traceroute 10.2.0.1 no-resolve
traceroute to 10.2.0.1 (10.2.0.1), 30 hops max, 40 byte packets
 1  10.0.0.2  2.1 ms  1.9 ms  1.8 ms
 2  10.2.0.1  2.5 ms  2.3 ms  2.2 ms

The whole tunnel appears as a single hop. The routers in the transit network do not show up, because the TTL of the inner packet is reduced only once at each tunnel end. To see the transit path, trace the outer addresses: traceroute 198.51.100.1 source 192.0.2.1 no-resolve. If the trace goes into the tunnel when you trace the tunnel destination, you have found recursive routing.

Show commands

CommandWhat it reveals
show interfaces terseWhich tunnel interfaces exist and their admin and link state
show interfaces gr-0/0/10.0Outer source, destination and protocol, MTU, and input and output counters
show interfaces gr-0/0/10.0 extensiveAdditional counters and error statistics
show route 198.51.100.1How the tunnel destination is reached; it must not be through the tunnel
show route 10.2.0.0/24Whether traffic for the remote network uses the tunnel
show route forwarding-table destination 10.2.0.1What the forwarding plane will actually do
show firewallCounters of filters, if a filter is applied
show ospf neighbor or show bgp summaryProtocol state across the tunnel

Read the counters as a diagnosis. If output rises and input stays at zero, packets leave but nothing returns, so look at the far end and the path back. If both stay at zero, no traffic is routed into the tunnel: check the route.

Traceoptions

Tunnel interfaces have no protocol of their own, so tracing means tracing what runs over the tunnel. Trace the routing protocol that forms the adjacency across it.

user@R1# set protocols ospf traceoptions file ospf.log size 1m files 3
user@R1# set protocols ospf traceoptions flag hello detail
user@R1# set protocols ospf traceoptions flag error
user@R1# commit
user@R1> show log ospf.log

The file shows whether Hello packets leave and arrive on gr-0/0/10.0. If hellos are sent but never received, the tunnel path or the far end is faulty. To trace route changes on the router in general, set routing-options traceoptions file rt.log with flag route records route table activity.

Watch out

Traceoptions use CPU and disk. Remove the traceoptions statement once the fault is found.

Logging

The system log records events such as protocol adjacency changes and interface state changes.

user@R1> show log messages | match "gr-0/0/10|OSPF"
user@R1> monitor start messages

monitor start messages follows the log live, and monitor stop ends it. Repeated adjacency up and down events every few minutes across a tunnel are a classic sign of recursive routing.

Common mistakes

Testing only with a default size ping. A 56 byte ping passes when a 1400 byte packet is dropped.

Skipping the outer test. Without it you cannot tell a transit fault from a tunnel fault.

Expecting traceroute to list transit routers. Inside the tunnel the path is a single hop.

What to remember for the exam

  • Test the outer path first with ping destination source local-address, then the inner path.
  • ping size N do-not-fragment finds the effective MTU.
  • Traceroute across a tunnel shows the tunnel as one hop.
  • show interfaces gr-0/0/10.0 shows outer header values and counters.
  • Output rising with no input points to the far end or the return path.
  • show route for the tunnel destination must not use the tunnel.
  • Traceoptions apply to the protocol over the tunnel, such as OSPF, and the log is read with show log.
  • Repeated adjacency flaps point to recursive routing.

Check yourself

  • Which test do you run before touching the tunnel configuration?

    Ping the far tunnel destination from the local tunnel source to prove the outer path works.

  • A 56 byte ping through the tunnel works but a file transfer stalls. What do you test next?

    Pings with a larger size and do-not-fragment, to find the effective MTU, then lower the tunnel interface MTU.

  • A routing protocol across the tunnel keeps dropping and reforming every few minutes. What is the likely cause?

    Recursive routing, where the tunnel destination is learned through the tunnel itself.

Topic 3 of 3 in Tunnels

Test yourself on Tunnels

The full Juniper JNCIS-ENT JN0-352 question bank has 846 questions, with timed practice exams and an explanation for every answer.

This study guide is independent and is not affiliated with, endorsed by, or sponsored by Juniper. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.