High Availability · Topic 5 of 5
Unified ISSU and HA Troubleshooting Tools
Learn Junos unified ISSU requirements and steps, plus the traceoptions, show commands and logs used to troubleshoot LAG, VRRP, BFD, VC, GRES and NSR.
- 5 min
- ISSU upgrades two REs one after the other without taking the device out of service.
- ISSU needs GRES, NSR (and NSB on switches),
commit synchronizeand a supported release path. - Command:
request system software in-service-upgrade package reboot.
What it is
Unified In-Service Software Upgrade (ISSU) upgrades the Junos OS on a device with two Routing Engines (REs) without taking it out of service. Traffic keeps flowing and the routing sessions stay up while both REs are upgraded one after the other. A normal upgrade needs a reboot and an outage.
ISSU is the planned-maintenance partner of the switchover features. It uses a graceful switchover from one RE to the other during the upgrade, so it depends on the same building blocks.
How it works
Requirements
| Requirement | Reason |
|---|---|
| Two REs (or two RE members of a virtual chassis) | One RE stays in service while the other is upgraded |
| GRES enabled | Forwarding and interface state continue through the switchover |
| NSR enabled | Routing sessions continue through the switchover |
| NSB enabled on switches | Layer 2 protocol state continues |
commit synchronize | Both REs have the same configuration |
| Both REs running the same release before the start, and the new release supported for ISSU from it | The upgrade path must be on the supported list for the platform |
| The software package copied to the device | Usually under /var/tmp |
If a requirement is missing, the ISSU request is refused before anything changes. Hardware and software support differs per platform and release, so check the release notes before relying on ISSU.
user@router# set chassis redundancy graceful-switchover
user@router# set system commit synchronize
user@router# set routing-options nonstop-routing
user@switch# set protocols layer2-control nonstop-bridging
The upgrade steps
- The software package is copied to the master RE.
- The system checks that the master and backup are ready, in sync, and that the upgrade is supported.
- The backup RE is upgraded and rebooted into the new release.
- A switchover makes the upgraded RE master. NSR and GRES keep protocols and forwarding running.
- The former master RE, now the backup, is upgraded and rebooted into the new release.
- By default, the roles are left as they are after the switchover.
user@router> request system software in-service-upgrade /var/tmp/junos-install-package.tgz reboot
The reboot option reboots the upgraded REs automatically. The package name must be the one that was actually downloaded for your platform, so replace the example name.
| Before the upgrade | Check with |
|---|---|
| Backup RE is ready | show system switchover |
| NSR is synchronized | show task replication |
| Both REs run the same version | show version invoke-on all-routing-engines |
| Enough disk space | show system storage |
| ISSU progress | show chassis in-service-upgrade |
Do not start ISSU when show system switchover is not ready or when NSR has not finished replicating. The request may be refused, or traffic may be affected.
ISSU keeps the control plane alive, but some platforms still have a short packet loss while the forwarding hardware is upgraded. It is not a guarantee of zero loss, which is why the maintenance window still matters.
Troubleshooting tools across the chapter
The HA features share the same three kinds of tools: show commands, traceoptions and logs.
| Feature | Show commands | Traceoptions |
|---|---|---|
| LAG | show lacp interfaces, show interfaces ae0 terse | protocols lacp traceoptions |
| RTG | show redundant-trunk-group | Not used; read the log |
| Virtual chassis | show virtual-chassis, show virtual-chassis vc-port | Read the log |
| GRES and NSR | show system switchover, show task replication, show bgp replication | Protocol traceoptions |
| BFD | show bfd session extensive | protocols bfd traceoptions |
| VRRP | show vrrp detail, show vrrp track | protocols vrrp traceoptions |
| ISSU | show chassis in-service-upgrade | Read the log |
user@router# set protocols vrrp traceoptions file vrrp.log size 1m files 3
user@router# set protocols vrrp traceoptions flag all
user@router# commit
user@router> show log vrrp.log
The same pattern works for LACP and BFD. Replace vrrp with lacp or bfd in the hierarchy and the file name. The flag all is simple but heavy, so use it briefly and remove the traceoptions when finished.
Logging
The system log records state changes: VRRP master changes, BFD session changes, LACP member changes, RE switchovers and virtual chassis member events.
user@router> show log messages | match "VRRP|BFD|LACP"
user@router> monitor start messages
show log messages reads the file, and monitor start messages follows it live until monitor stop. To keep the messages that matter, check that syslog is configured, for example set system syslog file messages any notice.
A simple method
Read the log first, because it shows what happened first. A VRRP master change preceded by a link down, for example, points to the link and not to VRRP.
Look at the timestamps. In a failover, the earliest message is usually the cause, and the later ones are consequences.
Common mistakes
Starting ISSU without NSR. Protocol sessions reset during the switchover, or the request is refused.
Using a package for the wrong platform. The request fails.
Leaving traceoptions on. Files fill the disk and the CPU is burdened.
Troubleshooting from the wrong RE. Some commands show the local RE only.
Ignoring the log. The first event in the log often names the cause.
What to remember for the exam
- ISSU upgrades two REs one after the other without taking the device out of service.
- ISSU needs GRES, NSR (and NSB on switches),
commit synchronizeand a supported release path. - Command:
request system software in-service-upgrade package reboot. show chassis in-service-upgradeandshow system switchovercheck the state.- Traceoptions exist for LACP, BFD and VRRP under
protocols, and files are read withshow log. monitor start filenamefollows a log live.show lacp interfaces,show vrrp detail,show bfd session,show virtual-chassisandshow system switchoverare the main show commands.- Remove traceoptions after use.
Check yourself
Which features must be on before running unified ISSU?
GRES and NSR (and NSB on switches), plus
commit synchronize, on a platform with two REs.Which RE is upgraded first in an ISSU?
The backup RE, which then becomes master by switchover, and the former master is upgraded next.
How do you see why a VRRP master changed?
Read
show log messagesfor the first event, and enableprotocols vrrp traceoptionsif the log is not enough.
Topic 5 of 5 in High Availability
Test yourself on High Availability
The full Juniper JNCIS-ENT JN0-352 question bank has 846 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Juniper. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.