Our network engineer sent out an email last week about a potential bug due to this year being a Leap Second Year. This wasn’t something I was aware of before so I did a bit of a search for not only the impact of the bug and what exactly a Leap Second is. As it turns out due to rotational variations of the planet the atomic clock can be out of sync. When this gets to 0.9 second the International Earth Rotation and Reference System (IERS) announces that a leap second will be added to the clock.

On midnight on June 30 this year the world atomic clock will have one second added to align the atomic clock to variances in the earths rotation. This is not the first occurance of this, there’s been 26 of these additional seconds added to the atomic clock since 1972. The last of these changes was in 2012. So what’s the big deal? Well, since the vast majority of computer systems use NTP to lock in their time settings the additional second will cause the same second to occur twice and this has the potential to cause some damage or downtime due to reboots. In 2012 some high profile companies such as Qantas, LinkedIn and Yelp suffered from outages as their equipment rebooted as it wasn’t able to handle the leap second. Cisco has worked to put both software/firmware updates or workarounds in place to help their customers resolve any potential impact. You can find more information about the Leap Second over on Cisco’s site.

As soon as I read the email I began to check out which systems are affected by this problem. The focus was obviously on the Cisco equipment within our Flexpod environment. This includes Nexus 7000, Nexus 5000, UCS Manager, UCS Fabric Interconnects, Cisco MDS switches and lastly Cisco UCM sitting on the infrastructure. I’ll go through each system, the symptoms, known affected systems and known firmware fixes. For more information on each component click on the header of the section and it’ll bring you directly to the Cisco bug search site.

Cisco Nexus 7000:

When the leap second update occurs a N7K SUP1 could have the kernel hit what is known a “livelock” condition under the following circumstances:

a. When the NTP server pushes the update to the N7K NTPd client, which in turn schedules the update to
the Kernel. This push should have happened 24 hours before June 30th, by most NTP servers.
b. When the NTP server actually updates the clock

Workaround:

On switches configured for NTP and running affected code, following workaround can be used.
1) Remove NTP/PTP configuration on the switch at least two days prior to June 30, 2015 Leap second event date.
2) Add NTP/PTP configuration back on the switch after the Leap second event date(July 1, 2015)

Known Affected Releases:

5.5(1)E2, 5.5(2), 6.0(4)

Known Fixed Releases:

5.2(6.16)S0, 5.2(7), 6.1(1)S28, 6.1(1.30)S0, 6.1(1.69), 6.2(0.217), 6.2(2)

Cisco Nexus 5000:

With this issue in the Kernel Nexus 50×0/55xx running affected NX-OS may hit what is known as a kernel “livelock” under the following conditions:

a. When the NTP server pushes the update to the N5K NTPd client, which in turn schedules the update
to the Kernel. This push should have happened 24 hours before June 30th, by most NTP servers.
b. When the NTP server actually updates the clock.

Workaround:

Note all NX-OS 6.x and 7.x versions has fix for this issue. In 5.x NX-OS 5.2(1)N1(9) will also have fix. Note: Nexus 5010/5020 CANNOT be upgraded to NX-OS 6.x and 7.x versions. On switches running affected code, following workaround can be used.

1)Remove NTP/PTP configuration on the switch at least two days prior to June 30, 2015 Leap second event date.
2)Add NTP/PTP configuration back on the switch after the Leap second event date(July 1, 2015)

Known Affected Releases:

5.1(3)N2(1c), 5.2(8)N1(1)

Known Fixed Releases:

5.2(1)N1(8.156), 5.2(1)N1(9), 6.0(2)N1(1)

 

Cisco UCS Manager & Fabric Interconnects:

When the leap second update occurs the system could crash causing a network outage.

Workaround:

1) Remove NTP configuration from the UCSM at least 25 hours before the scheduled leap second, so basically on or before June 29 22:59:59 PM UTC
Admin –>Time Zone Management –> NTP Servers: remove any present entries
2) Restore previous NTP configuration after the leap second event (2015 event is scheduled for June 30 23:59:59 UTC)

UCS 2.0 and 2.1 releases are not affected by this issue.

Known Affected Releases:

2.2(1b)A, 2.2(2c)A, 2.2(3a)A

Known Fixed Releases:

2.2(3e), 2.2(4b)

 

Cisco MDS 9148:

When the leap second update occurs the MMDS system clock does not update after an NTP leap second, but stays at the pre-leap second time.

Workaround:

When the leap second update is received the system clock can be nondisruptively updated by manually resyncing to the NTP peer with the ntp sync-retry CLI command.

Known Affected Releases:

6.2(11b)S4

Known Fixed Releases:

No release planned to fix this bug

 

Cisco UCM:

Customers running version 8 or older are susceptible to the leap second issue.

Known Affected Releases:

7.0, 7.1, 8.0(3)

Known Fixed Releases:

No planned releases

 

As well as review the sytems against the firmware versions we are running I have contacted our Cisco account team with details of our environment with a request to find out if any further upgrades need to be done in advance of the leap second. As yet I haven’t received a responds but I’ll add an addendum to this post if changes are required. To check on all Cisco systems to find out what is and isn’t affected by the Leap Second go to the Cisco Leap Second Product Information site. Hopefully June 30th goes well for everyone. I hope it does as I’m on holidays the next day 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *