One Ticking Second Until Your Network Crashes

Network CrashesRemember the Y2K panic? That was a bit of a pain to deal with, but in the end, everything was solved and all of the nuclear weapons in the world didn’t go off. Hooray!

However, in June 2015 a panic overcame IT departments as thousands of networks crashed in the blink of an eye. Overworked and stressed network and system admins arrived to work and found their end users unable to access their networks.

The problem boiled down to a single issue that keeps ticking on in IT departments.

That issue? The leap second.

Morty-ClockFor those uninitiated, a leap second is occasionally added to our Coordinated Universal Time (UTC), to keep our clocks in sync with tiny changes in Earth’s rotation speed. Now, don’t worry, the Earth isn’t coming to a screeching halt any time soon. According to The International Earth Rotation & Reference Systems Service (IERS), changes in Earth’s speed are nothing to worry about, at least not as far as “end-of-all-life” scenarios are concerned. However, astronomers (like all scientists) are in the business of being precise and therefore want to account for those minuscule differences in time, which is where leap seconds come in.

Leap seconds began to be added in 1972 and that year became its own “year zero”. When calculating leap seconds, IERS uses that year as the reference point. Since 1972, a total of 26 leap seconds have been added.

Generally, leap seconds are added on the last day of month in either June or December. Unlike leap days, leap seconds occur all at once, which can be quite tricky in certain places. The leap second is nestled in right at 11:59 p.m. Coordinated Universal Time (UTC), which is 6:59 p.m. Eastern Standard Time (EST). Therefore, on the East Coast the change might not be even be felt by most users. It falls just at the right time where IT guys have enough time at night to make sure all goes smoothly.

However, 11:59 p.m. UTC is 8:59 a.m. in Japan (Japan Standard Time) and is right at the beginning of their workday, giving IT people no time to fix this issue before their users get to work. If the leap second isn’t accounted for, their servers might be disrupted. Unlike adding leap years, which are added every four years, adding a leap second doesn’t follow an apparent schedule. Since 1972, there have been years where experts added two leap seconds in one year (1972) and then there were periods where zero leap seconds were added. For instance, there were zero leap seconds added between the years 2008 and 2012.

Infrequently adding leap seconds stems from the Earth’s temperamental and varying rotation speed. Climatic and geographical events greatly influence Earth’s rotation speed. For instance, scientists believe that the 2004 Indian Ocean earthquake shortened the rotation speed by 2.68 microseconds. Because of the unpredictability of Earth’s rotation, it’s decided merely six months in advance to add a new leap second.

The reason for that? The leap second is based on the discrepancy between an atomic clock and the Earth’s rotation. You see, it takes the Earth around 84,000 seconds to rotate to what we call one “day”. The way atomic clocks work is that they measure time from the time it takes a Cesium-133 atom to oscillate precisely 9,192,631,770 times — big words that mean it measures atom rotation, not the whole Earth.

This means that the nuclear clocks and regular clocks don’t always align perfectly, but most digital clocks try to keep up with atomic clocks anyway. In short, due to the aforementioned geological events, the difference between Earth’s rotation and atomic rotation averages about one second every 1.5 years.

Sick-Clock-2

While it seems that adding leap seconds shouldn’t be a big deal, as we saw in June, this can be catastrophic for some networks. This is because many computing systems use something called the “Network Time Protocol”, or NTP, to keep themselves in time with the world’s atomic clocks. When that extra second is sprinkled onto it, they just can’t compute, for lack of a better term, and crash.

Most famously, Linux and Java have the worst time with this extra second. This is what took down many sites during the last leap second, and with the addition of a leap second in 2012, Reddit, Huffington Post, and other major sites went down.

While you might not have had crashes on your own system, you may have seen it happen to those sites — that is because those sites likely run on the operating systems that seem to be more susceptible to the leap second crash, such as Linux and Java-based systems.

In 2015, however, these sites learned from past leap seconds and had their own solutions for the June 2015 jump. Google had its own system working to make sure that no users felt any sort of issue during the day.

Standard-Clock-1-e1452183203153

Because IT departments have to deal with leap seconds so infrequently, no one has created a standard system or program to implement leap seconds. Therefore there’s no real automated solution for adding this second to networks, which is what causes them to bite it. There are some pieces of software out there that can help fix your systems, but even those have been known to fail when the time actually comes for the second.

Without a standardized solution, each IT department creates a unique approach to dealing with the extra second. However, these different methods only add to the problem as it can take computers and networks out of sync with each other — sometimes, the different methods don’t interact well — and cause more crashes.

So…how do we fix this?

What if we told you that we should just get rid of the leap second? A proposed solution that’s been floating around suggests eliminating leap seconds once and for all. This would be specifically for computing reasons and uses, and it would provide accurate time as well as what networks around the world need in order to stay in sync: continuity.

The International Telecommunication Union, or ITU, meets every three to four years to review and change the Radio Regulations. They’re an entity of the United Nations, and they last met in 2012. The issue of the leap second was brought up (which is under their jurisdiction), and the concept of a continuous timescale was proposed. This timescale would basically just start from its own reference point, and continue going. There would be no turning the clock forward, and all operating systems would operate on this new timescale.

What happened then?

The delegates, who are representatives of different countries from around the world, voted. And they decided… to postpone the decision.

In fact, they decided to push the proposal of a solution until the conference in 2023. Yikes!

Why the wait? Well, the ITU is to conduct studies to “consider the feasibility of achieving a continuous reference time-scale, whether by the modification of UTC or some other method.”

Pipe-Bomb-2023

There’s over ten organizations working to gather this data and suitable solutions for this issue for the 2023 conference, where a solution will be proposed and, with any luck, accepted.

In the meantime, Sys Admins will have to just keep using their clever fixes, or external software, to help them with their leap second transition. Hopefully by 2023, we’ll have had enough time to figure out a better solution for this ticking time bomb of an issue.

Ed Clark| IT Advocate for LinkTek | LinkMail@LinkTek.com

Try Now!

Related Posts

SharePoint on premise vs online

Risks and Rewards: Migrating from SharePoint On-Premise to SharePoint Online

Mar 4, 2024

There’s no stopping the tidal wave of migration from SharePoint On-Premise to SharePoint Online. Here’s why, and what to expect

Read More
hardware upgrade hardware refresh

Hardware Upgrade? Hardware Refresh? A Guide to 5 Best Practices & 1 Major Misconception You Need to Know

Feb 23, 2024

The difference between a hardware upgrade and a hardware refresh — they’re not the same — plus 5 tips for best practices.

Read More
joseph popp

Malice, Money, Monkeys and a Madman: The Origin of Ransomware

Feb 19, 2024

IT Tips & Tricks Malice, Money, Monkeys and a Madman: The Origin of Ransomware Published 19 February 2024 Share on Facebook Share on Twitter Share…

Read More
box migration tool

Release of New Version of Link-Fixing Software Offers Support for Migrations To or From Box.com

Feb 25, 2022

Data migrations can be tricky. Migrating to or from Box is no different. Get some help.

Read More
Software Update

Latest Release of LinkFixer Advancedâ„¢ Streamlines Cloud Data Migrations

Jan 27, 2020

Migrating to the cloud? Link-fixing software helps prevent post-migration data-loss. Chat with us about options, including a free webinar.

Read More
Computer Server

Newest Release of Automatic Link Fixing Software Simplifies the Process of Protecting Links When Moving to SharePoint On-Premise or Online

Oct 2, 2018

Whether on-premises or online, a migration to SharePoint could result in data loss without the best tool for the job. Chat with us about your options.

Read More