Hats off to David Mills, NTP inventor and maintainer for close to 40 years, who passed away in January of this year.

I love that two of the universal constants of modern technology is a web address we can ping to get the current time, and a series of automated phone numbers we can call to do the same.

6:29 Ben went to college and wrote in the script… Description: written by Amy Now that’s hilarious

0:23 this is a video about bricks

RIP Dr. David L. Mills (1938-2024)

It's one of those things that just works great, but when it doesn't it manifests bizarrely. In the modern web it's often login issues Client: "Yes I'm logged in, here's my token valid from 09:00" Server: "... but it's only 08:57, buzz off"

I was a relatively new Linux users back in the early 90s. The kernel was at version 0.99, and NTP support was brand new. I was attending a local Unix Users Group, and the guy who'd be talking was just being introduced when it turned 7:00. There were maybe 30 people in the room and at least 25 of them had their digital watches (this was back when digital warches were still thought to be a neat idea) set to beep on the hour. And all of them had their watches synced to their Unix system, as did I. So there was one continuous beep as everyone's watch triggered. Not perfectly synchronized, they didnt all start at the same time, but the late beeps started before the early ceeps finished. So instead of multiple, closely spaced beeps, there was one continuous beep sweeping across the room, lasting perhaps 1.5 seconds.

NTP is great, but it only allows you to sync clocks within a few milliseconds, and that's not precise enough when you want to sync database replicas in a DB that uses timestamps, like Cassandra for example. It's _usually_ fine, but there are better alternatives now and those are used mostly in datacenters. The most well-known is probably Precision Time Protocol (PTP), which gets you to sub-microsecond accuracy. There's also a new system called chrony, which is an implementation of NTP that improves its precision to similar levels to PTP (~70 nanoseconds).

Minor correction, the building the NIST clock for NTP is in is actually on their Boulder campus, the Fort Collins clock ensemble operates as the source for radio time (WWVB) and as a backup clock. Many laboratories around the world synchronize their clocks with NIST using common view time transfer which acts like a calibrated GPS time signal.

After a long night at the bar, a guy invites his friend to see his new apartment. As they enter, the friend notices a large gong against the wall and asks, "What's with the gong?" The guy says, "Oh, that's not a gong, that's my talking clock". He picks up the mallet and hits the gong. From the other side of the wall they both hear, "Shut up! It's 3 in the goddamn morning!"

Just to be nitpicky: If you're talking about the power grid, a lot of equipment on the grid needs even higher precision. That's why a lot of that equipment is directly attached to a GPS clock and PTP / IEEE1588 is used to propagate even more precise timing information.

Lol Sam complaining about Ben at the end

For people who dont know founder of ntp, David Mills, died in january of this year at the age of 85. RIP

Wait till this man hears about PTP (Precision Time Protocol)

Time go thru wire, OR time go thru HF radio waves -- WWV is an amazing thing, the ability for devices to get the time via radio pretty much anywhere in North America is very important

I wouldn't really call NTP obscure, but then again I'm a network admin, so....

i cannot even put into words the emotions i'm feeling over that cardboard pc-case, the idea of bees coming out of a pc case LET ALONE that pulling the entire case away from it's cables X.X my heart hurts!

There's an old regularity rally saying: A man with one clock knows what time it is. A man with two clocks is never sure.

Perfect timing, I’m eating rn

The modern world is built on insane and incredibly passionate nerds.

NTP (especially combined with DHCP) is such a beautifully simple system and I wish more devices (like wall clocks, or IoT devices in general) supported it. Like I can buy a wall clock that's PoE, DHCP, and NTP - meaning it doesn't need a battery and simply plugging in a network cable will ensure it's always got the correct time. But it's $200. So many "atomic clocks" rely on the radio frequencies broadcast, which is great until you're until you're in a concrete building on the east coast meaning they never update and are just a regular clock.

The nintendo ds as shown in the video, does not use network time. They are manually set. They do not even adjust for daylight savings

The computer systems I work with require microsecond timing accuracy and we use GPS-disciplined NTP servers. Since they're referenced directly to an atomic clock they're Stratum 1. I can hear WWV on HF pretty well all the time from southern British Columbia. At night WWVB booms in on 60 kHz.

I like that this HAI has a higher proportion of actual footage shot by the team, versus stock footage. Keep it up folks!!

Chicago: Does anybody really know what time it is? HAI: well actually…

I love the attention to detail with the computer time, good job editors

Hi Sam! Thanks to Amy for building and operating her own time machine to get to the bottom of this problem. You should give her a raise!

lol I love how they put a motherboard in a cardboard box full of bees for a bit

There are a LOT more stratum 1 servers that this video makes you believe. All ntp server that get their time directly from a GPS receiver, will announce themselves as stratum 1. And a GPS receiver that can do that is < 100 USD (probobably as low as a few bucks but you get the idea). Basically anyone with a crappy PC and crappy GPS receiver can make a stratum 1 server. Stratum only cares about the hierarchy in NTP. There also companies like Meinberg that sell nice boxes that give better accuracy/reliability.

Finally a video i can show instead of explaining this to engineers - no your stratum 7 clock is not accurate :(

6:52 "You're paying too much for coffee" Joke's on you. I don't pay for coffee (I don't like coffee. Lol)

"Which you are, since that's the only country" and now I have coffee all over my keyboard... :)

I feel like knowing what the true time is, is the same feeling as morty feeling something perfectly flat for the first time.

My dad has a radio clock that adjusts its time based on the radio signal. It's cool to watch it set itself when you first put in the batteries. The hands zoom forward until the time is right.

Can't believe I watch all of this only to learn that time go through wire

There's also Precision Time Protocol (PTP), which is roughly half as old as NTP. It's only really suitable for local networks, but it's able to synchronize clocks to within less than a microsecond. In other words, three to four orders of magnitude better.

Very cool to see this video! For my work I'm implementing a client/server for NTP (called ntpd-rs) as well as a PTP implementation (called statime). The NTP protocol is actually a little more clever than what's being told in this video! NTP can combine the information from multiple time sources, it can do this because time information from one of these servers is generally still a little inaccurate (for a number of reasons). Using a filtering and combination process it can then pick the best source that is currently available. Modern implementations (such as ntpd-rs) even combine the information from multiple sources to get an even more accurate picture of the current time, allowing microsecond precision over the internet. That theoretically at least could make NTP more accurate than PTP (the precision time protocol) if given the same level of hardware support that PTP already has. There is one thing though that shows the age of NTP: it is completely insecure by default, anyone in between you and your time source can easily manipulate the messages being sent, allowing them to change your clock to any time they want. This could have large security implications for protocols such as HTTPS/TLS that secures all web traffic right now and requires knowing the current time to validate if the connection is secure. Let's not think about any high frequency trader or power grid that could possibly use NTP (or PTP) based time over an untrusted network. Luckily we're trying to make progress with NTS, Network Time Security, that allows securely transferring time information, but the protocol is horribly underused and servers are barely available. Hopefully someday we'll be able to make the internet a lot more secure! But I'm afraid an attack using NTP could still happen in the near future, with how important knowing accurate time has become.

Station WWV also broadcasts the time on a Ham radio frequency. When I was a kid I would sometimes tune my dad's radio to their channel because the rhythmic ticking and calming announcer voice helped me get to sleep.

You can also use a GPS receiver as a VERY accurate clock because every GPS satellite contains multiple atomic clocks and is constantly broadcasting the time. My company uses a NTP server that is connected to a GPS antenna mounted outside the building.

At 3:40, the clock denoting the start of each minute precisely is 2 seconds off because it shows the time as 21:04:02 instead of 21:04:00

4:36 It's possible to build your own stratum 1 NTP server for like $100 - all you need is a GPS receiver and a cheap computer to plug it into. GPS time broadcasts are accurate to within 3ns.

Computers that receive their clock via GPS are considered Stratum-1 NTP servers. I have a Raspberry Pi with a GPS hat that is a stratum-1 NTP server.

I love that the motherboard in the box of bees has a potato attached.

6:00 To be really pedantic: t1 - t0 would be the difference between the clocks *plus the travel time*.(Specifically, it's server time minus client time plus travel time to server.) And t3-t2 is the opposite time difference, plus the return travel time. So, adding these up and the clock difference cancel out, and you get just the round trip travel time: (t1-t0)+(t3-t2). This can be rearranged to give the result you show at 6:20, (t3-t0)-(t2-t1) so this is the total time for the signal to travel between the computers in both directions.

I know some of the guys who work on clocks at NPL in the UK. HAI's description of them and what they do is absolutely bang on.

Oh boy! I cant wait for another video including a lot of numbers in it! Not a sarcasms, i honestly enjoy it.

Being 2 seconds away from midrolls and deciding to not add an extra 2 seconds, legend

i cant imagine these videos do as well as the others but i really like when you cover more technical topics like this

Anyone who works in a Stratham zero facility, gets to call themselves, a time lord.

I wish you'd gone one step further to complete the picture: Your computer gets time from multiple upstream servers, discards obvious outliers, then uses the remainders to calculate the drift of its own clock quite precisely. That drift is used to correct the time your Operating System gets from it's own hardware. So if your computer gets disconnected from the internet it'll still run pretty reliably because NTP modeled how badly your own hardware clock runs and knows how to correct it.

I grew up on the nation time signal in Canada. 11:00 am on the CBC, love that long beep.

Most of the electric utilities use GPS time, since a lot of the protective relays are not connected to the internet by design. You'll have a GPS clock(s) and it sends IRIG signals throughout the substation to all the devices (IEDs) that need to be synced.

The calculation is pretty straightforward, the only downside being that it has to assume the speed of transfer was exactly the same in both directions, but I'm sure the difference is typically negligible (especially since a temporary inconsistency would likely be fixed with the next update).

Woooow being called a nerd by HAI editors when looking into a QR code Easter egg hits different

If you want to be super precise about it, there are two issues here: duration of time and current civil time. Duration of time is defined by the SI unit of the second (sometimes in astronomy called the ephemeris second), which is directly defined in terms of the frequency of radiation used to tune atomic clocks (more specifically, cesium clocks). Then 60 seconds are a minute, 60 minutes are an hour, 24 hours are a day, and 7 days are a week. This is in principle a universal definition, and durations can be measured at a single spot in a given reference frame unambiguously. Current civil time is way more confusing. First of all, the idea of having a "current" time implies a universal present. This is achieved by defining an average reference frame of all points on the surface of the earth. This is realized in practice by using a weighted average of all stratum 0 clocks, with the weights determined by how many clocks are in that part of the world and by their reported precision. Some of these are moving faster than others, since rotation causes most rapid movement at the equator, but the average time gives a single coherent reference frame . . . almost. The exact surface is hard to define, so a gravitationally-defined "geoid" is used instead, and the true surface is projected onto that geoid. There is still the problem that some stratum 0 clocks are in orbit and some are on the ground, and the ones in orbit are further from the center of the earth and thus in a flatter region of spacetime. The full definition accounts for the curvature of spacetime due to the earth to define a real actual (somewhat arbitrary) universal frame of reference. Once we have the reference frame, we need to define what the time is in that frame. We do this by establishing an epoch: a particular event in spacetime which can be referenced after the fact. The epoch used here was a particular event at the start of January 1 1958. The idea is that every clock registered 00:00:00 at the same instant on that day, but that's not really the case; however, their offsets were recorded. Thus, we simply use our atomic clocks to count the number of cesium oscillations since then, as converted to the standard reference frame, and that gives the number of seconds since 0:00:00 1 Jan 1958. However, even _that_ is not good enough. This gives the international atomic time, but not the terrestrial time. The problem with this approach is that each day is not in reality exactly 86400 seconds long and each year is not exactly 365 or 366 days long. The astronomical day is defined by the rotation and orbit of the earth, physical phenomena which don't really happen at a constant rate. In particular, the fixed definition of the second was made before we had an accurate measurement of the mean solar day, so it is off by a few parts per million. As a result, atomic time drifts relative to mean solar time in the long term. To correct this drift, leap seconds are added or removed one at a time on June 30 or December 31 on short notice. The rules state that if atomic time drifts from mean solar time by more than 0.9 seconds by one of these dates, a leap second must be inserted or removed to keep them in synch. The result is that the mean solar time (defined by observing the sun) is always within 0.9 seconds of the atomic time. A consequence of adding a leap second is that the civil time can read 11:59:60, and a full civil minute may be either 59 or 61 seconds long. This doesn't affect the definitions for duration of time, which means it's possible for the duration of time that passes between 11:59:00 and 12:01:00 to be not exactly 2 minutes on those days. Moreover, the date drifts with respect to the seasons, because the year is not exactly 365 days long. This is corrected using the same calendar devised way back in 1582 for Pope Gregory. Whenever a year is a multiple of 400, or whenever it is a multiple of 4 but not of 100, then that year is 366 days long instead of 365. This keeps the dates nearly synchronized with the seasons on average over the 400-year cycle, but not quite. However, this difference won't be noticeable to most people for tens of thousands of years. Finally, we add the window dressing of time zones. Everything I have described so far is UTC, but the civil time in a given jurisdiction may be offset from UTC. That is decided by statute, so it changes frequently. Some regions are in different time zones depending on the season. But any given time zone can be identified by adding or subtracting a stated number of hours and minutes from UTC. Although this is the easiest to define physically, it is the hardest to track in practice, because every jurisdiction sets its own rules, changing them sometimes with little notice.

There are also a few different programs that you can download to keep your computer clocks more accurate than the factory one. Ham radio operators will use those special programs for some of the software they use for data communication modes because the software used needs to have the most accurate time possible.

My brothers and I for years set our watches to WWV in the 70s and 80s. I'm the 90s I ran Unix servers connected only by dialup, so I ran a script that would dial a NIST number once a week to correct server times in 4 different states. Nowadays it's so much easier, getting down really to which NTP daemon you prefer, with time sync also built in to server and workstation management architecture. Still, it's cool to tune in to WWV.

Just wanted to say these videos keep getting better and better. Great job Sam and team! ❤

Wow! The first video I have ever seen by HAI that I ACTUALLY knew what they were talking about before I watched the video! And it was pretty spot on! Keep up the good work!

I always wondered what happened to the extra time that didn’t fit exactly between 365 .25 rotations per revolution! Thanks for explaining how they fluctuate the length of dec 31, fascinating

Half as Interesting: “you’re paying too much for coffee” Me, who doesn’t drink coffee: “say what now?!?”

Years ago, the radio station I worked at would have to call WWV (the place in Colorado you were talking about) in order to connect to the network News and traffic reports. Bring back memories, I still remember the phone number

0:15 Wrong. For some reason my Laptop clock is going like 5 minutes wrong 🤣

“Does anybody know what time it is” would have been a PERFECT Chicago pun. 🎶

WWV (USA) and CHU (Canada) have been doing their time broadcasting for decades. They were on the air when I got my ham radio license back in the early 1960's. WWV is a time and frequency standard.

I love the reference to the Potato Machine used as Editing Computer - by replacing the CPU by a real Potato. I wonder if that was a slight hint to Sam from his staff

Another incredibly interesting system is the one the kept clocks in sync before we had computers in our homes. The clocks are even super simple an dumb ac motors with simple gearboxes to drive the clock hands. They two are connected through a distributed time system called ... the power grid. The power to our house contains a clock that "ticks" 60 times each second (at least in the USA) and though that is not perfectly accurate, the power system keeps track on how many "ticks" it has sent and updates the rate to make sure at the end of the day, it has distributed exactly 24hx60mx60sx60 "ticks" each day so all our clocks can be the same.

And for those that can afford it there’s PTP (Precision Time Protocol) which can get you a few tens of nanoseconds from a reference clock. EDIT: You can relatively easily make a Stratum 1 NTP server with a GPS receiver board and a Raspberry Pi mini computer. Cost is about 200-300 USD total for hardware (software is all free).

When I saw the brick thrown through the window I had a moment of hope that this talk of time was a bait & switch and we were getting what we've really been waiting for: a brick video.

A nice document is online: "A Brief History of NTP Time: Confessions of an Internet Timekeeper " (Mills as mentioned in the video). How it began and evolved. Very nerdy. I've done some programming related to it, mainly testing various NTP servers against GPS time and the 1 PPS output of the GPS received (manufacturer spec to 35ns or better). Some deep in the h/w programming needed on an OS-less system to get pretty sharp accuracy overall (better than 200ns). Suffice it to say that NTP is in the millisecond domain for most users whereas GPS gets us down into the sub µs realm pretty cheaply and sub 100ns realm with a bit more money ...

Thank you for mentioning leap seconds and IERS. Their newsletter is one email i look forward to twice a year!

My father was an Air Force Chaplain. Due to the tight timing of multiple Sunday morning services he felt it necessary to have his watch set to the correct time. This being the 1950s and Al Gore had no yet invented the internet my job early Sunday morning was to tune his shortwave radio to the National Bureau of Standards and listen to the time tone Then set his watch.

A few clarifications: An atomic clock doesn't make a timescale stratum 0. You need specific classes of atomic clicks, like hydrogen masers or ensembled cesium clocks for make a valid stratum 0 timescale. Similarly, atomic clocks are not necessary for stratum 1 devices. We make stratum 1 grand masters that use OCXOs for instance. Additionally the T1-T4 thing is not really ntp, strictly speaking, that's ptp

6:13 -- It looks like the first term, (t3-t0), is the total round-trip time of the query to the Stratum 3 (S3) server. The second term, (t2-t1), is the processing time of the Stratum 3 server. So the difference would be the round-trip query time, minus the S3 processing time, giving you 2*(one-way signal delay). Divide by 2 and you have the average signal delay between your computer and the S3 server. (The average is between the one-way outgoing and one-way incoming legs of the full round-trip query). Hope this helps!

0:24 wonderful transition

i mean my computer claims it has been 3:17pm for 5 hours now.

Fun fact NTP is not the most accurate generally useable time protocol. PTP (Precision Time Protocol) is. "Precision Time Protocol (PTP) is a protocol to synchronize clocks in a computer network, similar to Network Time Protocol (NTP). NTP is accurate, under ten milliseconds. PTP, however, is accurate up to less than a microsecond and is measured in nanoseconds"

So the conclusion (which is implied) is that NTP keeps computer clocks in sync by measuring the delay between systems and compensating periodically. Your phone's GPS hardware is measuring differences between the times broadcast by at least 4 satellites and along with the satellites' position data. It then computes the distance you are from each of the satellites, based on how long it took the time signal to get to you. Then it can basically compute the intersection of 4 spheres in 3D space. The GPS satellites have their own internal atomic clocks, and they are kept in sync by ground stations that use a similar method to adjust the clocks in each of the 24 satellites.

The gag starting at 0:57 is the hardest I think I've ever laughed at an HaI video

The GPS satellites have atomic clocks in them, so there are much more than 19 stratum 0 clocks. Most (if not all, didn't check all) NTP servers and clients keep a drift file to improve their accuracy between syncs, which makes the difference between the accurate time and the computer lower over time.

1:03 Haha, a potato PC for real 😂

Sam’s computer is literally a potato 🥔 filled with bees 🐝 💀

last time I was this early we were using sundials

As a Signals Technician... yeah, that's actually quite a good explanation. Well done!

Sam, one additional thing to consider about the importance of time that was a little overlooked is: GPS and the fact that the 21st Century trading economy literally relies on GPS providing accurate time for ships at sea. Precision Navigation and Time is actually a key strategic concern for most nations that is wildly underestimated by the non-mariner public.

A lot of really cool things being left out, i encourage everybody to read or watch videos on this subject, because there are so many problems solved here. For instance: if you are being told your clock is 2 seconds late, in many systems you can't simply jump back in time, you have to slow down the ticking of the system clock ,to prevent sequences of events being recorded to be in the wrong order.

2:46 My maid of honor’s maternal grandfather helped create the atomic clock, & he lived in the Denver metropolitan area, so the clock in Colorado he likely helped build.

I trust those “3 guys in a room who REALLY care about X” agencies more than I trust the any branch of the military (except the coast guard)

This is the best HAI this week!

I still have a lot of clocks in my house which must be manually set, enough where adjusting for Daylight Saving Time is always a chore.

As someone studying networking this was a really fun video to watch!

I watched this video at exactly 1:05 PM so when you showed the graphic of the time and said "That little clock in the bottom of your screen..." I looked at my clock and they were the same xD

Large organisation use theyr own Stratum 1. I worked on these a few years ago. Multiple Stratum 1 with gnss Receivers, and some even had an cesium clock donnected. The cesium/atomic clock does not give out any time, just a realy percice sync signal. the time (mostly miliseconds since 01.01.1970) is counted on the stratum 1 server! If you need a percise clock you can use a gnss disiplines stratum 1 server, the cheap ones are about 200$, or you can bild one for even less.

I'm a network engineer and your explanation was perfect

The different layers can be explained a little bit simpler. Zero is all of the references time standards. 1 is disciplined from a 0 source. 2 is disciplined from a 1 source. 3 is disciplined from a 2 source. So on and so on, Stratton 1 servers generally have better time keeping hardware. But realistically all they need is to be disciplined from GPS or the radio if you're talking about usa. Where this gets particularly tricky is you might be running an NTP time server on your router and referencing that for everything else on your network. That would push everything down one more if we were using the example from the video. There are reasons why you would want to do this. But the average person and probably small business doesn't need to care about this.

I love the cardbord box around the PC^^

"Since you can't really make your own stratum 1 server without direct access to an atomic clock" This is easier than you may realize. I've been running my own stratum 1 timeserver for years, using a Raspberry Pi and a GPS module that outputs a 1PPS discipline signal.

When testing my ham radio rig, I’ll up the ham radio and tune to the Fort Collins HF frequency of 15 Mhz and listen to the announcements. Easy way to test various functions.

I’d like to thank Ben for doing math. I’m a horology geek from Fort Collins who does Internet plumbing all day, but I just tell myself NTP is powered by ancient magick. I hope you win the next individual season of Jetlag.

00:19 That freaked me out because at the very moment I looked at my clock and it said 1am… and I had to stop and think whether there was some way that Sam could change the time in a video dynamically for each viewer 😂