Time drift on MacOS

[oojacoboo]

Version 17.06.2-ce-mac27 (19124)

Can confirm that there is time drift on the Docker client for Mac. I was able to confirm that restarting the Docker client corrects the time issue. Issue is present across all docker containers and docker run busybox date.

There are quite a few reports of this on previous versions. And it was supposedly fixed, but it is not.

The time being off here was only about a minute prior to restarting the Docker client. I cannot say how far it would have been off had the host been put to sleep. In fact, that may have triggered an update and corrected the issue.

It seems the VM clock emulation is just not accurate. Is the VM not using NTP? Can we not have a cron re-sync regularly, even without a Docker client interruption?

[djs55]

Time is currently set by 2 mechanisms:

• NTP via chronyd is supposed to keep things in sync with the host
• when the host wakes from sleep -- and the clock is too out-of-sync for chronyd to correct -- there's an explicit hwclock -s

I'm not sure what's allowing it to drift in your case. Perhaps the machine wakes, and runs hwclock -s but chronyd has already given up trying to sync the clocks?

When next in this state, could you log into the VM with:

docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /bin/sh

and then query the NTP state with chronyc e.g.

/ # chronyc activity
200 OK
1 sources online
0 sources offline
0 sources doing burst (return to online)
0 sources doing burst (return to offline)
0 sources with unknown address

/ # chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 192.168.65.1                  2   6   377    53    +71us[ +120us] +/-   32ms

[oojacoboo]

@djs55 sure - will do. Also, please note that my system never went to sleep. And, I know this to be the case because I just installed OS X 10.13 the day before. But, I also rarely put the system to sleep in general.

[jpz]

I'm also seeing this - output captured below:

% docker --version
Docker version 17.09.0-ce, build afdb6d4

% docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /bin/sh
/ # chronyc activity
200 OK
1 sources online
0 sources offline
0 sources doing burst (return to online)
0 sources doing burst (return to offline)
0 sources with unknown address
/ # chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? 192.168.65.1                  0   9     0     -     +0ns[   +0ns] +/-    0ns

27secs of drift:

%docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /bin/date ; date
Thu Oct  5 22:53:50 UTC 2017
Fri  6 Oct 2017 09:54:17 AEDT

[ddollar]

Seeing this as well

$ docker --version
Docker version 17.09.0-ce, build afdb6d4

$ date
Thu Oct 26 11:57:14 EDT 2017

$ docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /bin/date
Thu Oct 26 16:01:25 UTC 2017

$ docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /usr/bin/chronyc activity
200 OK
1 sources online
0 sources offline
0 sources doing burst (return to online)
0 sources doing burst (return to offline)
0 sources with unknown address

$ docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /usr/bin/chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? 192.168.65.1                  0   5     0     -     +0ns[   +0ns] +/-    0ns

[felixbuenemann]

I'm also seeing considerable drift of several minutes on macOS 10.13.1 (17B46a). My system is always on, I only set the screens to sleep when I'm away from the computer.

$ docker --version
Docker version 17.09.0-ce, build afdb6d4

$ uname -a
Darwin couchbook.local 17.2.0 Darwin Kernel Version 17.2.0: Sun Oct  1 00:46:50 PDT 2017; root:xnu-4570.20.62~10/RELEASE_X86_64 x86_64

$ last reboot | head -n1
reboot    ~                         Sat Oct 28 20:24

$ uptime
16:17  up 5 days, 20:53, 1 user, load averages: 2,92 3,54 3,87

$ date -u ; docker run busybox date
Fr  3 Nov 2017 15:20:43 UTC
Fri Nov  3 15:26:26 UTC 2017

$ date -u; docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /sbin/hwclock -r
Fr  3 Nov 2017 15:25:11 UTC
Fri Nov  3 15:25:12 2017  0.000000 seconds

$ docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /usr/bin/chronyc activity
200 OK
1 sources online
0 sources offline
0 sources doing burst (return to online)
0 sources doing burst (return to offline)
0 sources with unknown address

$ docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /usr/bin/chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? 192.168.65.1                  0  10     0     -     +0ns[   +0ns] +/-    0ns

I also noticed that the chrony drift file can't be written because of a missing directory:

$ docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /bin/sh

# cat /etc/chrony/chrony.conf
server 192.168.65.1 trust iburst minpoll 2
makestep 0.5 -1
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/chrony.drift
cmdport 0

# stat /var/lib/chrony/chrony.drift
stat: can't stat '/var/lib/chrony/chrony.drift': No such file or directory

# ls /var/lib
dhcpcd  docker  misc

This probably doesn't matter much, since it only affects initial clock drift correction after chronyd restarts.

One interesting thing to note though is that the hwclock in the VM is not drifting, but the system time is drifting from RTC. This indicates that the clock provider is doing a bad job.

Currently it is using the TSC time source, it might make sense to experiment with other providers like hpet or acpi_pm. There are various post on the web with TSC causing trouble on HyperV, os it is likely it affects other hypervisors as well.

$ docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /bin/sh

# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
tsc

# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
tsc hpet acpi_pm

I'll try experimenting with hpet first and see if that helps and try acpi_pm next.

The clocksource can be changed at runtime:

$ docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /bin/sh

# echo hpet > /sys/devices/system/clocksource/clocksource0/current_clocksource

# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
hpet

If your VMs system time currently drifts from the rtc, sync it back after switching the clocksource:

# hwclock --hctosys

I found lots of useful info under the following links:

• redhat: How to change the clock source in the system
• vmware: Timekeeping best practices for Linux guests (1006427)

[OscarBarrett]

Same here, with a drift of 16 minutes. Fixed by restarting Docker.

$ docker --version                                                                                                                                                               
Docker version 17.09.0-ce, build afdb6d4

$ docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /bin/sh
/ # chronyc activity
200 OK
1 sources online
0 sources offline
0 sources doing burst (return to online)
0 sources doing burst (return to offline)
0 sources with unknown address

/ # chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? 192.168.65.1                  0  10     0     -     +0ns[   +0ns] +/-    0ns

/ # date
Thu Nov 23 02:35:33 UTC 2017
/# exit

$ date                                                                                                                                                                           
Thu 23 Nov 2017 10:19:57 AWST

[felixbuenemann]

Btw. none of the available clocksources fixed the time drift problem for me.

[ggascoigne]

I'm seeing a pretty consistent drift of about 1-2ms per second. This is pretty consistent. Resetting the clock time just changes the time, it doesn't address the drift.

[adrienjoly]

Same problem on Version 17.09.1-ce-mac42 (21090), Channel: stable, 3176a6af01.

My PostgreSQL container's clock gains several seconds in the future every time I wake up my MacBook Pro from sleep. I love Docker, but having to restart it so often is really annoying...

Let me know if I can help in any way.

[buchs]

@adrienjoly - here is a tip: docker run --rm --privileged alpine hwclock -s will reset the time in the docker server. No need to stop and restart.

[adrienjoly]

@buchs Thanks for the tip! But isn't there a risk of altering data integrity of my PostgreSQL database running on that container, if I change the time without rebooting?

[kbuchsicf]

Hmm, interesting question. If you are timestamping entries, and you turn the docker server clock back, you might have issues. What else can you do in a database to cause problems? Constraints based on time? Maybe the log would turn out funny, if you needed to do a recovery. It all depends on what and how you use the database, I suppose.