pthread_cond_timedwait: CLOCK_MONOTONIC hangs indefinitely

[tux21b]

I am really interested in the new pthread_cond_timedwait patches by @mliertzer since we have also already experience some issues with the previous libfaketime in combination with mysqld.

The compilation failed with a "possible uninitialized variable error" in libfaketime.c:2523, pthread_cond_timedwait_common() at int result;, therefore i changed the line to int result = 0;. Afterwards I was able to compile everything.

The following test case currently blocks indefinitely (when run from the test-suite) on Debian Jessie. That's the last output I see is:

$ LD_PRELOAD=../src/libfaketime.so.1 ./timetest
pthread_cond_timedwait: CLOCK_REALTIME test
(Intentionally sleeping 1 second...)
pthread_cond_timedwait: CLOCK_MONOTONIC test
(Intentionally sleeping 1 second...)

I have also tried executing the testcase directly. All programs where executed in a clean environment without any faketimerc files, faketime related enviornment variables or any global ld preload configuration. I have observed the following:

• ./timetest works
• LD_PRELOAD=../src/libfaketime.so.1 ./timetest blocks indefinitely
• LD_PRELOAD=../src/libfaketime.so.1 FAKETIME= ./timetest works
• LD_PRELOAD=../src/libfaketime.so.1 FAKETIME=-2 ./timetest blocks indefinitely
• LD_PRELOAD=../src/libfaketime.so.1 FAKETIME=+2 ./timetest blocks indefinitely

The cases that didn't work, were blocking at the CLOCK_MONOTONIC test like described above. Setting DONT_FAKE_MONOTONIC=1 solves the issue.

What's the difference between the 2nd and the 3rd case? Shouldn't they be equivalent without any other environment variables or faketimerc files?

[wolfcw]

You're right, it apparently hangs on Debian 8.x without DONT_FAKE_MONOTONIC. That needs investigation. On newer Debian releases, libfaketime does not have that problem.

As far as cases 2 and 3 are concerned, there in fact is a subtle difference between not setting FAKETIME and setting it to an empty string. It's actually more useful for testing and performance measurements, but not useful for real time-faking use. However, in this case, it's a trigger, but not a cause of the observed hang.

[mliertzer]

Most likely the culprit is the if condition in
https://github.com/wolfcw/libfaketime/blob/master/src/libfaketime.c#L2556

While implementing pthread_cond_timedwait support I noticed that glibc internally uses the faketime gettime functions for CLOCK_MONOTONIC for calculating how long to wait before timing out in pthread_cond_timedwait. Most likely earlier glibc versions use the kernel gettime syscall directly. Could you check what happens if you change the mentioned source line to if(false) such that the code in the else branch is always called?

However, if this fixes the problem, there is most likely no way to detect the correct behavior at runtime since faketime is loaded before libc. The best option would probably be another ifdef.

[tux21b]

Wow, many thanks for your reply and your amazing remote debugging skills! The fix you have suggested works fine on Debian Jessie and passes all test cases.

[lucasm-iRonin]

@mliertzer I've also encountered this issue when I ran Ruby script that calls sleep method. It basically sleeps forever.

Commenting out the following code to leave else branch (as you suggested above):

if(clk_id == CLOCK_MONOTONIC)
  timespecadd(&faketime, &tdiff_actual, &tp);
else

Solves the problem on Heroku. Is there a nice way to incorporate that so for example we can choose the correct behavior i.e. using the ENV variable or something similar?

[mliertzer]

The best option would probably be to have a configure script, which checks the behavior of the libc implementation at compiletime. Depending on the outcome of the check, the implementation for new or old glibc should then be used, which is guarded by an #ifdef block.

[lucasm-iRonin]

Thank you for suggestion.