[BUG] `user_counters_tabular_test` is failing

[senyai]

Hi!

I forked this repository and after one of the syncs one action failed on windows. I tried recreating it on Ubuntu 22.04 and was successful. I used debug build.

gdb -ex=r --args ./test/Debug/user_counters_tabular_test:

BM_CounterSet2_Tabular/threads:16                    9.18 ns         6.05 ns    121737072 Bat=30 Baz=40 Foo=10
/home/senyai/projects/extra/benchmark/test/output_test_helper.cc:118: CheckCase: Check `TC.match_rule != MR_Next' failed. Expected line "Benchmark                                               Time             CPU   Iterations UserCounters..." to match regex "^Benchmark %s Time %s CPU %s Iterations %s Bar %s Bat %s Baz %s Foo %s Frob %s Lob$"
    actual regex string "^Benchmark[ ]+Time[ ]+CPU[ ]+Iterations[ ]+Bar[ ]+Bat[ ]+Baz[ ]+Foo[ ]+Frob[ ]+Lob$"
    started matching near: Benchmark                                               Time             CPU   Iterations UserCounters...

Thread 1 "user_counters_t" received signal SIGABRT, Aborted.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140737352611648) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737352611648) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737352611648) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737352611648, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7842476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff78287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x0000555555575323 in benchmark::internal::CallAbortHandler () at /home/senyai/projects/extra/benchmark/test/../src/check.h:39
#6  0x000055555557547a in benchmark::internal::CheckHandler::~CheckHandler (this=0x7fffffffcf28, __in_chrg=<optimized out>)
    at /home/senyai/projects/extra/benchmark/test/../src/check.h:65
#7  0x000055555557af41 in internal::(anonymous namespace)::CheckCase (remaining_output=..., TC=..., 
    not_checks=std::vector of length 0, capacity 0) at /home/senyai/projects/extra/benchmark/test/output_test_helper.cc:118
#8  0x000055555557b26f in internal::(anonymous namespace)::CheckCases (checks=std::vector of length 41, capacity 41 = {...}, output=...)
    at /home/senyai/projects/extra/benchmark/test/output_test_helper.cc:139
#9  0x000055555557db5e in RunOutputTests (argc=1, argv=0x7fffffffdda8) at /home/senyai/projects/extra/benchmark/test/output_test_helper.cc:465
#10 0x0000555555565853 in main (argc=1, argv=0x7fffffffdda8) at /home/senyai/projects/extra/benchmark/test/user_counters_tabular_test.cc:565
(gdb) q

[senyai]

Oops, I this is a popular one. Mentioned in #1279 and #1665. Maybe it's time I have a look at it!

[LebedevRI]

IIRC the measured time for some previous benchmark happens to be 0, and the regex matches something else.

[dmah42]

this has been an ongoing issue with the windows benchmarks over years. i never got to the bottom of why it happens, whether it's a legitimate result (zomg windows so fast) or a bad timer implementation...

[LebedevRI]

If we want, we might be able to ditch windows runners altogether and do cross-compilation from linux through wine.
Though better yet, tests should not actually be using real timings, they should all use manual times. At least i would think so.

this has been an ongoing issue with the windows benchmarks over years. i never got to the bottom of why it happens, whether it's a legitimate result (zomg windows so fast) or a bad timer implementation...

:) It would most certainy be timing granularity, and rounding it down and not up.

[dmah42]

mocking the timer results sounds like a great idea. i'm not sure the project is well structured to allow it but maybe some judicious gmock usage could do the trick.

[senyai]

IIRC the measured time for some previous benchmark happens to be 0, and the regex matches something else.

All tests are run with --benchmark_min_time=0.01s. Isn't this supposed to address the issue? I can fix the issue by adding "inf" to the regex.