Build fails due to race condition: vcs_version.h: No such file or directory during parallel ninja build

[vibhoothi]

When building libvmaf with Meson and Ninja, the build occasionally fails due to a parallel build race condition. It appears that test_feature_collector.c (and potentially other files) attempts to compile before vcs_version.h has been fully generated by the build system.

Environment

• OS: Ubuntu 24.04 (Noble Numbat)
• Compiler: GCC 13.3.0
• Build System: Meson 1.11.1 / Ninja 1.11.1
• VMAF Version: v3.1.0 (or master if applicable)

Steps to Reproduce

Run a standard parallel build using the following commands:

1. mkdir -p build && cd build
2. meson .. --buildtype release
3. ninja -C .

Note: Because this is a race condition, it may require a machine with a high core count or multiple attempts to trigger reliably. My server was 100 thread machine

Error Logs

The build fails randomly with the following compiler error:

[38/163] Compiling C object test/test_feature_collector.p/test_feature_collector.c.o
FAILED: test/test_feature_collector.p/test_feature_collector.c.o 
cc -Itest/test_feature_collector.p -Itest -I../test -Iinclude -I../include -I../src/feature -Isrc -I../src -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -std=c11 -O3 -D_GNU_SOURCE -MD -MQ test/test_feature_collector.p/test_feature_collector.c.o -MF test/test_feature_collector.p/test_feature_collector.c.o.d -o test/test_feature_collector.p/test_feature_collector.c.o -c ../test/test_feature_collector.c
In file included from ../test/test_feature_collector.c:21:
../src/libvmaf.c:43:10: fatal error: vcs_version.h: No such file or directory
   43 | #include "vcs_version.h"
      |          ^~~~~~~~~~~~~~~
compilation terminated.

Full Logs:

Details

[99/163] Compiling C object test/test_feature_extractor.p/.._src_predict.c.o
[100/163] Compiling C object test/test_predict.p/test_predict.c.o
[101/163] Compiling C object test/test_psnr.p/test_psnr.c.o
[102/163] Compiling C object test/test_dict.p/test_dict.c.o
[103/163] Compiling C object test/test_model.p/.._src_pdjson.c.o
[104/163] Compiling C object test/test_ciede.p/test_ciede.c.o
[105/163] Compiling C object test/test_predict.p/.._src_pdjson.c.o
[106/163] Compiling C object src/liblibvmaf_feature.a.p/feature_third_party_xiph_psnr_hvs.c.o
[107/163] Compiling C object src/libx86_avx2.a.p/feature_x86_cambi_avx2.c.o
[108/163] Compiling C object src/liblibvmaf_feature.a.p/feature_integer_vif.c.o
[109/163] Compiling C object src/liblibvmaf_feature.a.p/feature_integer_motion.c.o
[110/163] Compiling C object src/libx86_avx512.a.p/feature_x86_motion_avx512.c.o
[111/163] Compiling C object tools/vmaf.p/y4m_input.c.o
[112/163] Compiling C object src/libx86_avx2.a.p/feature_x86_motion_avx2.c.o
[113/163] Compiling C object src/liblibvmaf_feature.a.p/feature_cambi.c.o
[114/163] Compiling C object src/libx86_avx512.a.p/feature_x86_vif_avx512.c.o
[115/163] Compiling C object src/libx86_avx2.a.p/feature_x86_vif_avx2.c.o
[116/163] Compiling C object src/libx86_avx2.a.p/feature_common_convolution_avx.c.o
In file included from /usr/lib/gcc/x86_64-linux-gnu/13/include/immintrin.h:43,
                 from ../src/feature/common/convolution_avx.c:19:
In function ‘_mm256_mul_ps’,
    inlined from ‘convolution_f32_avx_s_1d_h_scanline’ at ../src/feature/common/convolution_avx.c:136:10:
/usr/lib/gcc/x86_64-linux-gnu/13/include/avxintrin.h:320:10: warning: ‘f4’ may be used uninitialized [-Wmaybe-uninitialized]
  320 |   return (__m256) ((__v8sf)__A * (__v8sf)__B);
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/feature/common/convolution_avx.c: In function ‘convolution_f32_avx_s_1d_h_scanline’:
../src/feature/common/convolution_avx.c:66:48: note: ‘f4’ was declared here
   66 |                         __m256 f0, f1, f2, f3, f4, f5, f6, f7, f8;
      |                                                ^~
In function ‘_mm256_mul_ps’,
    inlined from ‘convolution_f32_avx_s_1d_v_scanline’ at ../src/feature/common/convolution_avx.c:508:10,
    inlined from ‘convolution_f32_avx_s_1d’ at ../src/feature/common/convolution_avx.c:810:3:
/usr/lib/gcc/x86_64-linux-gnu/13/include/avxintrin.h:320:10: warning: ‘f4’ may be used uninitialized [-Wmaybe-uninitialized]
  320 |   return (__m256) ((__v8sf)__A * (__v8sf)__B);
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/feature/common/convolution_avx.c: In function ‘convolution_f32_avx_s_1d’:
../src/feature/common/convolution_avx.c:438:48: note: ‘f4’ was declared here
  438 |                         __m256 f0, f1, f2, f3, f4, f5, f6, f7, f8;
      |                                                ^~
In function ‘_mm256_mul_ps’,
    inlined from ‘convolution_f32_avx_s_1d_v_sq_scanline’ at ../src/feature/common/convolution_avx.c:1224:10,
    inlined from ‘convolution_f32_avx_s_1d_sq’ at ../src/feature/common/convolution_avx.c:1561:3:
/usr/lib/gcc/x86_64-linux-gnu/13/include/avxintrin.h:320:10: warning: ‘f4’ may be used uninitialized [-Wmaybe-uninitialized]
  320 |   return (__m256) ((__v8sf)__A * (__v8sf)__B);
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/feature/common/convolution_avx.c: In function ‘convolution_f32_avx_s_1d_sq’:
../src/feature/common/convolution_avx.c:1149:48: note: ‘f4’ was declared here
 1149 |                         __m256 f0, f1, f2, f3, f4, f5, f6, f7, f8;
      |                                                ^~
In function ‘_mm256_mul_ps’,
    inlined from ‘convolution_f32_avx_s_1d_v_xy_scanline’ at ../src/feature/common/convolution_avx.c:2012:10,
    inlined from ‘convolution_f32_avx_s_1d_xy’ at ../src/feature/common/convolution_avx.c:2389:3:
/usr/lib/gcc/x86_64-linux-gnu/13/include/avxintrin.h:320:10: warning: ‘f4’ may be used uninitialized [-Wmaybe-uninitialized]
  320 |   return (__m256) ((__v8sf)__A * (__v8sf)__B);
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/feature/common/convolution_avx.c: In function ‘convolution_f32_avx_s_1d_xy’:
../src/feature/common/convolution_avx.c:1932:48: note: ‘f4’ was declared here
 1932 |                         __m256 f0, f1, f2, f3, f4, f5, f6, f7, f8;
      |                                                ^~
[117/163] Compiling C object test/test_cambi.p/test_cambi.c.o
[118/163] Compiling C object src/liblibvmaf_feature.a.p/feature_integer_adm.c.o
[119/163] Compiling C++ object src/liblibsvm.a.p/svm.cpp.o
[120/163] Compiling C object src/libx86_avx512.a.p/feature_x86_adm_avx512.c.o
[121/163] Compiling C object src/libx86_avx2.a.p/feature_x86_adm_avx2.c.o
ninja: build stopped: subcommand failed.
---> Removed intermediate container a638ce7bb007

[kylophone]

I've never seen this happen before. I wonder if this is a meson bug?

[jpujol]

happening to me too, v3.1.0

...
#15 5.773 [46/199] cc -Itest/test_feature_extractor.p -Itest -I../test -Iinclude -I../include -Isrc -I../src -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -std=c11 -O3 -D_GNU_SOURCE -pthread -MD -MQ test/test_feature_extractor.p/test.c.o -MF test/test_feature_extractor.p/test.c.o.d -o test/test_feature_extractor.p/test.c.o -c ../test/test.c
#15 5.774 [47/199] rm -f src/liblibvmaf_cpu.a && gcc-ar csrDT src/liblibvmaf_cpu.a src/liblibvmaf_cpu.a.p/cpuid.obj src/liblibvmaf_cpu.a.p/cpu.c.o src/liblibvmaf_cpu.a.p/x86_cpu.c.o
#15 5.774 [48/199] cc -Isrc/liblibvmaf_feature.a.p -Isrc -I../src -Iinclude -I../include -I../src/feature -I../src/feature/common -I../src/cuda -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -std=c11 -O3 -D_GNU_SOURCE -fPIC -MD -MQ src/liblibvmaf_feature.a.p/feature_ssim.c.o -MF src/liblibvmaf_feature.a.p/feature_ssim.c.o.d -o src/liblibvmaf_feature.a.p/feature_ssim.c.o -c ../src/feature/ssim.c
#15 5.774 [49/199] cc -Isrc/liblibvmaf_feature.a.p -Isrc -I../src -Iinclude -I../include -I../src/feature -I../src/feature/common -I../src/cuda -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -std=c11 -O3 -D_GNU_SOURCE -fPIC -MD -MQ src/liblibvmaf_feature.a.p/feature_iqa_convolve.c.o -MF src/liblibvmaf_feature.a.p/feature_iqa_convolve.c.o.d -o src/liblibvmaf_feature.a.p/feature_iqa_convolve.c.o -c ../src/feature/iqa/convolve.c
#15 5.774 [50/199] cc -Itest/test_feature_collector.p -Itest -I../test -Iinclude -I../include -I../src/feature -Isrc -I../src -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -std=c11 -O3 -D_GNU_SOURCE -MD -MQ test/test_feature_collector.p/test_feature_collector.c.o -MF test/test_feature_collector.p/test_feature_collector.c.o.d -o test/test_feature_collector.p/test_feature_collector.c.o -c ../test/test_feature_collector.c
#15 5.775 FAILED: test/test_feature_collector.p/test_feature_collector.c.o 
#15 5.775 cc -Itest/test_feature_collector.p -Itest -I../test -Iinclude -I../include -I../src/feature -Isrc -I../src -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -std=c11 -O3 -D_GNU_SOURCE -MD -MQ test/test_feature_collector.p/test_feature_collector.c.o -MF test/test_feature_collector.p/test_feature_collector.c.o.d -o test/test_feature_collector.p/test_feature_collector.c.o -c ../test/test_feature_collector.c
#15 5.775 In file included from ../test/test_feature_collector.c:21:
#15 5.775 ../src/libvmaf.c:43:10: fatal error: vcs_version.h: No such file or directory
#15 5.775    43 | #include "vcs_version.h"
#15 5.775       |          ^~~~~~~~~~~~~~~
#15 5.775 compilation terminated.

[eekay35]

I'm seeing the same thing on FreeBSD 15.1-RELEASE. Limiting the build to one job does indeed solve the problem. Adding the below to the ports/Poudriere make.conf is what I did.

.if ${.CURDIR:M*/multimedia/vmaf}
MAKE_JOBS_NUMBER=    1
.endif

[StormBytePP]

I've opened a PR that should address this issue: #1578

Unfortunately I don't have access to a machine with more than 32 cores, so I wasn't able to reproduce the race condition myself (I've never encountered it either).

The change is very small and should be completely innocuous. I tested it with Meson 1.10.0, as Meson 1.11.x currently has some bugs that prevent me from using it in production.

Please test with CUDA disabled (-Denable_cuda=false), as the CUDA-related compilation errors belong to a different issue/PR.

It would be great if someone who can reproduce the original problem could test the PR under similar high-parallelism conditions (preferably using ninja -jN directly). If it still fails with Meson 1.11.x, it would be very helpful if you could also try it with Meson 1.10.x — in that case the remaining issue would likely be on Meson’s side and should be reported upstream.

Thanks!

[eekay35]

I've opened a PR that should address this issue: #1578

This is excellent! Thank you!