BUILD: Race condition between .pxi.in and .pyx compiles in parallel build of 2.1.0

[mgorny]

Installation check

• I have read the installation guide.

Platform

Linux-6.4.7-gentoo-dist-x86_64-AMD_Ryzen_5_3600_6-Core_Processor-with-glibc2.38

Installation Method

Built from source

pandas Version

2.1.0

Python Version

3.11.5

Installation Logs

Build log excerpt

gpep517 build-wheel --backend mesonpy --output-fd 3 --wheel-dir /tmp/portage/dev-python/pandas-2.1.0/work/pandas-2.1.0-python3_10/wheel --config-json {"builddir": "/tmp/portage/dev-python/pandas-2.1.0/work/pandas-2.1.0-python3_10", "setup-args": [], "compile-args": ["-v", "-j12", "-l0"]}
2023-08-31 07:02:26,275 gpep517 INFO Building wheel via backend mesonpy
+ meson setup /tmp/portage/dev-python/pandas-2.1.0/work/pandas-2.1.0 /tmp/portage/dev-python/pandas-2.1.0/work/pandas-2.1.0-python3_10 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --vsenv --native-file=/tmp/portage/dev-python/pandas-2.1.0/work/pandas-2.1.0-python3_10/meson-python-native-file.ini
The Meson build system
Version: 1.2.1
Source dir: /tmp/portage/dev-python/pandas-2.1.0/work/pandas-2.1.0
Build dir: /tmp/portage/dev-python/pandas-2.1.0/work/pandas-2.1.0-python3_10
Build type: native build
Project name: pandas
Project version: 2.1.0
C compiler for the host machine: x86_64-pc-linux-gnu-gcc (gcc 13.2.1 "x86_64-pc-linux-gnu-gcc (Gentoo 13.2.1_p20230826 p7) 13.2.1 20230826")
C linker for the host machine: x86_64-pc-linux-gnu-gcc ld.bfd 2.41
C++ compiler for the host machine: x86_64-pc-linux-gnu-g++ (gcc 13.2.1 "x86_64-pc-linux-gnu-g++ (Gentoo 13.2.1_p20230826 p7) 13.2.1 20230826")
C++ linker for the host machine: x86_64-pc-linux-gnu-g++ ld.bfd 2.41
Cython compiler for the host machine: cython (cython 0.29.36)
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python found: YES (/usr/bin/python3.10)
Found pkg-config: /usr/bin/pkg-config (1.8.1)
Run-time dependency python found: YES 3.10
Build targets in project: 53

pandas 2.1.0

  User defined options
    Native files: /tmp/portage/dev-python/pandas-2.1.0/work/pandas-2.1.0-python3_10/meson-python-native-file.ini
    buildtype   : release
    vsenv       : True
    b_ndebug    : if-release
    b_vscrt     : md

Found samurai-1.9 at /usr/bin/samu

Visual Studio environment is needed to run Ninja. It is recommended to use Meson wrapper:
/usr/lib/python-exec/python3.10/meson compile -C .

Generating targets:   0%|          | 0/53 eta ?
Generating targets:  98%|█████████▊| 52/53 eta 00:00
                                                    

Writing build.ninja:   0%|          | 0/225 eta ?
                                                 
+ /usr/bin/samu -v -j12 -l0
[…]
samu: job failed: cython -M --fast-fail -3 --include-dir /tmp/portage/dev-python/pandas-2.1.0/work/pandas-2.1.0-python3_10/pandas/_libs '-X always_allow_keywords=true' /tmp/portage/dev-python/pandas-2.1.0/work/pandas-2.1.0/pandas/_libs/interval.pyx -o pandas/_libs/interval.cpython-310-x86_64-linux-gnu.so.p/pandas/_libs/interval.pyx.c
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.binomial�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.bytes�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.chisquare�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.choice�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.dirichlet�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.exponential�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.f�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.gamma�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.geometric�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.pareto�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.gumbel�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.poisson�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.negative_binomial�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.normal�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.laplace�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.logistic�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.lognormal�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.logseries�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.power�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.ranf�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.randint�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.random�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.random_integers�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.random_sample�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.rayleigh�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.sample�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.standard_exponential�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.standard_gamma�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.standard_normal�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.uniform�[0m
�[33mWARNING �[0m �[34mOverriding pythran description with argspec information for: numpy.random.weibull�[0m

Error compiling Cython file:
------------------------------------------------------------
...
    bint kh_exist_strbox(kh_strbox_t*, khiter_t) nogil

    khuint_t kh_needed_n_buckets(khuint_t element_n) nogil


include "khash_for_primitive_helper.pxi"
^
------------------------------------------------------------

/tmp/portage/dev-python/pandas-2.1.0/work/pandas-2.1.0/pandas/_libs/khash.pxd:129:0: 'khash_for_primitive_helper.pxi' not found

Full build log: dev-python:pandas-2.1.0:20230831-050223.log

$ find /tmp/portage/dev-python/pandas-2.1.0/work/pandas-2.1.0-python3_10/ -name '*.pxi'
/tmp/portage/dev-python/pandas-2.1.0/work/pandas-2.1.0-python3_10/pandas/_libs/intervaltree.pxi
/tmp/portage/dev-python/pandas-2.1.0/work/pandas-2.1.0-python3_10/pandas/_libs/sparse_op_helper.pxi

It looks that meson files do not declare dependencies between khash_for_primitive_helper.pxi and khash.pxd files, so the former isn't necessarily created before the latter is attempt to be compiled.

[lithomas1]

Hi thanks for the bug report.

I think the issue here is that interval is not declaring that it is depending on the hashtable Cython code, I think I missed that.

Does this patch help?

diff --git a/pandas/_libs/meson.build b/pandas/_libs/meson.build
index f302c649bc..c0a9d1ad8e 100644
--- a/pandas/_libs/meson.build
+++ b/pandas/_libs/meson.build
@@ -69,7 +69,8 @@ libs_sources = {
     'index': {'sources': ['index.pyx', _index_class_helper]},
     'indexing': {'sources': ['indexing.pyx']},
     'internals': {'sources': ['internals.pyx']},
-    'interval': {'sources': ['interval.pyx', _intervaltree_helper]},
+    'interval': {'sources': ['interval.pyx', _intervaltree_helper],
+                 'deps': _khash_primitive_helper_dep},
     'join': {'sources': ['join.pyx', _khash_primitive_helper],
              'deps': _khash_primitive_helper_dep},
     'lib': {'sources': ['lib.pyx', 'src/parser/tokenizer.c']},

[mgorny]

Thanks. I think it solves that indeed. I've tried three times and I couldn't reproduce it with the patch applied.