Cython dependency handling in .pxd and .pxi files

[ev-br]

Describe the bug

Cython has a notion of implementation source files (.pyx), declaration files (.pxd) and literal include files (.pxi). The distinction is similar to .c and .h files in C. The current syntax for tracking changes in declaration and include files is clumsy, especially for the cython->c++ transpile.

To Reproduce

In the current master branch, to specify a cython->c++ transile for a source with a .pyx and a .pxd file, depend_files argument to custom_target works (thanks to @rgommers for pointing this out). Here is an example where the implementation is split between observable.pxd and observable.pyx (https://github.com/ev-br/mc_lib/blob/master/mc_lib/meson.build):

# compile .cpp for the `observable` module
_observable_cpp = custom_target('observable_cpp',
  output : 'observable.cpp',
  input : 'observable.pyx',
  depend_files: 'observable.pxd',     # <---- this
  command : [cython, '--cplus', '-3', '--fast-fail', '@INPUT@', '-o', '@OUTPUT@']
)

# generate .so for observable
py3.extension_module(
  'observable', _observable_cpp,
  include_directories: inc_np,
  dependencies : [py3_dep],
  install: true,
  subdir: 'mc_lib'
)

With gh-9017, the syntax is greatly simplified:

# generate .so for the `observable` extension
py3.extension_module(
  'observable', 'observable.pyx',
  include_directories: inc_np,
  dependencies : py3_dep,
  override_options : ['cython_language=cpp'],   # <-----  no need for a custom_target, yay!
  install: true,
  subdir: 'mc_lib',
)

However, this does not track changes in observable.pxd. To reproduce the problem: compile once, introduce a syntax error into observable.pxd, try recompiling ( I did $ meson install -C build, but a direct ninja invocation would do) --- ninja does not see a syntax error:

ninja: Entering directory `/home/br/sweethome/proj/mc_lib/builddir'
ninja: no work to do.

Expected behavior

Expected behavior is that a change in a pxd or pxi file triggers a rebuild of a dependent extension module: similar to what happens for the .c / .h pairs. Either automatically, or via a simple meson.build syntax to declare a dependency.

system parameters

Native build on ubuntu focal, python 3.8.10,
$ meson --version
0.59.99
$ ninja --version
1.10.2.git.kitware.jobserver-1

[eli-schwartz]

cython/cython#1214

This was mentioned in the initial implementation of cython, and unfortunately meson has no way to guess what is needed without implementing a cython file parser. So we're blocked on upstream.

[ev-br]

Since that cython issue mentions cythonize.py, I cannot help mentioning the comment in a variant of that tool (https://github.com/scipy/scipy/blob/master/tools/cythonize.py#L21):

Simple script to invoke Cython ...; while waiting for a proper build system. Uses file hashes to figure out if rebuild is needed.

This comment dates back to at least 2012. We (at least some of us) are hoping that meson would be that proper build system we can use :-).

On a more serious note, two points:

• the OP cython example wraps a C++ header file, _observable/observable.h. Introducing a syntax error into the .h file, it's caught automatically somehow.
• a simple syntax to mirror depend_files for py3.extension_module would be a great solution.

[eli-schwartz]

I'm not so sure depend_files would actually work... it would apply to the extension module, not to the pyx file, so it would not trigger a cython rebuild of observable.pyx -> observable.cpp, it would just trigger a relink of all .o files.

As for the current behavior with observable.h

The syntax error in observable.h would be caught because C headers aren't exclusively used by cython, but also by GCC, and gcc emits depfiles and triggers rebuilds of observable.cpp -> observable.o when observable.h is modified.

Since observable.pyx does cdef extern from "_observable/observable.h" it should trigger a cython rebuild of observable.cpp, and since observable.cpp also does #include "_observable/observable.h" it should trigger a gcc rebuild of observable.o but...

Only the latter actually occurs. It may actually result in a build error even for valid changes to both .pyx and .h, because an old version of the .cpp file is compiled with the new version of the .h

Basically we need depfile support in cython, OR we need to add special syntax just for cython, which takes a dictionary of mappings rather than a simple list of files.

[rgommers]

Basically we need depfile support in cython

That's cython/cython#1459. Not much movement there, but the linked PR looks fairly straightforward (if incomplete, e.g. no handling of .pxi files).

I'm not so sure depend_files would actually work... it would apply to the extension module, not to the pyx file

Naive question: in the absence of Cython depfile support, why would it be so bad to apply the depend_files to both the cython->C and C->.so step? It may do a little too much work sometimes, but that's better than doing too little.

[eli-schwartz]

It is the wrong technical solution, and highly inconvenient to users because any time you update the implementation files with new dependencies you need to manually sync meson.build too -- which forces a full buildsystem reconfigure.

And according to that linked PR I guess it should be easier to add the correct technical solution to Cython?

[rgommers]

And according to that linked PR I guess it should be easier to add the correct technical solution to Cython?

I hope so. One concern I have is that even if it gets implemented, it first goes in the Cython 3.0 branch (which is at least a year away) and the maintainer may not be willing to sync it back to 0.29.x if it's a nontrivial change. But agreed, worth trying this first.

[ev-br]

Depfile support has landed in the cython main branch.

[ev-br]

Depfile support has landed in the cython main branch.

Also 0.29.x branch now. (0.29.27, to be specific)

[rgommers]

Cython 0.29.33 got released 5 days ago, and it adds depfile support in the cython interface (cython/cython#4949). So looks like we're finally unblocked here.

[eli-schwartz]

Yup -- I have the commit already and someone pinged me about it today: eli-schwartz/meson@c5a764d