cython / c++ meson upstream tracking issue

[ev-br]

This is to track the progress with upstream support of py3.extension_module(..., cython_args: '--cplus')

The issue description and links : #14 (comment)

@noDGodyaev do you know what is the expected meson version when this lands? And how would the build look like when it does? Maybe it'd be nice to have a PR now so that we can merge when the upstream is ready.

[ev-br]

Also need to remember to add meson to requirements.txt when it's a proper version, not git+https install.

[noDGodiaev]

Some kind of solution may appear from day to day.
Maybe not cython_args, but something else.

Full C++ in Cython support is expected in 0.60.
Some meson issues I mention here

[dcbaker]

I have a PR for this here: mesonbuild/meson#9017, if you wanted to test :)

[ev-br]

@dcbaker Tried testing your branch --- I am probably doing something wrong? --- with your meson branch, am building locally on linux this branch #42

(meson_cpp) br@gonzales:meson setup build --prefix=$PWD/installdir
The Meson build system
Version: 0.59.99
Source dir: /home/br/sweethome/proj/mc_lib
Build dir: /home/br/sweethome/proj/mc_lib/build
Build type: native build
Project name: mc_lib
Project version: 0.3
C++ compiler for the host machine: c++ (gcc 9.3.0 "c++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0")
C++ linker for the host machine: c++ ld.bfd 2.34
Cython compiler for the host machine: cython (cython 0.29.24)
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program cython found: YES (/home/br/.virtualenvs/meson_cpp/bin/cython)
Program python3 found: YES (/home/br/.virtualenvs/meson_cpp/bin/python)
Found pkg-config: /usr/bin/pkg-config (0.29.1)
C compiler for the host machine: cc (gcc 9.3.0 "cc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0")
C linker for the host machine: cc ld.bfd 2.34
Build targets in project: 4

Found ninja-1.10.2.git.kitware.jobserver-1 at /home/br/.virtualenvs/meson_cpp/bin/ninja

$ meson install -C build
ninja: Entering directory `/home/br/sweethome/proj/mc_lib/build'
[2/9] Compiling C object mc_lib/observable.../meson-generated_mc_lib_observable.pyx.c.o
FAILED: mc_lib/observable.cpython-38-x86_64-linux-gnu.so.p/meson-generated_mc_lib_observable.pyx.c.o 
cc -Imc_lib/observable.cpython-38-x86_64-linux-gnu.so.p -Imc_lib -I../mc_lib -I/home/br/.virtualenvs/meson_cpp/lib/python3.8/site-packages/numpy/core/include -I/usr/include/python3.8 -I/usr/include/x86_64-linux-gnu/python3.8 -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -w -g -fPIC -MD -MQ mc_lib/observable.cpython-38-x86_64-linux-gnu.so.p/meson-generated_mc_lib_observable.pyx.c.o -MF mc_lib/observable.cpython-38-x86_64-linux-gnu.so.p/meson-generated_mc_lib_observable.pyx.c.o.d -o mc_lib/observable.cpython-38-x86_64-linux-gnu.so.p/meson-generated_mc_lib_observable.pyx.c.o -c mc_lib/observable.cpython-38-x86_64-linux-gnu.so.p/mc_lib/observable.pyx.c
mc_lib/observable.cpython-38-x86_64-linux-gnu.so.p/mc_lib/observable.pyx.c:618:10: fatal error: ios: No such file or directory
  618 | #include "ios"
      |          ^~~~~
compilation terminated.
[4/9] Generating check_rndm_cpp with a custom command
ninja: build stopped: subcommand failed.
Could not rebuild /home/br/sweethome/proj/mc_lib/build

EDIT: in case it matters, observable.pyx wraps _observable/observable.h, which does #include<iomanip>.

[noDGodiaev]

Checked with dcbaker's PR.
@ev-br, we can use:

override_options : ['cython_language=cpp'],

In extension_module. Instead of cython_args
Locally tests passed)

[dcbaker]

If the whole project is cython -> c++ I'd put cython_language=cpp in your project(default options : )

[ev-br]

Thanks @dcbaker! For self-documenting purposes I'd stick with @noDGodyaev 's suggestion in gh-42, using override_options : ['cython_language=cpp'], per extension module.

And your mesonbuild/meson#9017 works great. Thanks for it, BTW.

The next issue is tracking the dependencies (can open a separate issue if needed, let me know) : with custom_targets, using depend_files: 'rndm.pxd', was working (cf gh-35). What would be an analog with your new way?

For context, cython has .pxd files which are sort of like .h files in C. In fact, there are .pxd files and .pxi files (the latter ones are literal includes).

To trigger the issue with dependencies: build the project, edit a syntax error into a pxd file, rerun e.g. $ meson install -C builddir, and

ninja: Entering directory `/home/br/sweethome/proj/mc_lib/builddir'
ninja: no work to do.

the syntax error is not picked up, i.e. the change in the pxd file is ignored.

[dcbaker]

ideally cython would just write a depfile for that, but we could probably add support for them the way we do with .h files, where adding them to the sources causes them to be listed as dependencies by ninja. I'd open a separate issue for that.

[ev-br]

@dcbaker done : mesonbuild/meson#9049 ; happy to provide further info or otherwise help however I can

[ev-br]

Playing a bit with Cython.Build module, it seems to me that the dependecy graphs can be extracted/constructed from Cython.Build.cythonize reasonably easily:
818f466

Not that I understand the depfile syntax though. Also not sure if cython devs are interested. And also, of course, how to make it useful for meson (assuming some from of this goes into cython.build).
@rgommers what would you think? (please let me know if I'm pinging you too much :-))

[rgommers]

Note that you want them from using cython as a command-line tool, which doesn't invoke cythonize (that's the higher-level interface which uses distutils). That said, if the dependencies are available that easily, this should not be difficult to do I guess.

Not that I understand the depfile syntax though. Also not sure if cython devs are interested.

Same here. I believe it's just Make syntax, but I'm not fluent in Make. If it's a smalll patch, should be doable to get it merged I'd think. Stefan has already commented a few times on the Cython-Meson interaction recently. And I'm getting more familiar with Meson, I should be able to help (also Meson maintainers are quite helpful).

please let me know if I'm pinging you too much

not at all, it's great to collaborate like this

[dcbaker]

@ev-br I just realized, if you don't have any C targets you probably want to change the default to c++, even if you're going to override on a per-target basis, otherwise meson will look for an unnecessary C compiler when it sees that you're using cython.

[dcbaker]

Depfile syntax is indeed a makefile snippet, it looks like this:

name/of/source.pyx: imported/source/1.pyx \
imported/source/2.pyx \
source.pyd \
...

indentation is significant in make.

[ev-br]

@dcbaker Thanks. So for me to specify the requirements before writing code for the cython devs to consider. For this to be usable to you:

• would it be acceptable to have it as a python call, not a cython exectutable. Something like $ python -c'from Cython.Build import cythonize_depfile; cythonize_depfile("source.pyx")'? Here "source.pyx" comes from meson.build.
• would it be enough to only list cython dependencies and leave the C headers and source files to the C/C++ compiler to handle? To be specific, if there is an observable.pyx which cimports observable.pxd and inputs foobar.pxi, these two files will be listed as dependencies only, and _observable/observable.h which is def extern from in the pxd file will be left out.
• In the example above, the depfile syntax is then a single file observable.pyx.d with a single line

observable.pyx : observable.pxd observable.pxi

where all paths are relative to the project root, is this correct?

P.S. Your comment on the global setting for the C++ compiler is noted, will specify it here. Thanks for this!

[dcbaker]

It would be best if the cython executable itself emitted the depfile, as that's what ninja needs. I'd rather not be wrapping up calls like python -c 'generated_depfile(infile, outfile); cython(...). Compilers do this with switches like gcc -o out.o -c -M out.c.d out.c

The only files that cython should be adding to the depfile are any files that it's using itself, so if changes to the C/C++ headers should trigger a fresh transpile from cython->C/C++ then they need to be in the depfile, if changes to the C/C++ headers only require that the C/C++ compiler regenerate a .o file then cython doesn't need to do anything with them, the C/C++ compiler will emit it's own depfile and ninja will do the right thing with that.

[eli-schwartz]

Hey, did you ever get around to plugging depfile support into cython? :)

[ev-br]

@eli-schwartz not yet, no. This is at the top of my to do list for next OSS time slot though. (Teaching term has started, so time's a bit of an issue)

[ev-br]

cython/cython#4563 which is now merged.