Use meson in sage-the-distro

[tobiasdiez]

Replace the old setuptools-based build by the new meson-based one in sage-the-distro. Delete most of the old stuff that is no longer needed now.

Definitely shouldn't be merged in this release cycle. I'm happy to discuss the timeline and prerequisites of fully migrating from setuptools to meson.

📝 Checklist

• The title is concise and informative.
• The description explains in detail what this PR is about.
• I have linked a relevant issue or discussion.
• I have created tests covering the changes.
• I have updated the documentation and checked the documentation preview.

⌛ Dependencies

[tobiasdiez]

@dimpase @orlitzky Here is a first version of replacing the old setuptools-based build with meson in sage-the-distro. I'm not familiar enough with sage-the-distro to effectively finish this PR. Do you have the resources to take over from here?

[dimpase]

Sure, I'm ready to help as much as I can. Could you share an outline of how this is meant to work?

I imagine it is an adaptation to the Sage's venv (which is a more or less standard venv, no?) of what one can get e.g. with a standard venv on Gentoo Linux, or another environment where all the dependencies are available. Would this approach also work for a Conda-based environment?

[tobiasdiez]

As a first step, I would not change how sage-the-distro works. So leave all the dependency installation (python and non-python deps) untouched for now. (We can discuss later how to transfer the configure checks to meson, and perhaps even completely replace the install scripts by mesons wrap files)

So, basically the only required change would be pip install ./src to pip install . , i.e. instead of using the pyproject.toml in src, the meson-based one in the root should be used to install sagelib. I tried to that in build/pkgs/sagelib/spkg-install.in in this branch, but not sure if the changes are correct. Then see what errors popup and resolve those.

[dimpase]

running ./bootstrap gives

sed: can't read ../sagelib/package-version.txt: No such file or directory

[dimpase]

that's cause

$ git grep "sagelib/package-version.txt"
pkgs/sagemath-bliss/requirements.txt.m4:sagemath-standard==esyscmd(`printf $(sed "s/[.]p.*//;" ../sagelib/package-version.txt)')
pkgs/sagemath-coxeter3/requirements.txt.m4:sagemath-standard==esyscmd(`printf $(sed "s/[.]p.*//;" ../sagelib/package-version.txt)')
pkgs/sagemath-mcqd/requirements.txt.m4:sagemath-standard==esyscmd(`printf $(sed "s/[.]p.*//;" ../sagelib/package-version.txt)')
pkgs/sagemath-meataxe/requirements.txt.m4:sagemath-standard==esyscmd(`printf $(sed "s/[.]p.*//;" ../sagelib/package-version.txt)')
pkgs/sagemath-sirocco/requirements.txt.m4:sagemath-standard==esyscmd(`printf $(sed "s/[.]p.*//;" ../sagelib/package-version.txt)')
pkgs/sagemath-tdlib/requirements.txt.m4:sagemath-standard==esyscmd(`printf $(sed "s/[.]p.*//;" ../sagelib/package-version.txt)')

[tobiasdiez]

I think these requirements files can be safely deleted. At least I'm not aware of any usage of them.

[dimpase]

I think these requirements files can be safely deleted. At least I'm not aware of any usage of them.

these are for building all these sagemath-* pseudo-packages.

By the way, can you resolve the git conflict?

[tobiasdiez]

I think these requirements files can be safely deleted. At least I'm not aware of any usage of them.

these are for building all these sagemath-* pseudo-packages.

You are right, these are actually used.

Does ae33069 works for you? (I cannot test it atm on linux)

[dimpase]

Sage venv can only be characterised as a totally overdesigned steampunk.

that's from our basement here, in fact. The thing on the right leaked, and took 5 days and $12K to fix.

[dimpase]

The copying comes from the lines around SAGE_PYTHON_DISTUTILS_C_CONFTEST. For some reason the temporary conftest is a placed in a directory where src is present as well (probably via a symlink). As the standard directory for source files, src is then picked up and its content is copied over.

CC=gcc CXX=g++ -std=gnu++11 conftest_venv/bin/python3 conftest.py --verbose build --build-base=conftest.dir
No `packages` or `py_modules` configuration, performing automatic discovery.
`src-layout` detected -- analysing ./src

I tried to fix it (untested).

The cython error seems to come from a problem in

sage/src/sage/env.py

Lines 324 to 327 in 56f95c9

	import sage
	dirs = [os.path.dirname(directory)
	for directory in sage.__path__]
	try:

I would assume if you fix the failing test for this function, then the other cython tests should pass as well. I'm not totally sure but my guess would be adding the directory returned by importlib.util.find_spec("sage") to dirs there should work.

Doesn't seem to be any different from sage.__path__:

sage: importlib.util.find_spec("sage")
ModuleSpec(name='sage', loader=<_frozen_importlib_external.NamespaceLoader object at 0x7f5ddc461220>, submodule_search_locations=['/mnt/opt/Sage/sage-dev/local/var/lib/sage/venv-python3.12/lib/python3.12/site-packages/_sagemath_editable_loader.py/sage'])
sage: 
sage: sage.__path__
['/mnt/opt/Sage/sage-dev/local/var/lib/sage/venv-python3.12/lib/python3.12/site-packages/_sagemath_editable_loader.py/sage']

I'm attaching the new config.log, doesn't seem to make a difference.
config.log

[dimpase]

looks like my rebase didn't fly. I'll do a merge and post update

[dimpase]

This test fails - but actually I don't know its purpose:

File "src/sage/misc/cython.py", line 232, in sage.misc.cython.?
Failed example:
    cython('''
    from sage.misc cimport cachefunc
    ''')
Expected:
    Traceback (most recent call last):
    ...
    RuntimeError: Error compiling Cython file:
    ...
    ...: 'sage/misc.pxd' not found
Got:
    <BLANKLINE>

with the current fix - adding SAGE_SRC to the list of directories to search for Cython headers (see the change in src/sage/misc/cython.py) - it just works.

SAGE_SRC isn't quite right in some cases, e.g. for a setup where Sage is installed from a package manager.
Can @orlitzky @antonio-rojas @tornaria comment?

In particular, 5ee730d was added by @tornaria - for the purpose I don't understand

[dimpase]

targets sagelib-clean and fast-rebuild-clean in the top-level Makefile refer to build/pkgs/sagelib/src/ and should be fixed or removed.

[tobiasdiez]

The difference comes from the fact that cython doesn't fully work with PEP 420 implicit namespace packages (essentially the test is recording this fact). However, when we use meson to build sagelib, then __init__ files are created during build so that sage.misc is no longer an implicit namespace package. Hence the compilation in this example actually works.

Can we guard the test be the absence of sage/misc/__init__.py?

[tornaria]

Some quick comments:

• I think the test is reasonably explained there; since sage.misc is a PEP 420 "namespace package" (i.e. no __init__.py) using from <PACKAGE> cimport <MODULE> does not work (shortcoming of cython, it's not fixed in cython 3).
• Please do not add random directories to search paths. Sometimes it might seem to fix something, but in the end it often ends up biting back and causing pain.
• For similar reasons, I think sage.misc should either be a namespace package, or else be a regular namespace. Making different choices on different build systems seems a bad idea to me, since there are subtle differences in behavior (as we can see here).

[dimpase]

meson is meant to become the only system to build sagelib, as far as I am concerned. Transition is not straightforward, though.

[dimpase]

@tornaria - my question was: how do you set paths to sagelib Cython header files on Void, say. I imagine you keep a full source directory somewhere (to provide the Cython functionality and the docs), but where? How can it be customised?

I am talking about cythonising at Sage prompt, not the package building

[dimpase]

Can we guard the test be the absence of sage/misc/__init__.py?

I don't think this is easily doable in our current doctest framework. Unless, perhaps, pytest gets used.

[orlitzky]

Why keep this test at all?

"""
    In Cython 0.29.33 using `from PACKAGE cimport MODULE` is broken            
    when `PACKAGE` is a namespace package, see :issue:`35322`::                
                                                                               
        sage: cython('''                                                       
        ....: from sage.misc cimport cachefunc                                 
        ....: ''')                                                             
        Traceback (most recent call last):                                     
        ...                                                                    
        RuntimeError: Error compiling Cython file:                             
        ...                                                                    
        ...: 'sage/misc.pxd' not found
"""

If the test fails, then it's no longer broken: great! Meson solved the problem.

[orlitzky]

with the current fix - adding SAGE_SRC to the list of directories to search for Cython headers (see the change in src/sage/misc/cython.py) - it just works.

SAGE_SRC isn't quite right in some cases, e.g. for a setup where Sage is installed from a package manager. Can @orlitzky @antonio-rojas @tornaria comment?

I'm far from an expert on this. For the pxd files, things usually just work if they're installed next to the pyx files? For the *.h files, the last time I needed to do this, I copied Tobias's solution of importing the corresponding module and using __file__.

[dimpase]

with the current fix - adding SAGE_SRC to the list of directories to search for Cython headers (see the change in src/sage/misc/cython.py) - it just works.
SAGE_SRC isn't quite right in some cases, e.g. for a setup where Sage is installed from a package manager. Can @orlitzky @antonio-rojas @tornaria comment?

I'm far from an expert on this. For the pxd files, things usually just work if they're installed next to the pyx files? For the *.h files, the last time I needed to do this, I copied Tobias's solution of importing the corresponding module and using __file__.

Sage's .pxd files (aka Cython headers) are akin to the usual .h etc headers, in the situation where Sage is installed system-wide. If a user writes Cython code using Sage's system-wide headers, asking them to copy these over looks a bit unclean to me. So there ought to be a solution to this, and in such a case SAGE_SRC might not be appropriate to use.

[dimpase]

If the test fails, then it's no longer broken: great! Meson solved the problem.

OK, "plonk" it goes then. e1167ee

[orlitzky]

Sage's .pxd files (aka Cython headers) are akin to the usual .h etc headers, in the situation where Sage is installed system-wide. If a user writes Cython code using Sage's system-wide headers, asking them to copy these over looks a bit unclean to me. So there ought to be a solution to this, and in such a case SAGE_SRC might not be appropriate to use.

I think that there are two related problems here. The first is the pxd files, which are needed to cimport stuff from other cython programs, and the problem is that cython will look for the pxd file in the wrong place unless there is an __init__.py. For example,

sage: cython('''from sage.misc.cachefunc cimport cachefunc''')
...
_tmp_tmp7m7cufjo_tmp_cl1obc0v_pyx_0.pyx:1:0: 'sage/misc/cachefunc/cachefunc.pxd' not found

should be looking for sage/misc/cachefunc.pxd instead. If I touch sage/misc/__init__.py, this error goes away. So I think we are waiting for cython to fix this. In the meantime, should we install __init__.py? (Is anyone using the modularized distributions?)

The other issue is the *.h files. If you write an external cython module that builds against sage, you will find that the cythonized (i.e. C) code expects to find headers like sage/cpython/cython_metaclass.h, which won't be installed in the C compiler's include path. I don't really know how this is supposed to work, but Tobias's solution seems good enough if your external module has a build system. I do not know if it is possible to trick e.g. cythonize() into needing these headers.

[dimpase]

Is anyone using the modularized distributions?

in my book they are basically abandonware.

[orlitzky]

Is anyone using the modularized distributions?

in my book they are basically abandonware.

Well, meson generates the necessary __init__.py already. All we need to do is install them, probably by adding them to py.install_sources(...).

[tobiasdiez]

Is anyone using the modularized distributions?

in my book they are basically abandonware.

Well, meson generates the necessary __init__.py already. All we need to do is install them, probably by adding them to py.install_sources(...).

This is now #39139

[dimpase]

This needs more work on macOS, where different C++ interfaces need different options, e.g. linbox needs std=gnu++11
(due to std::binary_function used) and similarly with giac (using pointer_to_binary_function, also removed in c++17)

This probably explains why macOS CI fails here.