pkgs/*/setup.cfg: Reduce boilerplate by m4-include, reduce complexity of m4 use #35867

mkoeppe · 2023-06-30T15:34:20Z

📚 Description

This makes the pkgs/*/setup.cfg.m4 and pkgs/*/pyproject.toml.m4 files less scary and less repetitive.

Split out from:

Modularization of sagelib: Distributions sagemath-{brial,combinat,eclib,flint,gap,giac,glpk,graphs,groups,homfly,lcalc,libbraiding,libecm,linbox,modules,mpmath,ntl,pari,plot,polyhedra,schemes,singular,standard-no-symbolics,symbolics} #35095

📝 Checklist

The title is concise, informative, and self-explanatory.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation accordingly.

⌛ Dependencies

…duce boilerplate

…to requirements.txt

…yproject.toml.m4

…yproject.toml.m4 (fixup)

github-actions · 2023-06-30T20:01:09Z

Documentation preview for this PR (built with commit 5ecf106; changes) is ready! 🎉

dimpase · 2023-07-01T08:50:53Z

This pytest-related error is everywhere in CI output, it seems

============================ test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.3.2, pluggy-1.0.0
rootdir: /sage
plugins: mock-3.11.1
collected 0 items / 1 error

==================================== ERRORS ====================================
________________________ ERROR collecting test session _________________________
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1014: in _gcd_import
    ???
<frozen importlib._bootstrap>:991: in _find_and_load
    ???
...

dimpase · 2023-07-01T08:51:32Z

Can it be fixed by dropping Python 3.8 ?

mkoeppe · 2023-07-01T16:06:59Z

This error is new, from the incremental build introduced in #35652. Unrelated to the present PR and to unrelated to the Python version. Tracked in:

build.yml: Incremental build&test fails with pytest/matplotlib error #35813

kiwifb · 2023-07-07T20:26:26Z

Not sure about less scary, but factorisation of the repeated bits is good. It is much more easy to understand I think. What makes the build/test fail?

mkoeppe · 2023-07-07T20:45:08Z

What makes the build/test fail?

An occurrence of #35813, unrelated

kiwifb · 2023-07-07T20:50:21Z

I understand what Dima was talking about now :) Looks good to me.

mkoeppe · 2023-07-07T21:32:26Z

Thanks! May I set to "positive review"?

tornaria · 2023-07-13T13:46:04Z

After this was merged I can't bootstrap sagelib on its own.

This works in 10.0.beta5:

$ cd build/pkgs/sagelib
$ PATH=../../bin:$PATH ./bootstrap

After bootstraping sagelib like that one can build sagemath-standard on its own and it works perfectly well (I've been doing it for the void linux package since 9.5).

Same thing in 10.0.beta6:

$ cd build/pkgs/sagelib
$ PATH=../../bin:$PATH ./bootstrap)
m4:src/setup.cfg.m4:1: cannot open `sage_spkg_versions.m4': No such file or directory
m4:src/setup.cfg.m4:9: cannot open `setup_cfg_metadata.m4': No such file or directory

Half of this is due to SAGE_ROOT missing so M4PATH is not correctly set. But still:

$ SAGE_ROOT=../../.. PATH=../../bin:$PATH ./bootstrap
m4:src/setup.cfg.m4:1: cannot open `sage_spkg_versions.m4': No such file or directory

Now the problem is that m4/sage_spkg_versions.m4 is not available and the only way to create it seems to be running $SAGE_ROOT/bootstrap which wants to bootstrap the whole sage-the-distro I don't need or want at all (for one, it needs autotools which is not otherwise needed to build sagelib, and worse it will take minutes vs. less than 1s).

Actually creating m4/sage_spkg_versions.m4 is done early and it's very quick. Can you provide a standard simple way to create it on its own? More importantly, could we actually support a standard way to bootstrap just sagelib / sagemath-standard and build it on its own?

dimpase · 2023-07-13T13:50:53Z

On Thu, Jul 13, 2023 at 2:46 PM Gonzalo Tornaría ***@***.***> wrote: After this was merged I can't bootstrap sagelib on its own. This works in 10.0.beta5: $ cd build/pkgs/sagelib $ PATH=../../bin:$PATH ./bootstrap After bootstraping sagelib like that one can build sagemath-standard on its own and it works perfectly well (I've been doing it for the void linux package since 9.5). Same thing in 10.0.beta6: $ cd build/pkgs/sagelib $ PATH=../../bin:$PATH ./bootstrap) m4:src/setup.cfg.m4:1: cannot open `sage_spkg_versions.m4': No such file or directory m4:src/setup.cfg.m4:9: cannot open `setup_cfg_metadata.m4': No such file or directory Half of this is due to SAGE_ROOT missing so M4PATH is not correctly set. But still: $ SAGE_ROOT=../../.. PATH=../../bin:$PATH ./bootstrap m4:src/setup.cfg.m4:1: cannot open `sage_spkg_versions.m4': No such file or directory Now the problem is that m4/sage_spkg_versions.m4 is not available and the only way to create it seems to be running $SAGE_ROOT/bootstrap which wants to bootstrap the whole sage-the-distro I don't need or want at all (for one, it needs autotools which is not otherwise needed to build sagelib, and worse it will take minutes vs. less than 1s). Actually creating m4/sage_spkg_versions.m4 is done early and it's very quick. Can you provide a standard simple way to create it on its own? More importantly, could we actually support a standard way to bootstrap just sagelib / sagemath-standard and build it on its own?

I don't think inventing workarounds to avoid using autotools is very useful way to spend our time :-(

…

— Reply to this email directly, view it on GitHub <#35867 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJXYHDR7SG3BOJUGCSKBLDXP732PANCNFSM6AAAAAAZ2ACTPA> . You are receiving this because your review was requested.Message ID: ***@***.***>

tornaria · 2023-07-13T14:08:27Z

I don't think inventing workarounds to avoid using autotools is very useful way to spend our time :-(

This has nothing to do with autotools, which is not used or needed to build sagemath-standard.

This PR broke the bootstrap script in build/pkgs/sagelib without offerning any alternative.

What is the point of modularization if we don't support the most basic split, namely at the boundary between sage the library and distribution packages.

dimpase · 2023-07-13T14:26:50Z

This PR broke the bootstrap script in build/pkgs/sagelib without offerning any alternative.

No alternative? Why, you run the main ./bootstrap, if you must, or use pre-built configure spkg.

I don't think modularisation at the moment means that parts of the source tree become independent on each other - it's still one tree.

mkoeppe · 2023-07-13T14:56:13Z

By design, the various source trees become self-contained after running bootstrap.

mkoeppe · 2023-07-13T14:58:23Z

Actually creating m4/sage_spkg_versions.m4 is done early and it's very quick. Can you provide a standard simple way to create it on its own?

Yes, I think some refactoring of the top-level bootstrap script will be useful. I'm also annoyed by bootstrap taking too long. I'll look into it.

tornaria · 2023-07-13T15:22:47Z

Actually creating m4/sage_spkg_versions.m4 is done early and it's very quick. Can you provide a standard simple way to create it on its own?

Yes, I think some refactoring of the top-level bootstrap script will be useful. I'm also annoyed by bootstrap taking too long. I'll look into it.

Since there is a Makefile maybe some of these bootstrap steps could be handled by that?

E.g. make bootstrap for which a step could be make bootstrap-sagelib using make m4/sage_spkg_versions.m4, etc.

mkoeppe · 2023-07-13T15:51:31Z

No, I definitely want to keep the bootstrap phase separate from the build system of the Sage distribution.

mkoeppe · 2023-07-13T16:17:05Z

@tornaria please take a look if #35950 goes in the right direction for you

kiwifb · 2023-07-13T20:23:39Z

Yes, I had the problem after the merge and I had myself to blame because I positively reviewed it. What really kills our old non-autotools way of boostraping sagemath-standard is that two generated .m4 files are missing without running the top bootstrap.

But, and I want to stress that out, this is a problem only if you build from the dev branch - mostly via cloning. For stable releases, I build from the tarballs on pypi which are already bootstrapped. While it is annoying and non-minimalistic (and I tried to extract a minimum process for a few hours but it always ended broken in strange way) I accepted that running autoconf for something I pull from git is acceptable. I would need to do it for any autotooled package I pull from git out there. Of course it is weird for what is supposed to be python package.

kiwifb · 2023-07-13T20:38:49Z

I also still do not run configure for people who wonder. While my attempt to generate only m4/sage_spkg_versions.m4 and m4/sage_spkg_versions_toml.m4 failed, I suspect that the way we build sagemath-standard in distro, providing empty files (or some minimal file with a bit of formatting) would be enough to restore our old way of bootstraping only build/pkg/sagelib.

Matthias Koeppe added 6 commits June 30, 2023 08:20

m4/setup_cfg_metadata.m4: New, m4-include it in setup.cfg files to re…

955c9f4

…duce boilerplate

bootstrap: Generate m4/sage_spkg_versions.m4, use in pkgs/*/setup.cfg.m4

d7d19e8

bootstrap: Flag bad uses of SPKG_INSTALL_REQUIRES_ macros; fall back …

66a053f

…to requirements.txt

bootstrap: Write m4/sage_spkg_versions_toml.m4 and use it in pkgs/*/p…

0dd1e09

…yproject.toml.m4

bootstrap: Update 'Installing...' messages

f17b341

bootstrap: Write m4/sage_spkg_versions_toml.m4 and use it in pkgs/*/p…

a788485

…yproject.toml.m4 (fixup)

mkoeppe added the c: distribution label Jun 30, 2023

mkoeppe self-assigned this Jun 30, 2023

Matthias Koeppe added 2 commits June 30, 2023 08:38

src/setup.cfg.m4: Back out unintended change

6adc33f

Remove file added by mistake

5ecf106

mkoeppe added the s: needs review label Jun 30, 2023

mkoeppe requested a review from dimpase June 30, 2023 20:30

mkoeppe requested a review from kiwifb July 7, 2023 17:43

kiwifb approved these changes Jul 7, 2023

View reviewed changes

kiwifb added s: positive review and removed s: needs review labels Jul 8, 2023

vbraun merged commit e3c11b6 into sagemath:develop Jul 9, 2023

mkoeppe added this to the sage-10.1 milestone Jul 9, 2023

mkoeppe removed the s: positive review label Jul 9, 2023

tornaria added the s: needs work label Jul 13, 2023

mkoeppe deleted the m4_simplify branch July 13, 2023 14:51

tornaria removed the s: needs work label Jul 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pkgs/*/setup.cfg: Reduce boilerplate by m4-include, reduce complexity of m4 use #35867

pkgs/*/setup.cfg: Reduce boilerplate by m4-include, reduce complexity of m4 use #35867

mkoeppe commented Jun 30, 2023 •

edited

Loading

github-actions bot commented Jun 30, 2023

dimpase commented Jul 1, 2023

dimpase commented Jul 1, 2023

mkoeppe commented Jul 1, 2023 •

edited

Loading

kiwifb commented Jul 7, 2023

mkoeppe commented Jul 7, 2023

kiwifb commented Jul 7, 2023

mkoeppe commented Jul 7, 2023

tornaria commented Jul 13, 2023

dimpase commented Jul 13, 2023 via email

tornaria commented Jul 13, 2023

dimpase commented Jul 13, 2023

mkoeppe commented Jul 13, 2023

mkoeppe commented Jul 13, 2023

tornaria commented Jul 13, 2023

mkoeppe commented Jul 13, 2023

mkoeppe commented Jul 13, 2023

kiwifb commented Jul 13, 2023

kiwifb commented Jul 13, 2023

pkgs/*/setup.cfg: Reduce boilerplate by m4-include, reduce complexity of m4 use #35867

pkgs/*/setup.cfg: Reduce boilerplate by m4-include, reduce complexity of m4 use #35867

Conversation

mkoeppe commented Jun 30, 2023 • edited Loading

📚 Description

📝 Checklist

⌛ Dependencies

github-actions bot commented Jun 30, 2023

dimpase commented Jul 1, 2023

dimpase commented Jul 1, 2023

mkoeppe commented Jul 1, 2023 • edited Loading

kiwifb commented Jul 7, 2023

mkoeppe commented Jul 7, 2023

kiwifb commented Jul 7, 2023

mkoeppe commented Jul 7, 2023

tornaria commented Jul 13, 2023

dimpase commented Jul 13, 2023 via email

tornaria commented Jul 13, 2023

dimpase commented Jul 13, 2023

mkoeppe commented Jul 13, 2023

mkoeppe commented Jul 13, 2023

tornaria commented Jul 13, 2023

mkoeppe commented Jul 13, 2023

mkoeppe commented Jul 13, 2023

kiwifb commented Jul 13, 2023

kiwifb commented Jul 13, 2023

mkoeppe commented Jun 30, 2023 •

edited

Loading

mkoeppe commented Jul 1, 2023 •

edited

Loading