Avoid downloading ninja source

[FRidh]

I'm trying to package this package as it is a dependency of meson-python. ninja-python-distributions tries to download a ninja source from the web. How can we avoid that and let it use a local version that?

[eli-schwartz]

If you know that you have a ninja installed globally as a C++ command-line utility, maybe you can just package dummy metadata for it to satisfy meson-python?

Alternatively patch meson-python to not depend on ninja if you know it's already satisfiable...

[FRidh]

If you know that you have a ninja installed globally as a C++ command-line utility, maybe you can just package dummy metadata for it to satisfy meson-python?

I was hoping there was a way to generate the metadata or let this package use an existing build.

Alternatively patch meson-python to not depend on ninja if you know it's already satisfiable...

That's what I did now. I doubt there will be many other packages that will use it aside from meson-python.

[eli-schwartz]

The meson[ninja] extra does depend on this ninja package, but as it's only an extra that doesn't really matter. In theory projects could depend on that extra, but the real reason I merged that extra into meson's own metadata is because the PR author pointed out that pipx run meson[ninja] works (and you can't do pipx run meson ninja, of course).

pipx run isn't really relevant to distro packaging, obviously.

[henryiii]

FYI, see scikit-build/scikit-build#731 - A build backend can specify dynamic dependencies; scikit-build-core will probably do this do only add cmake and/or ninja if they are not present on the system and of a high enough version. meson-py could do the the same thing (but not meson, as it's part of the PEP 517 interface, and it will be important to add to [[extensions]] too, CC @ofek). FYI, this comes up because there are platforms where wheels can't be shipped, like the BSD's, Android, WebAssembly, etc., but those systems may have cmake and ninja (or just ninja for meson's case) installed already via some other package manager.

[mofeing]

FYI, this comes up because there are platforms where wheels can't be shipped, like the BSD's, Android, WebAssembly, etc., but those systems may have cmake and ninja (or just ninja for meson's case) installed already via some other package manager.

Any news on this? We are trying to install scipy in an offline supercomputer with POWER9 architecture and we are running into problems because of this.

[henryiii]

Mesonpy should list ninja using the requires for build wheel hook and only include it if it's not already installed. CC @FFY00.

https://github.com/FFY00/meson-python/blob/1a4882e9f6d973934185a0f5fec052750983b960/mesonpy/__init__.py#L965

Should have something like this:

        ninja = shutil.which("ninja")
        # (could check version here)
        if ninja is None:
            packages.append("ninja")

(Also assuming it can get ninja via shutil.which("ninja") instead of only using import ninja). And ninja then wouldn't be a hard dependency.

[FRidh]

Mesonpy should list ninja using the requires for build wheel hook and only include it if it's not already installed. CC @FFY00.

https://github.com/FFY00/meson-python/blob/1a4882e9f6d973934185a0f5fec052750983b960/mesonpy/__init__.py#L965

Should have something like this:

        ninja = shutil.which("ninja")
        # (could check version here)
        if ninja is None:
            packages.append("ninja")

(Also assuming it can get ninja via shutil.which("ninja") instead of only using import ninja). And ninja then wouldn't be a hard dependency.

I disagree with this approach. Listing requirements should not be impure, that is, depend on the actual build machine. These requirements are actual requirements, it is just that they can get fulfilled in different ways.

In my opinion the right approach here is to have a build system flag for this ninja package such as USE_SYSTEM_NINJA. And I also think it should be the default, since build systems fetching from the web is trouble.

If you know that you have a ninja installed globally as a C++ command-line utility, maybe you can just package dummy metadata for it to satisfy meson-python?

Exactly, that is what I think this package should provide.

[henryiii]

I disagree with this approach. Listing requirements should not be impure, that is, depend on the actual build machine.

The actual requirement is not a python package. It's the command line tool "ninja". That's why there's a complaint that "it's already installed". The Python package is just one way to get it.

If you know that you have a ninja installed globally as a C++ command-line utility, maybe you can just package dummy metadata for it to satisfy meson-python?

No. If someone installs this version, they should get this version. It is up to the build tool to specify what it needs (via get_requires_for_build_wheel), this package should always install exactly the version it claims to install. If pip install ninja==<this_version> triggers an SDist and that SDist just installs a dummy version that doesn't really install this version, that's a disaster.

The build tool can decide if it wants to allow configuration, it can via this dynamic hook - like USE_SYSTEM_NINJA, or tool.x.min_ninja_version, etc - but the sdist of this package should not.

[henryiii]

A compiled build system in python is never pure. You’re calling a compiler with ninja. We don’t pip install GCC or Clang.

Your proposed dummy package would not be pure, either; it would depend on the system installed version of ninja. Only now, instead of being controlled by the build system, which should know exactly what it requires, it’s just happening as part of an installing an SDist. Two similar systems should produce the same wheel!

Accessing the Internet during the build process (by default) is probably something we should fix. I’m also OK to have opt-in workarounds for some things if needed - though I’d say this one isn’t needed, mesonpy should declare exactly what it needs - it should check to see if there’s an installed ninja and check the version of the installed ninja to verify it is adequate; if it is, it should not add the dependency. Or it might check to see if it's on a system that has wheels, and always add it if it is. Etc.

[eli-schwartz]

No. If someone installs this version, they should get this version. It is up to the build tool to specify what it needs (via get_requires_for_build_wheel), this package should always install exactly the version it claims to install. If pip install ninja==<this_version> triggers an SDist and that SDist just installs a dummy version that doesn't really install this version, that's a disaster.

My apologies, I'm having an exceedingly hard time parsing this paragraph.

I proposed that @FRidh could solve distro integration concerns by performing third-party patching upon this repository and installing dummy "ninja" metadata for python via an OS package manager that uses the OS package manager metadata to record a dependency on an OS package manager version of ninja.

Can you clarify how one might pip install to trigger an SDist, whereby that SDist comes from the .dist-info subdirectory of the current environment's global site-packages? My naive understanding of python packaging was that SDist ("source dist") per definition comes from, well, sources. Not installed binary artifacts.

I would also be interested to know, in the context of "this package should always install exactly the version it claims to install", what possible errors you are afraid might occur if someone tries to pip install ninja and nothing happens because it is already recorded as installed, but users get the "wrong" OS ninja instead of the one lovingly compiled and authoritatively codesigned for authenticity under the aegis of the ninja-python-distributions project. Perhaps there are people who are concerned that the OS distribution contains shady malware and they only trust the ninja-python-distributions developer team?

[henryiii]

I proposed that @FRidh could solve distro integration concerns by performing third-party patching upon this repository

I think that was what the issue was about originally, though I read things like:

In my opinion the right approach here is to have a build system flag for this ninja package such as USE_SYSTEM_NINJA. And I also think it should be the default, since build systems fetching from the web is trouble.

This is suggestion that pip install ninja would revert to a dummy wrapper if ninja was already present. There are multiple problems with this. For one, this means that pip install ninja==1.10.2.4 would not install 1.10.2.4, but would simply do nothing if ninja 1.9 was already installed¹. Also, a valid reason to pip install ninja is that we provide a fork of ninja that adds a feature, jobserver support, that is missing from ninja you likely already have installed. You might be trying to get that, and instead you get the system ninja. But mostly it's the version issue - you should be able to specify a version and you can't specify a version if you are just taking whatever is installed on your system.

I'm not against a distributor providing the "ninja" package by third party patching of this repo. In fact, I can't be, as I'm not a third party. :) Though having talked with conda-forge (I'm a ninja maintainer there too, actually - maybe I'm also a third party in that case), I agree that conda install ninja should not install the ninja python package, because the 'ninja' formula doesn't and shouldn't require Python. It's better to have a python-ninja package, for example (though I can't think of a case where it's useful, see below). If there's a way for me to make that third party patching easier, I'm happy to help, though I'd expect it to be the same ninja version we provide. I'd love to have jobserver support merged or to have this package split into ninja and ninja-jobserver or something, since I can't update to 1.11 because of the fork being behind.

Distributions don't/shouldn't be using isolated builds anyway (for one, isolated builds use internet during install, for a second, they ignore distribution packaging), so this doesn't really matter for distributions, though - just don't pip install ninja.

---

But there's an underlying problem that's actually easily fixable with python-meson. In that case, the exact version of ninja and the job server support isn't a problem at all. In fact, if ninja's already on the system, you can save a tiny bit of time & storage by simply checking for the presence of a valid version of ninja, and only adding ninja to the get_requires_for_build_wheel hook.

I've thought about this extensively for the last couple of months, and have come to the conclusion that it's the right path for common command line tools needed for building. And later I found the completely unrelated example: https://github.com/tttapa/py-build-cmake/blob/6e583bb7f6813794abefb1ef990fb8d20e47bd8f/src/py_build_cmake/build.py#L39 , so it's not just me. Command line build requirements should be declared in the build hooks.

---

Can you clarify how one might pip install to trigger an SDist, whereby that SDist comes from the .dist-info subdirectory of the current environment's global site-packages? My naive understanding of python packaging was that SDist ("source dist") per definition comes from, well, sources. Not installed binary artifacts.

I'm not sure what this refers to. pip install package or pipx run build package can trigger one of two paths if only an SDist of package exists. The "normal" one is an isolated build. This will make a virtual environment, install everything in pyproject.toml: build-system.requires, call the hooks in pyproject.toml: build-system.backend (which includes installing dependencies the build system requests via get_requires_for_build_wheel or get_requires_for_build_sdist (this is how setuptools only requires wheel for building wheels), then builds the sdist or wheel inside this temporary environment. Those dependencies are usually available in wheel form. For non-universal wheels (like ninja), then the tool will download the ninja SDist and try to build it - that's what I've been discussing.

The other path is if you deactivate isolated installs, in which case, it simply tries to build in the existing environment and both the build-system.requires and the hooks are ignored. You can install or do whatever you want manually yourself before starting the build.

It should be noted, in the default "isolated" build case, adding a ninja "python package" to the host environment will not fix the problem at all, since it's building an isolated environment and using pip, not your package manager. So I'd worry you might not actually be fixing the problem I think you are trying to fix with this dummy package approach.

Footnotes

1. A package can't tell if you pinned it, requested a range, or just pip install ninja'd it. "lovingly compiled and authoritatively codesigned for authenticity under the aegis" is a bit overkill, don't you think? ↩

[henryiii]

Do you know what the minimum version of ninja required for meson? I can make a PR to meson-python that fixes this to show what I mean.

[mofeing]

@henryiii According to meson's Release Notes, from version 0.54 it requires ninja 1.7 at least.