cmake_with_sdist equivalent in scikit-build-core

[MuellerSeb]

Is there something similar to the legacy cmake_with_sdist in scikit-build-core (beside the setuptools backend)?

An if so (or if planed), is there a CMake variable to indicate we are currently building sdist?

Cheers,
Sebastian

[henryiii]

See #319 (which is about the backward compat option, but I'd expect a pyproject.toml equivalent to be available about the same time), also #249, which discusses the need. I think the main issue is coming up with a good design. Would love as many examples of usages of this as possible, I only have two so far (in #319).

I don't have custom variables to indicate what state scikit-build-core is in, but that's a good idea. You could also detect wheel vs. metadata preparation vs. editable install with a variable like that.

[MuellerSeb]

Cool to hear. I think variables like ${SKBUILD_STATE_SDIST} would be cool to do stuff like this.
For me this (see #249) is the only blocker for switching to scikit-build-core ATM.

[henryiii]

Do you have a pre-switch example? I'd like to have examples of what this looks like in practice to have a good idea of what to design.

I'd probably go with ${SKBUILD_STATE} which would be sdist (when that's added), metadata_wheel, wheel, "metadata_editable", or editable.

[MuellerSeb]

Here:

• https://github.com/mhm-ufz/mHM/blob/v5.13.0/setup.py#L27

This is a custom sdist command with scikit-build. Would like to do that downloading in CMake.

PS: editable could then be used to infer the correct install location, right?

[henryiii]

FWIU, that's not actually running CMake when making the SDist, but downloading a tarball?

Ahh, yes, "Would like to do that downloading in CMake". FWIW, you can do this with git submodules today with scikit-build-core.

Okay, thanks, good to have a data point.

[MuellerSeb]

git submodules is of course an option, but we are relying on CPM.cmake, so I think knowing the skbuild-state is sdist to copy the downloaded folder could be convenient.

[LecrisUT]

Is the issue that skbuild does not run cmake before sdist creation? Otherwise it should be possible with a combination of setting the cmake source location and sdist.include option right?

[henryiii]

Currently CMake does not run (and isn't even required) when making an SDist. It's only run for the wheel step. The tricky part here is figuring out what you'd do with CMake during the SDist step. You don't want the config output build directory in the SDist, but you do want the downloaded packages (which are in the build directory as output...)

Examples I see of this currently with scikit-build (classic) generally have a lot of python dedicated to copying in an overloaded sdist command.

[henryiii]

For CPM.cmake, maybe we could integrate with them, and their CMake files could be even distributed in a package. Maybe we should target solving that specific case first, then a more general solution might be more obvious.

[LecrisUT]

The download location is configurable via a cache variable FETCHCONTENT_BASE_DIR, and the download is done at configure time. Similar case is for cpm. So as long as a hook is added to run configure, it should be good to go right? Although having a sdist.artifact would also be beneficial in this case

[henryiii]

Hmm, we could save {build_dir}/deps in the SDist then populate the build directory with it when making the wheel. As long as it's fine to delete all of the other files & still be able to build without internet (which would be the reason to put it into the SDist). I believe that's exactly what it does if there's existing content there.

I'd also want to avoid saving the config output of the fetch content, as the environment when making the SDist is not the environment when making the wheel.

[LecrisUT]

Why not have the user setup dedicated folder in the source directory? That way it does not have to be relocated, and if the setting is in pyproject.toml, then it is reproducible at the wheel stage without internet connection. The folder structure might end up a bit weird, but that can be corrected by setting SOURCE_DIR in the individual package's configuration. So I think it's enough to simply implement the trigger to configure and a sdist.artifact. It shouldn't otherwise be a default to include the dependencies because of distribution licensing issues.

[henryiii]

It won't ever be a default since we won't ever default to running cmake when making an SDist. :) Making it too automatic, even when enabling CMake during the SDist step, might be something to keep in mind as possibly undesirable, though.

We may want to implement hatchling-style force-include first, that likely would be handy for this (as long as there was some way to point at the build directory). Actually, I wonder if force-include + running cmake in the SDist step would be enough? I guess it would need to go both directions... Okay, maybe sdist.artifact should be a dict - mapping an sdist path (as a key) to a build path (it still would share a lot of implementation with force-include).