Conda installer

[benvanwerkhoven]

It appears there are some HPC systems where users that are allowed to install software using conda, but not pip. It would be great if people could install Kernel Tuner using conda in addition to the currently supported installation options. I don't know how to do this nor much work this is, so it would be great if someone who knows could help out.

[isazi]

I read this article today, so it seems that a conda package can be created very easily. I tried and it actually works, because kernel_tuner is in pypi. Should we just point the users to this? Or create the packages and upload them to the Anaconda cloud? The latter requires to have an anaconda account.

[wjp]

The big question here is probably how to handle the (required and optional) kernel-tuner dependencies in a conda-friendly way.

For the required dependencies, the recent addition of python-constraint is moderately problematic, as that package is not in base conda. It is in conda-forge, so this may require going for the conda-forge community package ecosystem rather than the base conda ecosystem.

For the optional ones, pycuda, pyopencl and scikit-optimize are all similarly not in base conda but in conda-forge. nvidia-ml-py is in base conda under the name nvidia-ml.

The current conda version is also not great with optional dependencies. We'd have to decide if we want to make pycuda/nvidia-ml/pyopencl required dependencies, or if we want to do some meta-packages or package variants to let the user choose if they want to install the cuda/opencl support, or if we want to require the user to install the dependencies manually if they want to use cuda/opencl functionality. (Or some other option I'm not immediately seeing)

[fjwillemsen]

For python constraint we could opt to include it directly as a submodule or similarly, because it appears to be no longer maintained and we may want to adapt it ourselves in the future.

[wjp]

By the way, I can confirm that using conda-skeleton as @isazi mentioned does indeed produce a working conda package, ignoring how to handle dependencies for the moment. Installing python-constraint cupy pycuda pyopencl nvidia-ml ocl-icd-system scikit-learn scikit-optimize pytest from the conda-forge conda channel makes all tests pass.

[bouweandela]

Having the package available on conda-forge would be convenient for the project that I'm working on. Would you like some help with this?

[benvanwerkhoven]

Yes, please! You're more than welcome to help out with creating a conda installer!

[bouweandela]

I just opened a pull request to add it on conda-forge conda-forge/staged-recipes#24420.

[bouweandela]

Does kernel-tuner support Windows? I'm seeing some test failures in the conda-forge build here.

[fjwillemsen]

As far as I'm aware it only supports Linux and Mac.

[benvanwerkhoven]

We indeed don't actively support Windows right now. Kernel Tuner did work with Windows at some stage in the past, but Windows support is not regularly tested nor maintained. I do use Windows on my laptop, but doing everything in WSL is so much more convenient not having to use Windows-based compilers. I think most of the other core developers are fully on Linux or MacOS.

Looking at the errors, it's likely not even that much work to support it, but at least I can say that we are currently not supporting it. If anyone reads this in the future and would like to use Kernel Tuner with Windows, then please feel free to let me know.

[bouweandela]

Thanks! In that case I will just build the package for Linux and OSX.

[bouweandela]

The kernel-tuner package is now available on conda-forge: https://anaconda.org/conda-forge/kernel-tuner and can be installed with conda install -c conda-forge kernel-tuner.

The feedstock is here: https://github.com/conda-forge/kernel-tuner-feedstock

A bot will automatically open a pull request there when it finds an updated version of kernel-tuner on PyPI, but updating the list of dependencies in recipe/meta.yaml needs to be done manually. Since there are fairly strict upper pins on the dependencies, this needs a careful review with each update.