conda-forge packaging of pyclaw

[zezl7]

Hello 👋,

I am a student at the University of Stuttgart following the Simulation Software Engineering course, and would like to package your software in order to submit it to the conda-forge channel following the official contributing packages guide.

After conducting a small research, I found no pyclaw package on the conda-forge channel, neither did I find any pyclaw package using the official anaconda search.

Before I attempt to do this, I would like to ask if there is a reason for the lack of the packaged software, and if there are any impediments that should be taken into account if one decides to do so?

Thank you in advance and looking forward to hearing from you.

[mandli]

Welcome @zezl7! We have supported conda installation in the past, but ran into issues that we were unable to resolve cleanly, I believe due to dependencies that we were trying to include. The conda-recipe is actually located in the super-repository to this one, so I might suggest taking a look at what's there.

Definitely would be nice to have a conda-forge package working again. I think possibly paring back what we were trying to do (dependencies that were giving us trouble) would be an option.

[zezl7]

Hello @mandli,

Thank you for your fast reply!

For my Simulation Software Engineering course, my plan will then be to:

1. Work on the existing conda-recipe in the super‑repository by addressing the dependency issues
2. Submit a PR with the updated recipe in the super‑repository for your review
3. If accepted/merged, submit a PR to conda-forge/staged-recipes repository to be reviewed and have the PR merged, creating the pyclaw feedstock

As mentioned in the first comment, this approach will follow the official conda-forge process for new packages, and I will document all changes, incl. crediting your original recipe.

If you have any tips from your past dependency issues, that would be much appreciated! Thank you once again for the encouragement and invested time.

[mandli]

This sounds good @zezl7. It may be worth thinking a little bit about how PyClaw vs. all the rest of the packages interact, but I think you should proceed as you suggested and ask questions as they arise and we can maybe figure out what the right end installation scope should be. @ketch and @rjleveque do you have any thoughts?

[ketch]

@zezl7 Thanks; it would be really great to have a conda package of Clawpack again. Just please note that the Python package name is clawpack, and that should be the name of the recipe. pyclaw lives under the clawpack namespace.

Clawpack includes one pure Python submodule (visclaw) so it would be nice if the recipe installed that as well. The other main sub-packages of Clawpack (Geoclaw, AMRClaw) have chosen not to adopt a library-style structure and can't really be installed through a package manager, so what we did in the past was to have the Conda package just ignore them.

[rjleveque]

Thanks @zezl7 for tackling this.

In addition to pyclaw and visclaw, the clawutil module includes some things that might be useful more generally.

Regarding amrclaw and geoclaw, I thought the main reason for not being able to fully include these is that the Fortran code isn't exposed in the way needed to run the Fortran versions, as we usually do with those packages. But the Python code that's included has the same structure as in visclaw, for example (e.g. in $CLAW/geoclaw/src/python/geoclaw) and includes many Python tools (e.g. topotools.py) that are often useful even if one isn't running the Fortran codes. In particular, many other Python codes and Jupyter notebooks (e.g. in the gallery) need to import these tools in order to work for post-processing GeoClaw run data, often by collaborators who aren't running the Fortran code themselves. (E.g. I have Jupyter notebooks running on the Design Safe Jupyter Hub that our collaborators use.)

If you pip install clawpack then these geoclaw tools are available, so I would think it should be possible to do the same with Conda? On Design Safe we have to first create a Conda environment and then pip install clawpack within it to get the geoclaw tools, but it would be nice if this was a single step.

[ketch]

Thanks @rjleveque , and sorry that I wasn't very clear. I think the conda package should install all the Python code that pip install clawpack does. I just meant that the conda package shouldn't try to install compiled binaries (of the Fortran code) for Geoclaw and AMRClaw.

[zezl7]

Thank you for the insights @mandli , @ketch , and @rjleveque! Taking them into account, I have refined the proposed plan and included the requirements. In case of any discrepancies, please let me know.

Requirements

• The name of recipe should be clawpack
• The recipe should install all Python code that pip install clawpack installs, including the submodules visclaw, geoclaw, amrclaw, clawutil. This should result in a singular step, e.g., conda install -c conda-forge clawpack
• The recipe should not install compiled binaries of the Fortran code of geoclaw and amrclaw

Contribution Plan

1. Work on the existing conda-recipe in the super‑repository by addressing the dependency issues and integrating the specified requirements
2. Submit a PR with the updated recipe in the super‑repository for your review
3. If accepted/merged, submit a PR to conda-forge/staged-recipes repository to be reviewed by following the official conda-forge process for new packages (incl. crediting your original recipe)
4. Have the conda-forge/staged-recipes PR merged, creating the clawpack feedstock
5. Update the clawpack installation instructions for developers by following the official guide

Q&A

• Further communication: Is there a dedicated channel you are using, or should I ask arising questions here in this thread?

[zezl7]

Hello @mandli , @ketch , and @rjleveque.

As you can see from the associated PR, I managed to create a working and updated version of the conda-forge recipe.

Approach

I started by exploring what clawpack is all about, what its submodules (including pyclaw) are, how to install it, and how the source code is structured. Specifically, I inspected the build files: pyproject.toml and setup.py, to better understand how the current PyPi package is built. By doing this, I realized that there are three different ways to install clawpack depending on your needs, seeing that this is a hybrid-project containing a mixed source code suite of Python and Fortran code. Lastly, I inspected the original recipe and tried to understand how it lead to the dependency issues explained by you.

The requirements for the new recipe were to

• ... mirror the behavior of pip install clawpack, as these Python parts are available.
• ... include the pure Python submodule visclaw.
• ... include the Python codes of the Fortran submodules geoclaw and amrclaw (e.g., $CLAW/geoclaw/src/python/geoclaw) because they "[include] many Python tools (e.g. topotools.py) that are often useful even if one isn't running the Fortran codes" in Jupiter notebooks for example.
• ... include the clawutil submodule as it "includes some things that might be useful more generally".
• ... keep in mind how pyclaw interacts with the rest of the packages.

Based off these requirements, I started experimenting with the old recipe, trying to reproduce the errors and spot more precise dependency misconfigurations from the error traces. For this, I cloned the clawpack feedstock on my Ubuntu 24.04, where I have Miniforge pre-installed, and created a dedicated environment with conda-build.

Changes

After testing the original recipe, I noticed it was using the build system setup.py, which was the standard at the time the recipe was created - at Clawpack 5.9.0. Now, 5.14, Clawpack has mainly transitioned to meson-python as specified in pyproject.toml. However, pip install triggers setup.py for dependencies, which requires the recipe to compile Fortran code. So, in the end either build system needs Fortran code to be compiled.

The workaround I came up with, in order to avoid compilation of Fortran code, was to unfortunately move away from the already existing meson-python and create a custom setup.py and pyproject.toml within the recipe's directory. The idea is to recursively find all Python packages and filter only the clawpack-related ones (pyclaw, visclaw, clawutil, geoclaw, amrclaw, riemann, classic). This installs 186 files without Fortran extensions (e.g., clawpack.riemann can be imported but advection_1D is absent, or geoclaw.topotools works but its Fortran solvers fail at runtime). This is an outtake of the command line output after building the recipe:

Packaging clawpack
Packaging clawpack-5.14.0-py310he53d0f1_0
compiling .pyc files...
number of files: 186
Fixing permissions
Packaged license file/s.

Additionally, the recipe was using MPI dependencies (e.g., mpich, mpi4py, petsc4py, h5py * mpi_*), which are marked as optional in pyproject.toml. Removing these helped fix the MPI related errors. If needed, users can still install them separately.

I followed the v0 template specified by the official conda-forge contributing documentation.

From the original recipe, I removed

• ... optional dependencies (mpi, h5py, mpi4py, petsc4py) and their tests.
• ... Fortran and C compilers.
• ... the patch, as it is no longer needed.
• ... the conda_build_config.yaml, which is also not needed anymore, as it is MPI dependent.

Results

In the end, the fixed and updated recipe works. The created conda-forge package is Python-only. It installs all Python code matching pip install clawpack, and the Fortran submodules (geoclaw, amrclaw, classic) can be imported, with only the respective Python tools working (e.g., topotools).

Notes

• Including clawpack.pyclaw triggered a logging error , KeyError, I was not able to fix, hence there is no test for pyclaw in the recipe. The reason for the error is that clawpack/pyclaw/__init__.py tries to load a log.config file that the current build does not have.
• The PR is failing the macOS checks. This needs to be further tested and inspected.

[ketch]

Thanks, but (I'm sorry to say it) I think this result is totally useless and would be actively harmful to advertise. The whole point of a conda package was to allow users to have a working pyclaw installation without needing to compile the code themselves. This just gives them a broken copy of pyclaw. For the other parts, the same thing can be achieved by just cloning from github and setting the python path. It gives users no path to having a working installation, and will likely make it even more difficult to later get clawpack working on their system if they have already installed this broken version.

[zezl7]

I understand your feedback @ketch , however in order to implement the requirement that Fortran binaries should not be compiled, the state of the current recipe is the only way I made it build without errors. Before that it was a constant fight between the two build systems in the existing setup.py and pyproject.toml that both one way or another end up building the Fortran code.
I also think that even if the current state of the recipe is not satisfactory to your expectations/needs, it is still a better starting point for further development than the original recipe dating back to ClawPack 5.9.0. Therefore, I would not call the entire contribution „totally useless“, as the changes that were implemented, were done so to avoid currently occurring errors for users.
I am opened to discuss further improvements, and try to apply them, as the goal here is to produce an updated recipe that both fits your expectations and those of the users.

[ketch]

Let's continue the conversation over in the PR.