Replace SCon with modern build tools, rewrite setup.py and add pyproject.toml

[Tieqiong]

This is tested (with the original unittest) with python3.13 and numpy2.2.1 on Mac. Not sure about the state of pyobjcryst so not currently testing with it. I also tried one of the example codes and it works until pyobjcryst is needed.

One thing I'm confused about is I neither have libdiffpy installed in my testing environment nor mentioned it in any of the building files... But srreal still weirdly works

@sbillinge also maybe it's better to make a new branch cookie and also commit this to the cookie branch

[sbillinge]

thanks so much for this @Tieqiong, very exciting.

This failed pre-commit. You will have to update your local from your branch to get the auto-changes and then repush, but better yet, initialize pre-commit on your local so this happens locally when you commit and before the push and all the auto-updates (black, isort etc.,) pass without them any updates being done on GitHub.

I wonder why no other tests are running. Is it because this has not been cookiecut?

When this is ready to merge I will merge it into cookie.

That's super-weird about libdiffpy. Lets see if it passes CI on GitHub which will be a tougher test.....

[Tieqiong]

@sbillinge I think the pre-commit (and other tests) are not running because this repo haven't been set up for them (it doesn't have .pre-commit-config.yaml and the .github workflows). This should be fixed by cookie cutting.

I don't think it's a good idea to bring back the tests and pre-commit here, as it will generate all of the black, flake8, isort, codespell changes which would caused hundreds of modifications... Let's fix them in a separate PR after merging this to a cookie branch. Also I attached the local unittest result to show my changes are working fine.

After merging this to a cookie branch, I can make further PRs on the pre-commit and auto fixing stuff. Thanks

[Tieqiong]

Also for the libdiffpy I think the reason is because I installed diffpy-cmi sometime before in my top local/include directory so now srreal can find the required c++ files. Still need to make changes to update libdiffpy so that it can be co-installed with srreal.
libdiffpy might also have that Mac arm problem on conda-forge... Need to figure out way to build it with GitHub workflow. This might be harder than pdffit2 because libdiffpy is not a python package yet

[sbillinge]

Agreed, let's not cookie cut this yet. I can merge this PR into a cookie branch, but without tests running in CI how do we know if we are making progress. What is easiest for you? If I merge this to cookie?

[Tieqiong]

@sbillinge With the problem about libdiffpy figured out (as I stated before), I think we still (of course) need libdiffpy to be fixed before being able to run a meaningful CI here... At least my local test is running, which suggests we are making progress!

What I would do is to mainly focusing on updating libdiffpy while slowly cookie cutting this on a cookie branch. I'll make sure the test still runs on my end before every PR. As long as we have both set up correctly everything will be fine I hope.

[sbillinge]

@Tieqiong ok, I merged this into cookie. Is there a way to have CI run tests as a next step, without a full cookie cut?

[Tieqiong]

@sbillinge I think I can make the tests work and bypassing the libdiffpy conda version limit by changing the CI workflow files a bit. However things may not necessarily be the same after updating libdiffpy.

The problem with libdiffpy is that it's essentially a c++ library, and we distribute it using conda-forge. However Conda-forge can't build for Arm Mac. The solution in pdffit2 was to generate a wheel and distribute via pypi. This could be a bit tricky for libdiffpy as it's not a python package. However I think this is still the most viable way.
Some other ways could be merging libdiffpy into srreal (like what we have for pdffit2: libpdffit2 is inside pdffit2) or find another way to distribute this c++ library for Mac.
Maybe this should be better be discussed in libdiffpy. I'll make an issue there

[sbillinge]

I see, that's very tricky. Can we separate the arm64 issue from the others and do a release for everything except arm64? then figure out an arm64 solution afterwards?

I honestly doubt that anyone is using libdiffpy except as part of diffpy-cmi, so we could include it inside cmi somehow, though this is a pretty big perturbation to everything else just to make it work for arm64, so I would rather not if we can find another way to distribute libdiffpy for arm64. But can we put this off to later?

[Tieqiong]

Yes, I agree we should as least make it work for windows and linux first then figure out the other things. Additionally, I think it would be beneficial to aim for better clarity and make things easier to maintain and update in the future.

[sbillinge]

yes I agree. We have done this for the pure python packages, but the packages containing c seem to always be a bit harder!