Split out nightly tests?

[icweaver]

Following up from this discussion in slack #general, and the linked discourse post, I've got half a mind to split out our nightly tests from stable. Is there any reason that we shouldn't do this? Otherwise, it seems like we're just losing information

[icweaver]

Started a PR #158

And added a package-level example here: JuliaAstro/Astroalign.jl#15

And another one using pre instead of nightly: JuliaAstro/Photometry.jl#85

[icweaver]

I'm leaning towards the pre-release route over nightly for more useful reporting (e.g., JuliaAstro/Photometry.jl#85). Will give it one more week then start implementing this in our other JuliaAstro packages unless folks feel strongly otherwise

cc: @cgarling @abhro

[cgarling]

Using pre-release rather than nightly is cool with me. I have not personally noticed failures on nightly for the packages I help maintain and splitting out the tests like you do here will require maintainers to edit two different workflows if they ever need to make changes the CI job, which seems like an easy thing to forget. Have failures on nightly have been more of an issue for the packages you maintain? It would be helpful if you could demonstrate the extent of this issue for our repositories before we implement a solution that complicates our CI setup.

We could also try sticking to a single CI workflow but switching from nightly -> pre and see if that fixes any CI failures you've been encountering.

[icweaver]

Thanks, Chris! That's a good point about increased maintenance burden. Ok, just flipping from nightly to pre sounds like a good compromise. Nope, I've only noticed the sporadic failure on nightly for Photometry.jl or GeneralAstrodynamics.jl, which looks to be a monorepo going through some pretty comprehensive development rn anyway. It's honestly pretty cool seeing how stable everything has been

[abhro]

I also agree on the experimental testing being done on pre-releases rather than nightly builds.