Reduce the number of places where deps versions are picked up from

[ViralBShah]

Currently we have to do version bumps for a deps library update in 3 places:

1. Project.toml
2. Versions.make
3. libname.version

The Project.toml is for picking up the right library from Yggdrasil, but the other two could perhaps be consolidated a bit in the build system. In an ideal situation, we would centralize all this information in our build system for each library in only one place.

[ViralBShah]

I know that libname.version allows picking a branch and commit, in a more fine grained way than Versions.make does. We have needed that for many libraries (esp. libuv and openblas) in the past, so we don't want to lose that - but it would be nice if it were all consolidated in one file per library or something.

Ideally the Project.toml versions also get picked up from this same file, but sometimes we need to pass the BB number as well - so we need to accommodate that too.

[StefanKarpinski]

I think the ultimate plan here is to make all our build deps into "just JLLs" and then the differences will go away. At that point we'll need some way of doing source builds and using those instead, but I think there's a rough plan for that as well. cc @staticfloat — thoughts?

[staticfloat]

You're right about the "happy path", but I don't have a good plan for how to separate the from-source build steps from the rest of the build system. It may be that we write some kind of tool that downloads the build_tarballs.jl scripts from Yggdrasil and lays them out on disk such that you can run them, but that's not very well thought-through yet, as there's often BB-specific stuff in build scripts. But it shouldn't be too bad to restrict ourselves from using those fancy tools in just the Julia deps.

[ViralBShah]

Isn't it possible for now to just pick all the version numbers from one place - by reorganizing the Makefiles a bit? At least I feel like 2 and 3 should be possible to easily combine since they both live in deps. The issue is that I always forget where all to update version numbers.

[staticfloat]

I agree that there doesn't need to be both Versions.make and libname.version. We could combine the two (probably by just putting everything into Versions.make).

[inkydragon]

Put everything in Versions.make may cause some problems (See comments in #45691).

@vtjnash suggest that we can split Versions.make into separate .version files.

We don't want to do this for deps either, since it means we can't determine as accurately when they need to be updated, and it increases the chances that backporting something ends up in a merge conflict. But we could probably move the BB versions into the .version files, to keep all of the ways of specifying it in one place.

Originally posted by @vtjnash in #45691 (comment)

We want to "centralize all this information in our build system".
I have no preference between the two options.

So I would like to ask your opinion (@ViralBShah @staticfloat), if you agree to split Versions.make, I will close #45691 and open a new pr.

Draft patches: Comparing JuliaLang:master...inkydragon:build-rm-Versions.make · JuliaLang/julia

[ViralBShah]

Splitting Versions.make into separate .version files seems like a great idea to me. That should also remove redundancies across Versions.make and .versions files.

[inkydragon]

There are utf8proc_jll.jl and Patchelf_jll.jl, so why we don't use it?
Of course they may need some updating.

Other deps without a jll package

• libwhich
• mozillacert: as a part of libgit2

Should we add two jll packages for them?

[ViralBShah]

That's a good idea. It would be good to understand if this is by design, and if so, what the issues are. It is quite likely nobody got around to them!

[ViralBShah]

I believe we don't use those JLLs because they are only used at build time, but are not distributed.

Otherwise, I feel that the recent round of cleanups that @inkydragon did cleaned this up sufficiently that I can close this issue.