[OCaml] Add (cross-)compiler shards

[topolarity]

I'm sure there's plenty that needs fixing here, but the main build is working at least. I also had to move to a pre-release of v5.4.0 since OCaml's cross-compiler support was only recently shored up in Dec 2024 (great timing for us!)

companion PR's: JuliaPackaging/BinaryBuilderBase.jl#425 and JuliaPackaging/BinaryBuilder.jl#1382

[topolarity]

I opted to go the GCC route and use the compiler target as the "target" for the build
(even though the toolchain we use during the build is the host toolchain for the most part, we do need both)

[topolarity]

I noticed during this process that I also need to declare a runtime dependency on gcc (it is invoked by ocamlopt)

I don't know whether that's necessary for these shards, since that dependency should be handled pretty much automatically (unless someone specifically excludes the :c compiler I suppose), but it is needed for the JLL artifacts in O/OCaml/

[topolarity]

I think I have most of the bones in-place here

@giordano should I start generating + uploading these shards, or is there something else to do first?

[giordano]

Uhm, no, I think building and publishing the shards should be good now.

[maleadt]

Hm, I'm surprised CI doesn't work here. Does it maybe not use the manifest from the PR, but instead the one from master?

[giordano]

There's a merge conflict in the Manifest, unfortunately the buildkite plugin ignores conflicts and moves on, instead of erroring out (been there already...).

[maleadt]

While working on this, I realized we not only need a host-native OCaml compiler during generating of target compiler shards (to bootstrap OCaml), but often also during JLL generation in order to bootstrap projects that rely on executing OCaml binaries during their build process. So I'm currently distributing that as part of $prefix/host, which is ugly.

@giordano I was wondering if it would be feasible to have OCaml recipes rely on both a host-native and cross-compiler shard, with the former "activated" by default (wrappers being generated for and env vars pointing to). I guess we do something similar already to provide both $cc and $host_cc.

EDIT: Actually, the Rootfs logic in BinaryBuilderBase looks pretty straightforward. I'll try to add the logic there.

[giordano]

I guess we do something similar already to provide both $cc and $host_cc.

If I understood correctly what you're suggesting, that's pretty much it, yes.

EDIT: Actually, the Rootfs logic in BinaryBuilderBase looks pretty straightforward. I'll try to add the logic there.

🚀

[maleadt]

Should compiler shards be built for x86_64-apple-darwin/aarch64-apple-darwin or x86_64-apple-darwin14 and aarch64-apple-darwin20?

[giordano]

That's a good question. In https://github.com/JuliaPackaging/BinaryBuilderBase.jl/blob/c25f6edc4be8c610d830468fe1154dfd6281d582/Artifacts.toml I see both shards with the OS version (for the C toolchains) and without (for the Rust toolchain).

[maleadt]

The problem is that even when building the shard for x86_64-apple-darwin14, it gets mounted at /opt/x86_64-apple-darwin, while the other shards appear at /opt/x86_64-apple-darwin14. Worse, the generated wrappers appear at / point to /opt/x86_64-apple-darwin14/bin, resulting in broken invocations:

sandbox:${WORKSPACE}/srcdir # which ocamlc
/opt/bin/x86_64-apple-darwin14-libgfortran3-cxx11/ocamlc
sandbox:${WORKSPACE}/srcdir # ocamlc
/opt/bin/x86_64-apple-darwin14-libgfortran3-cxx11/ocamlc: line 5: /opt/x86_64-apple-darwin14/bin/ocamlc: No such file or directory
sandbox:${WORKSPACE}/srcdir # /opt/x86_64-apple-darwin/bin/ocamlc
No input files

EDIT: ah, I still had a x86_64-apple-darwin shard in BinaryBuilderBase's Artifacts.toml, and apparently find_shard prefers to return the one without an OS version...

[maleadt]

Okay, this finally seems to be working. I uploaded compiler shards for all supported platforms, and verified that I can build Heptagon using them.