Relocatable compiler in dune

[Alizter]

Thanks to David's great work upstream, OCaml 5.5 will be relocatable. Older versions of the compiler also have backports living in a custom opam repository https://github.com/dra27/opam-repository/blob/relocatable/packages/relocatable-compiler/relocatable-compiler.5.4.1.20251109.1/opam.

Supporting the relocatable compiler in package management would mean we can move away from toolchains and build the compiler like a regular package allowing it to be cached.

The constant rebuilds of the toolchains have been a common pain-point for users and the main focus of alternative proposals such as #12179.

In order to use this work in Dune, we need to be able to detect when a given compiler is relocatable. Checking the compiler version for 5.5 is a good start, but for older versions living in David's opam repo we can also check if we are using the relocatable meta package.

For the relocatable compiler to be fully integrated in Dune we will need to do the following work:

🧱 Done

• Be able to interpret the build plans appearing in David's opam repo. They include special opam variables such as build-id which we have so-far avoided interpreting in dune.
  • feat(pkg): interpret %{pkg-self:build-id} variable #13312
  • followup cleanup test(pkg): cleanup opam-var-pkg.t #13318
• Come up with a way to resolve symlinks in the install layout. The compiler will symlink ocamlc -> ocamlc.opt, this is fine but will mean that Dune will refuse to cache the directory target.
  • fix(pkg): resolve symlinks from (install) action  #13239
• Interpret conditional opam variables like %{pkg:ocaml-system:installed?system}
  • support %{pkg:foo?bar} syntax from opam #13322
  • feat(pkg): translate additional conditional syntax #13320
  • Was a typo in opam-repository
  • Typos in dra27/opam-repository#relocatable fix: missing colon in conditional dra27/opam-repository#27
  • Typos in ocaml/opam-repository Correct compiler variable generation in ocaml.5.5.0 opam-repository#29224

🚧 Work-in-progress (in priority order)

• Catch more undefined variables than we currently do. %{pkg:dkml-base-compiler:version} should be an empty string according to opam semantics.
  • fix(pkg): catch undefined package variables #13319
  • test: variables of undefined packages #13465
  • fix(pkg): unknown packages should have variables evaluate to empty #13466
• Detect if a compiler is relocatable (using the heuristic detailed above)
  • feat(pkg): relocatable compiler support #13321
• Always use the relocatable compiler over the older OCaml ones.
• relocatable ocamlfind
  • need to match overlay from relocatable with our overlay
• relocatable ocamlbuild
  • ocamlbuild failing to build dependents if sandbox is not properly cleared up #13467
  • need to match overlay from relocatable with our overlay