core_kernel__.cmx is not installed in beta18

[ivg]

As a result code that even doesn't use jbuilder but depends on core_kernel and that worked for the past couple of years now fails to compile with the following message:

+ /home/ivg/.opam/bap-testing/bin/ocamlfind ocamlopt -linkpkg -g -linkpkg -predicates used_config -predicates used_bap-build -pp 'ppx-jane -dump-ast -inline-test-drop' -predicates ppx_driver -package ppx_jane -package core_kernel -package ocamlbuild -package findlib -package compiler-libs -I lib/bap_config -I lib/bap_build -I tools lib/bap_config/config.cmxa lib/bap_build/bap-build.cmxa tools/bapbuild.cmx -o tools/bapbuild.native
-build.cmxa tools/bapbuild.cmx -o tools/bapbuild.native
# File "_none_", line 1:
# Error: No implementations provided for the following modules:
#          Core_kernel__ referenced from lib/bap_build/bap-build.cmxa(Bap_build)

It works fine with jbuilder beta16 and beta17 (and probably with all previous versions) but doesn't work with beta18. This is reproducible for the following versions of the OCaml compiler: 4.03, 4.04, 4.05

It looks like that the problem is that the core_kernel__.cmx file is no longer installed with jbuild.1.0-beta18:

ls ~/.opam/jbuilder16/lib/core_kernel/core_kernel__.*
/home/ivg/.opam/dev/lib/core_kernel/core_kernel__.cmi
/home/ivg/.opam/dev/lib/core_kernel/core_kernel__.cmx
/home/ivg/.opam/dev/lib/core_kernel/core_kernel__.cmt 
/home/ivg/.opam/dev/lib/core_kernel/core_kernel__.ml-gen

$ ls ~/.opam/jbuilder18/lib/core_kernel/core_kernel__.*
/home/ivg/.opam/bap-testing/lib/core_kernel/core_kernel__.cmi  
/home/ivg/.opam/bap-testing/lib/core_kernel/core_kernel__.mli-gen
/home/ivg/.opam/bap-testing/lib/core_kernel/core_kernel__.cmti

P.S. This code is from BAP

[ghost]

Ok, we stopped installing it as we thought it wasn't necessary but it seems that it sometimes is in the end. I suppose it must be when not building with -no-alias-deps.

[ghost]

Sorry for the troubles

[ivg]

Thanks for a super rapid response :)

[rgrinberg]

Wonder how our revdeps missed this. cc @jpdeplaix.

@ivg we'll make a release soon.

[kit-ty-kate]

@rgrinberg probably because we do not test dependencies recursively, only package that depends directly on the new/changed package.
Also we only test the latest version available of each packages, not the older versions but this is not the case on the ARM64 CI. However the ARM64 CI does not have as much computing power as the x86_64 one and it was stalling at the time.

[ivg]

Wonder how our revdeps missed this.

Because we do not depend on jbuilder directly, we're depending on a library that is dependent on it. In this case it was core_kernel. The problem was that jbuilder produced an incorrect library that references a module but doesn't provide it, as Core_kernel__ is in the list of interfaces imported by core_kernel.cmxa, but the cmx file is not present.

Ok, we stopped installing it as we thought it wasn't necessary but it seems that it sometimes is in the end. I suppose it must be when not building with -no-alias-deps.

It also might depend on -linkpkg option of ocamlfind and -linkall option of ocamlopt. I think a good smoke test would be to verify that there is a cmx file for each imported interface.

Since jbuilder produces libraries as artifacts, I think it would be a good idea to test not only direct dependencies, but also those packages that are dependent on produced libraries. (that probably will mean the whole opam repo).

[ghost]

BTW, the problem was that we were using a mli only module for Core_kernel__, so the issue might arise with any library that uses mli only modules. This is a problem that would need to be fixed in the compiler.

It's unlikely that -linkpkg has an effect here, all it does is cause ocamlfind to pass the .cma/.cmxa files to the compiler. -linkall might have however.