Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelise building dynlink_compilerlibs #3250

Merged
merged 1 commit into from
Nov 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Parallelise building dynlink_compilerlibs
This performs two weird tricks that together knock ~30 seconds off the build
time in dev mode and ~40 seconds outside dev mode:

- To build `dynlink_compilerlibs`, copy both `parser.ml` and `parser.mli` from
  the main compiler source rather than just `parser.ml`. Copying just
  `parser.ml` means that the native build depends on the .cmi produced by the
  bytecode build.

- Explicitly add the .cma and .cmxa files for `dynlink` and
  `flambda2_from_lambda` to the `install` alias. It's not clear why this helps,
  but it appears that Dune is being excessively lazy in discovering installable
  targets, waiting until `ocamlcommon` is built before it even considers any
  targets in these libraries, including those that don't depend on `ocamlcommon`
  (and take quite a while to build). Redundantly adding targets to `install`
  seems to be the nudge Dune needs.
  • Loading branch information
lukemaurer committed Nov 11, 2024
commit a64e43ec3ac7d617c8ef94641bb5c2dc80cd4f53
17 changes: 17 additions & 0 deletions dune
Original file line number Diff line number Diff line change
Expand Up @@ -866,3 +866,20 @@
(package ocaml)
(files
(include compiler-libs-installation.sexp)))

; This is entirely redundant, but without it, each target in these libraries
; and their dependencies waits until ocamlcommon is built, including those
; targets that don't actually depend on it. This produces severe bottlenecking,
; costing about 15 seconds when building in dev mode. It's not immediately clear
; why Dune fails to parallelise here. Going by the progress indicator, my best
; guess is that targets in subdirectories aren't even discovered until
; ocamlcommon is built, unless something in the subdirectory is explicitly made
; a dependency.
(alias
(name install)
(package ocaml)
(deps
otherlibs/dynlink/dynlink.cma
otherlibs/dynlink/dynlink.cmxa
middle_end/flambda2/from_lambda/flambda2_from_lambda.cma
middle_end/flambda2/from_lambda/flambda2_from_lambda.cmxa))
19 changes: 12 additions & 7 deletions otherlibs/dynlink/dune
Original file line number Diff line number Diff line change
Expand Up @@ -42,14 +42,21 @@
(targets parser.ml)
(deps ../../parser.ml)
(action
(bash
"echo \"module MenhirLib = CamlinternalMenhirLib\" > parser.ml && cat ../../parser.ml >> parser.ml")))
(with-stdout-to %{targets}
(progn
(echo "module MenhirLib = CamlinternalMenhirLib")
(cat %{deps})))))

(rule
(targets camlinternalMenhirLib.ml)
(deps ../../parsing/camlinternalMenhirLib.ml)
(targets parser.mli)
(deps ../../parser.mli)
(action
(copy %{deps} %{targets})))
(with-stdout-to %{targets}
(progn
(echo "module MenhirLib = CamlinternalMenhirLib")
(cat %{deps})))))

(copy_files ../../parsing/camlinternalMenhirLib.ml{,i})

(library
(name dynlink_compilerlibs)
Expand All @@ -59,8 +66,6 @@
(:standard
-strict-sequence
-absname
-w
+67
-bin-annot
-safe-string
-strict-formats))
Expand Down
Loading