[new release] dune (1.0.0)

[ghost]

Fast, portable and opinionated build system

• Project page: https://github.com/ocaml/dune
• Documentation: https://dune.readthedocs.org/

CHANGES:

• Do not load the user configuration file when running inside dune
  (Do not load the user configuration file when INSIDE_DUNE is set dune#700 @diml)

• Do not infer ${null} to be a target (Add test to show that jbuilder mishandles ${null} dune#693 fixes Jbuilder infers ${null} to be a target dune#694 @rgrinberg)

• Introduce jbuilder.configurator library. This is a revived version of
  janestreet's configurator library with better cross compilation support, a
  versioned API, and no external dependencies. (Add a blackbox test suite for configurator dune#673, Use Whitequark's Code to extract #define's and other compile time info from object files dune#678 Move current configurator API to V1 module dune#692, Pass ocamlc explicitly to configurator using DUNE_CONFIGURATOR dune#695
  @rgrinberg)

• Register the transitive dependencies of compilation units as the
  compiler might read .cm* files recursively (Compute transitive modules depencies dune#666, fixes inconsistent assumptions when using module aliases dune#660,
  @emillon)

• Fix a bug causing jbuilder external-lib-deps to crash (Make sure all_lib_deps_by_context doesn't report invalid contexts dune#723,
  @diml)

• -j now defaults to the number of processing units available rather
  4 (Make -j default to the number of processing units available dune#726, @diml)

• Fix attaching index.mld to documentation (Fix 717 dune#731, fixes Problem generating index.mld pages dune#717 @rgrinberg)

• Scan the file system lazily (Scan the file-system lazily dune#732, fixes Unscannable directory stopping the build dune#718 and jbuild-ignore not respected when there's a nested symlink dune#228, @diml)

• Add support for setting the default ocaml flags and for build
  profiles (Environment & build profiles dune#419, @diml)

• Display a better error messages when writing (inline_tests) in an
  executable stanza (Better error message for #745 dune#748, @diml)

• Restore promoted files when they are deleted or changed in the
  source tree (Fix #759 dune#760, fix .merlin doesn't get regenerated when deleted dune#759, @diml)

• Fix a crash when using an invalid alias name (Fix 761 dune#762, fixes Warn/error when alias have invalid characters dune#761,
  @diml)

• Fix a crash when using c files from another directory (Fix 734 dune#758, fixes
  Exception "Build_system.get_collector called on closed directory" dune#734, @diml)

• Add an ignored_subdirs stanza to replace jbuild-ignore files
  (Add ignored_subdirs stanza dune#767, @diml)

• Fix a bug where Dune ignored previous occurrences of duplicated
  fields (Fix sexp record bug dune#779, @diml)

• Allow setting custom build directories using the --build-dir flag or
  DUNE_BUILD_DIR environment variable (Implement --build-dir and change Path.t to use symbolic paths dune#846, fix Ability to tell jbuilder where to place build artifacts. dune#291, @diml @rgrinberg)

• In dune files, remove support for block (#| ... |#)) and sexp
  (#;) comments. These were very rarely used and complicate the
  language (Block strings + remove block/sexp comments dune#837, @diml)

• In dune files, add support for block strings, allowing to nicely
  format blocks of texts (Block strings + remove block/sexp comments dune#837, @diml)

• Remove hard-coded knowledge of ppx_driver and
  ocaml-migrate-parsetree when using a dune file (Generic ppx driver dune#576, @diml)

• Make the output of Dune slightly more deterministic when run from
  inside Dune (Make the output of Dune deterministic in tests dune#855, @diml)

• Simplify quoting behavior of variables. All values are now multi-valued and
  whether a multi valued variable is allowed is determined by the quoting and
  substitution context it appears in. (Remove Concat_or_split dune#849, fix Quoting problem when there are several targets dune#701, @rgrinberg)

• Fix documentation generation for private libraries. (Refactor odoc generation and fix the private doc alias dune#864, fix odoc doesn't get called properly dune#856,
  @rgrinberg)

• Use Marshal to store digest and incremental databases. This improves the
  speed of 0 rebuilds. (Use Marshal for the incremental and digest databases dune#817, @diml)

• Allow setting environment variables in findlib.conf for cross compilation
  contexts. (Allow setting ENV vars in findlib.conf per toolchain dune#733, @rgrinberg)

• Add a link_deps field to executables, to specify link-time dependencies
  like version scripts. (Add link_deps field dune#879, fix Link-time dependencies dune#852, @emillon)

• Rename files_recursively_in to source_tree to make it clearer it
  doesn't include generated files (Rename files_recursively_in to source_tree dune#899, fix files_recuirsively_in -> source_tree dune#843, @diml)

• Present the menhir stanza as an extension with its own version
  (Make the menhir stanza an extension dune#901, @diml)

• Improve the syntax of flags in (pps ...). Now instead of (pps (ppx1 -arg1 ppx2 (-foo x))) one should write (pps ppx1 -arg ppx2 -- -foo x) which looks nicer (Improve the syntax of ppx rewriters and flags dune#910, @diml)

• Make (diff a b) ignore trailing cr on Windows and add (cmp a b) for
  comparing binary files (Make diffing work on Windows dune#904, fix Make diffing work better on Windows dune#844, @diml)

• Make dev the default build profile (Make --dev the default dune#920, @diml)

• Version dune-workspace and ~/.config/dune/config files (Version dune-workspace and ~/.config/dune/config files dune#932, @diml)

• Add the ability to build an alias non-recursively from the command
  line by writing @@alias (Add a default alias dune#926, @diml)

• Add a special default alias that defaults to (alias_rec install)
  when not defined by the user and make @@default be the default
  target (Add a default alias dune#926, @diml)

• Forbid #require in dune files in OCaml syntax (Forbid #require in dune files in OCaml syntax dune#938, @diml)

• Add %{profile} variable. (Forbid #require in dune files in OCaml syntax dune#938, @rgrinberg)

• Do not require opam-installer anymore (Do not use opam-installer to copy files dune#941, @diml)

• Add the lib_root and libexec_root install sections (Add the lib_root and libexec_root install sections dune#947, @diml)

• Rename path:file to dep:file (Rename path variables in dune files dune#944, @emillon)

• Remove path-no-dep:file (Remove path-no-dep dune#948, @emillon)

• Adapt the behavior of dune subst for dune projects (Adapt the behavior of dune subst for dune projects dune#960, @diml)

• Add the lib_root and libexec_root sections to install stanzas
  (Add the lib_root and libexec_root install sections dune#947, @diml)

• Add a Configurator.V1.Flags module that improves the flag reading/writing
  API ([WIP] configurator: add write_lines function dune#840, @avsm)

• Add a tests stanza that simlpified defining regular and expect tests
  (Test Stanza Proposal dune#822, @rgrinberg)

• Change the subst subcommand to lookup the project name from the
  dune-project whenever it's available. (Adapt the behavior of dune subst for dune projects dune#960, @diml)

• The subst subcommand no longer looks up the root workspace. Previously this
  detection would break the command whenever -p wasn't passed. (Adapt the behavior of dune subst for dune projects dune#960, @diml)

• Add a # DUNE_GEN in META template files. This is done for consistency with
  # JBUILDER_GEN. (Rename JBUILDER_GEN to DUNE_GEN dune#958, @rgrinberg)

• Rename the following variables in dune files:
  SCOPE_ROOT to project_root
@ to targets
< to first-dep
^ to deps
  (Rename SCOPE_ROOT, ${@}, ${^}, etc. dune#957, @rgrinberg)

• Rename ROOT to workspace_root in dune files (Rename %{root} to %{workspace_root} dune#993, @diml)

• Lowercase all built-in %{variables} in dune files (Lowercase builtin vars dune#956, @rgrinberg)

• New syntax for naming dependencies: (deps (:x a b) (:y (glob_files *.c*))).
  This replaces the use for ${<} in dune files. (Syntax for naming dependencies dune#950, @diml, @rgrinberg)

• Fix detection of dynamic cycles, which in particular may appear when
  using (package ..) dependencies (Fix detection of dynamic cycles dune#988, @diml)

[kit-ty-kate]

There are some failures already. The main cause seems to be this (I haven't checked everything though):

#=== ERROR while compiling sexplib.v0.10.0 ====================================#
# context              2.0.0~rc3 | linux/x86_64 | ocaml-base-compiler.4.06.1 | git+file:///home/opam/opam-repository
# path                 ~/.opam/4.06/.opam-switch/build/sexplib.v0.10.0
# command              ~/.opam/4.06/bin/jbuilder build -p sexplib -j 71
# exit-code            1
# env-file             ~/.opam/log/sexplib-18-8262d6.env
# output-file          ~/.opam/log/sexplib-18-8262d6.out
### output ###
#           sh src/pre_sexp.pp.ml (exit 126)
# (cd _build/default && /bin/sh -c '${PA_CPP} src/pre_sexp.ml') > _build/default/src/pre_sexp.pp.ml
# /bin/sh: 1: src/pre_sexp.ml: Permission denied

[ghost]

Indeed, I fixed it in master. Is there a way to grep through the logs? To see if there are other failures before I update the release

[kit-ty-kate]

There are still some errors:

#=== ERROR while compiling kafka.0.3 ==========================================#
# context              2.0.0~rc3 | linux/x86_64 | ocaml-base-compiler.4.06.1 | git+file:///home/opam/opam-repository
# path                 ~/.opam/4.06/.opam-switch/build/kafka.0.3
# command              ~/.opam/4.06/bin/jbuilder build -p kafka . -j 71 @install
# exit-code            1
# env-file             ~/.opam/log/kafka-18-ed9ebe.env
# output-file          ~/.opam/log/kafka-18-ed9ebe.out
### output ###
# Don't know how to build .

#=== ERROR while compiling opam-core.2.0.0~rc =================================#
# context              2.0.0~rc3 | linux/x86_64 | ocaml-base-compiler.4.06.1 | git+file:///home/opam/opam-repository
# path                 ~/.opam/4.06/.opam-switch/build/opam-core.2.0.0~rc
# command              /usr/bin/make opam-core.install
# exit-code            2
# env-file             ~/.opam/log/opam-core-18-e73fd7.env
# output-file          ~/.opam/log/opam-core-18-e73fd7.out
### output ###
# jbuilder build  -p opam-core opam-core.install
#        ocaml (internal)
# fatal: Not a git repository (or any of the parent directories): .git
#        ocaml src/ocaml-flags-standard.sexp (exit 2)
# (cd _build/default/src && /home/opam/.opam/4.06/bin/ocaml ../shell/subst_var.ml CONF_OCAMLFLAGS '' ${!^}) > _build/default/src/ocaml-flags-standard.sexp
# Exception: Sys_error "${!^}: No such file or directory".
#        ocaml src/core/opamCoreConfig.ml (exit 2)
# (cd _build/default/src/core && /home/opam/.opam/4.06/bin/ocaml ../../shell/subst_var.ml DEVELOPER false ${!^}) > _build/default/src/core/opamCoreConfig.ml
# Exception: Sys_error "${!^}: No such file or directory".
#        ocaml src/core/opamVersion.ml (exit 2)
# (cd _build/default/src/core && /home/opam/.opam/4.06/bin/ocaml ../../shell/subst_var.ml PACKAGE_VERSION <error> ${!^}) > _build/default/src/core/opamVersion.ml
# Exception: Sys_error "${!^}: No such file or directory".
# Makefile:93: recipe for target 'opam-core.install' failed
# make: *** [opam-core.install] Error 1

#=== ERROR while compiling reed-solomon-erasure.1.0.1 =========================#
# context              2.0.0~rc3 | linux/x86_64 | ocaml-base-compiler.4.06.1 | git+file:///home/opam/opam-repository
# path                 ~/.opam/4.06/.opam-switch/build/reed-solomon-erasure.1.0.1
# command              ~/.opam/4.06/bin/jbuilder build
# exit-code            1
# env-file             ~/.opam/log/reed-solomon-erasure-18-ff8c14.env
# output-file          ~/.opam/log/reed-solomon-erasure-18-ff8c14.out
### output ###
# File "dune-project", line 1, characters 11-14:
# Error: Version 0.1 of dune is not supported.
# Supported versions:
# - 0.0
# - 1.0

#=== ERROR while compiling yaml.0.1.0 =========================================#
# context              2.0.0~rc3 | linux/x86_64 | ocaml-base-compiler.4.06.1 | git+file:///home/opam/opam-repository
# path                 ~/.opam/4.06/.opam-switch/build/yaml.0.1.0
# command              ~/.opam/4.06/bin/jbuilder build -p yaml -j 71
# exit-code            1
# env-file             ~/.opam/log/yaml-18-50b255.env
# output-file          ~/.opam/log/yaml-18-50b255.out
### output ###
#           cc vendor/parser.o
# parser.c: In function 'yaml_parser_parse_block_sequence_entry':
# parser.c:762:21: warning: variable 'dummy_mark' set but not used [-Wunused-but-set-variable]
#          yaml_mark_t dummy_mark;     /* Used to eliminate a compiler warning. */
#                      ^~~~~~~~~~
# parser.c: In function 'yaml_parser_parse_block_mapping_key':
# parser.c:872:21: warning: variable 'dummy_mark' set but not used [-Wunused-but-set-variable]
#          yaml_mark_t dummy_mark;     /* Used to eliminate a compiler warning. */
#                      ^~~~~~~~~~
# parser.c: In function 'yaml_parser_parse_flow_sequence_entry':
# parser.c:955:17: warning: variable 'dummy_mark' set but not used [-Wunused-but-set-variable]
#      yaml_mark_t dummy_mark;     /* Used to eliminate a compiler warning. */
#                  ^~~~~~~~~~
# parser.c: In function 'yaml_parser_parse_flow_mapping_key':
# parser.c:1107:17: warning: variable 'dummy_mark' set but not used [-Wunused-but-set-variable]
#      yaml_mark_t dummy_mark;     /* Used to eliminate a compiler warning. */
#                  ^~~~~~~~~~
#           cc vendor/api.o
# api.c: In function 'yaml_document_delete':
# api.c:1121:7: warning: variable 'context' set but not used [-Wunused-but-set-variable]
#      } context;
#        ^~~~~~~
#           cc vendor/emitter.o
# emitter.c: In function 'yaml_emitter_write_plain_scalar':
# emitter.c:28:6: warning: value computed is not used [-Wunused-value]
#       && ((emitter->line_break == YAML_CR_BREAK ?                                \
#       ^
# emitter.c:56:11: note: in expansion of macro 'PUT_BREAK'
#           (PUT_BREAK(emitter),                                                   \
#            ^~~~~~~~~
# emitter.c:1931:18: note: in expansion of macro 'WRITE_BREAK'
#              if (!WRITE_BREAK(emitter, string)) return 0;
#                   ^~~~~~~~~~~
# emitter.c: In function 'yaml_emitter_write_single_quoted_scalar':
# emitter.c:28:6: warning: value computed is not used [-Wunused-value]
#       && ((emitter->line_break == YAML_CR_BREAK ?                                \
#       ^
# emitter.c:56:11: note: in expansion of macro 'PUT_BREAK'
#           (PUT_BREAK(emitter),                                                   \
#            ^~~~~~~~~
# emitter.c:1988:18: note: in expansion of macro 'WRITE_BREAK'
#              if (!WRITE_BREAK(emitter, string)) return 0;
#                   ^~~~~~~~~~~
# emitter.c: In function 'yaml_emitter_write_literal_scalar':
# emitter.c:28:6: warning: value computed is not used [-Wunused-value]
#       && ((emitter->line_break == YAML_CR_BREAK ?                                \
#       ^
# emitter.c:56:11: note: in expansion of macro 'PUT_BREAK'
#           (PUT_BREAK(emitter),                                                   \
#            ^~~~~~~~~
# emitter.c:2251:18: note: in expansion of macro 'WRITE_BREAK'
#              if (!WRITE_BREAK(emitter, string)) return 0;
#                   ^~~~~~~~~~~
# emitter.c: In function 'yaml_emitter_write_folded_scalar':
# emitter.c:28:6: warning: value computed is not used [-Wunused-value]
#       && ((emitter->line_break == YAML_CR_BREAK ?                                \
#       ^
# emitter.c:56:11: note: in expansion of macro 'PUT_BREAK'
#           (PUT_BREAK(emitter),                                                   \
#            ^~~~~~~~~
# emitter.c:2300:18: note: in expansion of macro 'WRITE_BREAK'
#              if (!WRITE_BREAK(emitter, string)) return 0;
#                   ^~~~~~~~~~~
#           cc vendor/scanner.o
# scanner.c: In function 'yaml_parser_decrease_flow_level':
# scanner.c:1189:23: warning: variable 'dummy_key' set but not used [-Wunused-but-set-variable]
#      yaml_simple_key_t dummy_key;    /* Used to eliminate a compiler warning. */
#                        ^~~~~~~~~
#           cc vendor/dllyaml_c_stubs.so (exit 1)
# (cd _build/default/vendor && /usr/bin/cc -shared -o dllyaml_c_stubs.so ${!^})
# cc: error: ${!^}: No such file or directory
# cc: fatal error: no input files
# compilation terminated.
#           ar vendor/libyaml_c_stubs.a (exit 1)
# (cd _build/default/vendor && /usr/bin/ar rcs libyaml_c_stubs.a ${!^})
# /usr/bin/ar: ${!^}: No such file or directory

[rgrinberg]

The kafka thing is old. It was an error with the current jbuilder. I fixed the build instructions in kafka's master branch but it still needs a release. @didier-wenzek another ping :)

The maintainer of reed-solomon-erasure needs to update his package. He is relying on a detail of dune that never made it to any release. The 0.1 version of the dune language never made it to any releases.

@diml I thought we couldn't find any sightings of bang patterns? In any case, I'll bring em back. It's a tiny fix to maintain the compat in jbuild files.

[ghost]

@rgrinberg I don't think that's worth it, only two projects are affected and they both already have newer releases that are not affected. I added a conflit with dune to reed-solomon-erasure, the opam-*.*~rc packages and yaml.0.1.0.

[didier-wenzek]

Indeed, I did not updated the Kafka binding since months. And I missed many discussions. => I will catch up asap.

…

----- Mail original ----- De: "Rudi Grinberg" <notifications@github.com> À: "ocaml/opam-repository" <opam-repository@noreply.github.com> Cc: "Didier Wenzek" <didier.wenzek@acidalie.com>, "Mention" <mention@noreply.github.com> Envoyé: Mardi 10 Juillet 2018 23:28:51 Objet: Re: [ocaml/opam-repository] [new release] dune (1.0.0) (#12323) The kafka thing is old. It was an error with the current jbuilder. I fixed the build instructions in kafka's master branch but it still needs a release. @didier-wenzek another ping :) The maintainer of reed-solomon-erasure needs to update his package. He is relying on a detail of dune that never made it to any release. The 0.1 version of the dune language never made it to any releases. @diml I thought we couldn't find any sightings of bang patterns? In any case, I'll bring em back. It's a tiny fix to maintain the compat in jbuild files. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .

[kit-ty-kate]

I can spot only one error remaining:

#=== ERROR while compiling ocamlformat.0.4 ====================================#
# context              2.0.0~rc3 | linux/x86_64 | ocaml-base-compiler.4.06.1 | git+file:///home/opam/opam-repository
# path                 ~/.opam/4.06/.opam-switch/build/ocamlformat.0.4
# command              ~/.opam/4.06/bin/jbuilder build -p ocamlformat -j 71
# exit-code            1
# env-file             ~/.opam/log/ocamlformat-18-5005e5.env
# output-file          ~/.opam/log/ocamlformat-18-5005e5.out
### output ###
# File "_build/.jbuilds/default/src/jbuild", line 7, characters 4-34:
# Error: path outside the workspace: ../../../.git/index from default/src

(same for ocamlformat.0.3)
If it's not related to dune and if it only requires a constraint to be fixed, it would be better to do that in a separate PR to avoid taking some more resources/time

[rgrinberg]

So the error message is a bit sucky because you cannot tell if the paths are local to the build dir or your source dir but the error is essentially write. If you're inside a context's build dir, you cannot it with ../. So the build of ocamlformat would be broken if you set --build-dir. I recommend adding a conflict and moving on...

[kit-ty-kate]

Fair enough. Let's do that!
Thanks a lot!

[didier-wenzek]

@rgrinberg: I release your fixes to the dune config in the 0.4 release of the kafka bindings (in February). Should I patch & release the previous one? I'm not sure it worth the value. But I can do it if this hampers the release of dune.

[rgrinberg]

I dont think it's worth it. It's your call however. Thanks for the release.

…

On Wed, Jul 11, 2018, 9:48 PM Didier Wenzek ***@***.***> wrote: @rgrinberg <https://github.com/rgrinberg>: I release your fixes to the dune config in the 0.4 release of the kafka bindings (in February). Should I patch & release the previous one? I'm not sure it worth the value. But I can do it if this hampers the release of dune. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12323 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIe-5n1-uHPn6l0Erhou4f2ZOQo-Munks5uFlZwgaJpZM4VJoO0> .

[AltGr]

Please don't do this, use a multi-line string instead. Or better, let opam reformat it, it knows about the terminal width. This actually prints 2 messages, and is e.g. inconsistent with the use of conditional filters.