Bootstrap Opam with OCaml 4.11.1 #4242

MisterDA · 2020-06-23T15:53:03Z

This PR builds on #4234 to fix the little things that are missing and completely replaces 4.07.1 with 4.10.0 for bootstrapping.
Updating findlib to 1.8.1 is mandatory because 1.8.0 checks for the -vmthread flag of ocamlc which has been removed in 4.10.0.
EDIT: I removed the 'flexdll appveyor patch' in favor of using the latest flexdll commit available at the time of writing.
I have also updated and expanded Travis test matrix.

dra27 · 2020-06-24T14:11:29Z

shell/bootstrap-ocaml.sh

-URL=`sed -ne 's/URL_ocaml *= *//p' ../src_ext/Makefile | tr -d '\r'`
-MD5=`sed -ne 's/MD5_ocaml *= *//p' ../src_ext/Makefile | tr -d '\r'`
-V=`echo ${URL}| sed -e 's|.*/\([^/]*\)\.tar\.gz|\1|'`
-FV_URL=`sed -ne 's/URL_flexdll *= *//p' ../src_ext/Makefile | tr -d '\r'`
-FLEXDLL=`echo ${FV_URL}| sed -e 's|.*/\([^/]*\)|\1|'`
+URL=$(sed -ne 's/URL_ocaml *= *//p' ../src_ext/Makefile | tr -d '\r')
+MD5=$(sed -ne 's/MD5_ocaml *= *//p' ../src_ext/Makefile | tr -d '\r')
+V=$(echo ${URL}| sed -e 's|.*/\([^/]*\)\.tar\.gz|\1|')
+FV_URL=$(sed -ne 's/URL_flexdll *= *//p' ../src_ext/Makefile | tr -d '\r')
+FLEXDLL=$(echo ${FV_URL}| sed -e 's|.*/\([^/]*\)|\1|')


There is still the odd Solaris user out there, and the default sh on still just-about supported Solaris doesn't recognise $(). These need to stay as they are (the ones in the Windows-specific part don't matter).

oh wow! I did not know that. They’re not using xpg4 or xpg6?

I don't know the full detail - I just know that Solaris 10 uses an incredibly old sh by default (and IIRC a crazy patch too). I'm all for $() but for the most part, I tend to leave existing uses of backtick alone (you can still shoot yourself in the foot: cf. #3699 (comment))

What I’m seeing is that Solaris provides multiples implementations of the tools, each following a standard.

An application that wants to use standard-conforming utilities must set the PATH (sh(1) or ksh(1)) or path (csh(1)) environment variable to specify the directories listed below in the order specified to get the appropriate utilities:

POSIX.1–2001, SUSv3

/usr/xpg6/bin

/usr/xpg4/bin

/usr/bin

directory containing binaries for your compiler

other directories containing binaries needed by the application

https://docs.oracle.com/cd/E86824_01/html/E54776/xpg-5.html

We could set the PATH to use the latest stuff 😄

dra27 · 2020-06-24T14:13:31Z

You do actually need to configure the Windows build 😉 This is working for me:

diff --git a/shell/bootstrap-ocaml.sh b/shell/bootstrap-ocaml.sh
index 3cf90659..13aea4fd 100755
--- a/shell/bootstrap-ocaml.sh
+++ b/shell/bootstrap-ocaml.sh
@@ -51,12 +51,23 @@ PATH_PREPEND=
 LIB_PREPEND=
 INC_PREPEND=
 if [ -n "$1" -a -n "${COMSPEC}" -a -x "${COMSPEC}" ] ; then
+  case "$(uname -m)" in
+    'i686')
+      BUILD=i686-pc-cygwin
+    ;;
+    'x86_64')
+      BUILD=x86_64-pc-cygwin
+    ;;
+  esac
   case "$1" in
-    "mingw"|"mingw64")
-      BUILD=$1
+    "mingw")
+      HOST=i686-w64-mingw32
+    ;;
+    "mingw64")
+      HOST=x86_64-w64-mingw32
     ;;
     "msvc")
-      BUILD=$1
+      HOST=i686-pc-windows
       if ! command -v ml > /dev/null ; then
         eval `${ROOT}../../shell/msvs-detect --arch=x86`
         if [ -n "${MSVS_NAME}" ] ; then
@@ -67,7 +78,7 @@ if [ -n "$1" -a -n "${COMSPEC}" -a -x "${COMSPEC}" ] ; then
       fi
     ;;
     "msvc64")
-      BUILD=$1
+      HOST=x86_64-pc-windows
       if ! command -v ml64 > /dev/null ; then
         eval `${ROOT}../../shell/msvs-detect --arch=x64`
         if [ -n "${MSVS_NAME}" ] ; then
@@ -88,24 +99,24 @@ if [ -n "$1" -a -n "${COMSPEC}" -a -x "${COMSPEC}" ] ; then
       fi

       if [ ${TRY64} -eq 1 ] && command -v x86_64-w64-mingw32-gcc > /dev/null ; then
-        BUILD=mingw64
+        HOST=x86_64-w64-mingw32
       elif command -v i686-w64-mingw32-gcc > /dev/null ; then
-        BUILD=mingw
+        HOST=i686-w64-mingw32
       elif [ ${TRY64} -eq 1 ] && command -v ml64 > /dev/null ; then
-        BUILD=msvc64
+        HOST=x86_64-pc-windows
         PATH_PREPEND=`bash ${ROOT}../../shell/check_linker`
       elif command -v ml > /dev/null ; then
-        BUILD=msvc
+        HOST=i686-pc-windows
         PATH_PREPEND=`bash ${ROOT}../../shell/check_linker`
       else
         if [ ${TRY64} -eq 1 ] ; then
-          BUILD=msvc64
-          BUILD_ARCH=x64
+          HOST=x86_64-pc-windows
+          HOST_ARCH=x64
         else
-          BUILD=msvc
-          BUILD_ARCH=x86
+          HOST=i686-pc-windows
+          HOST_ARCH=x86
         fi
-        eval `${ROOT}../../shell/msvs-detect --arch=${BUILD_ARCH}`
+        eval `${ROOT}../../shell/msvs-detect --arch=${HOST_ARCH}`
         if [ -z "${MSVS_NAME}" ] ; then
           echo "No appropriate C compiler was found -- unable to build OCaml"
           exit 1
@@ -123,9 +134,8 @@ if [ -n "$1" -a -n "${COMSPEC}" -a -x "${COMSPEC}" ] ; then
   PREFIX=`cd .. ; pwd`/ocaml
   WINPREFIX=`echo ${PREFIX} | cygpath -f - -m`
   if [ ${GEN_CONFIG_ONLY} -eq 0 ] ; then
-    sed -e "s|^PREFIX=.*|PREFIX=${WINPREFIX}|" -e "s|/lib|/lib/ocaml|" config/Makefile.${BUILD} > config/Makefile
-    cp config/s-nt.h byterun/caml/s.h
-    cp config/m-nt.h byterun/caml/m.h
+    # --disable-ocamldoc can change to --disable-stdlib-manpages when bumped to 4.11
+    PATH="${PATH_PREPEND}${PREFIX}/bin:${PATH}" Lib="${LIB_PREPEND}${Lib}" Include="${INC_PREPEND}${Include}" ./configure --prefix "$WINPREFIX" --build=$BUILD --host=$HOST --disable-ocamldoc
   fi
   cd ..
   if [ ! -e ${ROOT}${FLEXDLL} ]; then
@@ -136,7 +146,9 @@ if [ -n "$1" -a -n "${COMSPEC}" -a -x "${COMSPEC}" ] ; then
     tar -xzf ${ROOT}../${FLEXDLL}
     rm -rf flexdll
     mv flexdll-* flexdll
-    PATH="${PATH_PREPEND}${PREFIX}/bin:${PATH}" Lib="${LIB_PREPEND}${Lib}" Include="${INC_PREPEND}${Include}" make flexdll world.opt install
+    PATH="${PATH_PREPEND}${PREFIX}/bin:${PATH}" Lib="${LIB_PREPEND}${Lib}" Include="${INC_PREPEND}${Include}" make -j flexdll
+    PATH="${PATH_PREPEND}${PREFIX}/bin:${PATH}" Lib="${LIB_PREPEND}${Lib}" Include="${INC_PREPEND}${Include}" make -j world.opt
+    PATH="${PATH_PREPEND}${PREFIX}/bin:${PATH}" Lib="${LIB_PREPEND}${Lib}" Include="${INC_PREPEND}${Include}" make install
     mkdir -p ../../src_ext
   fi
   OCAMLLIB=${WINPREFIX}/lib/ocaml

MisterDA · 2020-06-24T14:58:23Z

@dra27 there’s something weird in your patch. The line mkdir -p ../../src_ext is not present in the file in master, but it is not marked as added in your patch. Is it a left over from something else?

EDIT: oh, it’s not just that. My guess is that the diff wasn’t made against master. There are maybe other changes that are important. For instance, you have

eval `${ROOT}../../shell/msvs-detect --arch=x86`

but master contains

eval `../../shell/msvs-detect --arch=x86`

dra27 · 2020-06-24T17:38:07Z

Oops, that patch’s against a different branch which supports building 4 bootstrap compilers! It should be sufficient just to ignore the ROOT variable!

MisterDA · 2020-06-25T08:35:17Z

@dra27 an idea why the msvc builds are failing?
And what's happening in Travis with the last failing builds?

dra27 · 2020-06-25T09:11:28Z

This should unlock the Travis failure:

diff --git a/.travis-ci.sh b/.travis-ci.sh
index 9d85a845..f4621b91 100755
--- a/.travis-ci.sh
+++ b/.travis-ci.sh
@@ -192,6 +192,7 @@ EOF
           make world.opt
         fi
         make install
+        rm -rf "ocaml-$OCAML_VERSION.tar.gz"
         echo "LOCAL_OCAML_VERSION=$OCAML_VERSION" > ~/local/versions
         echo -en "travis_fold:end:ocaml\r"
       fi

dra27 · 2020-06-25T09:12:07Z

The msvc failure is more subtle - I'm looking into it

dra27 · 2020-06-25T09:58:28Z

OK, there's a path-blowout in msvs-detect - I'm just working on a fix. Hopefully the upstream maintainer for msvs-tools will merge it quickly 😉

dra27 · 2020-06-25T09:59:53Z

Sorry - that patch I gave you was wrong - it's the directory which needs removing, not the tarball.

dra27 · 2020-06-25T10:00:34Z

i.e.

diff --git a/.travis-ci.sh b/.travis-ci.sh
index 9d85a845..73bc1670 100755
--- a/.travis-ci.sh
+++ b/.travis-ci.sh
@@ -192,6 +192,7 @@ EOF
           make world.opt
         fi
         make install
+        rm -rf "ocaml-$OCAML_VERSION"
         echo "LOCAL_OCAML_VERSION=$OCAML_VERSION" > ~/local/versions
         echo -en "travis_fold:end:ocaml\r"
       fi

MisterDA · 2020-06-25T10:02:29Z

I think that at this point we’re in the directory "ocaml-$OCAML_VERSION" (line 173) so we should cd .. before removing it.

dra27 · 2020-06-25T12:25:27Z

~~Incidentally, and hopefully trivially, this should be 4.10.1~~ (it’s apparently been a long day!)

rjbou · 2020-06-26T14:39:36Z

@MisterDA Please cherry-pick 1b0f4ba as it is reverted in master

MisterDA · 2020-07-07T09:54:09Z

Rebased against master. Some of the changes were integrated when transitioning from 4.07 to 4.09.1 in bootstrap, and some other fixes were integrated independently.

dra27

Various bits and bobs - always worth checking the overall diff after rebasing!

#4260 will fix the transient problems affecting Travis - AppVeyor will fail because you need the upstream fix to caml/domain_state.h header. I'd suggest stalling this therefore until 4.10.1 is released (which I believe should be relatively soon, as it should shortly precede the release of 4.11).

.gitattributes

.travis-ci.sh

.travis.yml

master_changes.md

src_ext/Makefile

> the pattern is not necessary, as the scripts with those extensions > are not executable and are crunched into an OCaml script ocaml#4242 (comment)

MisterDA · 2020-08-17T16:59:19Z

Rebase on master (2.1.0-alpha3) and test OCaml 4.10.1+rc1.

> the pattern is not necessary, as the scripts with those extensions > are not executable and are crunched into an OCaml script ocaml#4242 (comment)

This is because gcc10 defaults to -fcommon which causes compatibility carnage. See ocaml/opam-repository#16583. Co-authored-by: Antonin Décimo <antonin.decimo@gmail.com>

When cloning the repository under Cygwin, CRLF are not set on batch scripts. This is irrelevant for most CIs because the repository is cloned outside of Cygwin, but this can happen to a user building Opam from Cygwin and experimenting with the appveyor_build.cmd script.

> the pattern is not necessary, as the scripts with those extensions > are not executable and are crunched into an OCaml script ocaml#4242 (comment)

dra27

I just pushed one additional commit to action a comment in shell/bootstrap_ocaml.sh. Good to merge once CI comes back confirming that I didn't break the script!

AltGr · 2020-09-09T09:09:35Z

Thanks!

dra27 reviewed Jun 24, 2020

View reviewed changes

MisterDA force-pushed the cold-with-4.10 branch from 39dc5c7 to ad7559c Compare June 24, 2020 15:11

MisterDA force-pushed the cold-with-4.10 branch from ad7559c to 79129e4 Compare June 25, 2020 09:51

MisterDA force-pushed the cold-with-4.10 branch from 79129e4 to f005e46 Compare June 25, 2020 10:07

dra27 mentioned this pull request Jun 25, 2020

Fix path blow-up with environment compiler metastack/msvs-tools#6

Merged

rjbou mentioned this pull request Jun 26, 2020

Cold boostrap back to 4.07.1 #4247

Merged

MisterDA force-pushed the cold-with-4.10 branch from f005e46 to 19be512 Compare June 26, 2020 15:37

MisterDA mentioned this pull request Jun 26, 2020

Compilation failure in domain_state.h for librairies with OCaml ≥ 4.10 ocaml/ocaml#9714

Closed

MisterDA force-pushed the cold-with-4.10 branch from 19be512 to 66c1fcb Compare June 29, 2020 19:34

MisterDA force-pushed the cold-with-4.10 branch from 66c1fcb to 7b0c4f9 Compare July 7, 2020 09:51

dra27 reviewed Jul 7, 2020

View reviewed changes

MisterDA added a commit to MisterDA/opam that referenced this pull request Jul 7, 2020

Don't force LF line endings on zsh script

d722636

> the pattern is not necessary, as the scripts with those extensions > are not executable and are crunched into an OCaml script ocaml#4242 (comment)

MisterDA force-pushed the cold-with-4.10 branch from 7b0c4f9 to 5df6376 Compare July 7, 2020 13:39

MisterDA force-pushed the cold-with-4.10 branch from 5df6376 to 897040d Compare August 17, 2020 16:56

MisterDA added a commit to MisterDA/opam that referenced this pull request Aug 17, 2020

Don't force LF line endings on zsh script

bab0c25

> the pattern is not necessary, as the scripts with those extensions > are not executable and are crunched into an OCaml script ocaml#4242 (comment)

MisterDA changed the title ~~Bootstrap Opam with OCaml 4.10.0~~ Bootstrap Opam with OCaml 4.10 Aug 17, 2020

MisterDA added a commit to MisterDA/opam that referenced this pull request Aug 18, 2020

Don't force LF line endings on zsh script

0627cbe

> the pattern is not necessary, as the scripts with those extensions > are not executable and are crunched into an OCaml script ocaml#4242 (comment)

MisterDA force-pushed the cold-with-4.10 branch from 897040d to 96709d2 Compare August 18, 2020 16:00

MisterDA added a commit to MisterDA/opam that referenced this pull request Sep 2, 2020

Don't force LF line endings on zsh script

2c9f5aa

> the pattern is not necessary, as the scripts with those extensions > are not executable and are crunched into an OCaml script ocaml#4242 (comment)

MisterDA force-pushed the cold-with-4.10 branch from 96709d2 to b8110e0 Compare September 2, 2020 14:53

MisterDA changed the title ~~Bootstrap Opam with OCaml 4.10~~ Bootstrap Opam with OCaml 4.11.1 Sep 2, 2020

MisterDA added a commit to MisterDA/opam that referenced this pull request Sep 3, 2020

Don't force LF line endings on zsh script

98c19df

> the pattern is not necessary, as the scripts with those extensions > are not executable and are crunched into an OCaml script ocaml#4242 (comment)

MisterDA force-pushed the cold-with-4.10 branch from b8110e0 to 13916c0 Compare September 3, 2020 11:31

avsm and others added 2 commits September 3, 2020 13:36

build cold bootstrap with OCaml 4.11.1 to fix gcc10 compilation

33e0f49

This is because gcc10 defaults to -fcommon which causes compatibility carnage. See ocaml/opam-repository#16583. Co-authored-by: Antonin Décimo <antonin.decimo@gmail.com>

travis: fix warnings

aff67d7

MisterDA force-pushed the cold-with-4.10 branch from 13916c0 to 6f067d2 Compare September 3, 2020 13:21

MisterDA added 5 commits September 3, 2020 15:24

travis: bootstrap Opam with 4.11.1

f368400

appveyor: init opam with 4.11.1 switch

d9e9762

Don't force LF line endings on zsh script

152f618

> the pattern is not necessary, as the scripts with those extensions > are not executable and are crunched into an OCaml script ocaml#4242 (comment)

update changes

6f067d2

MisterDA marked this pull request as ready for review September 5, 2020 11:52

Use --disable-stdlib-manpages in bootstrap

b2126fa

dra27 approved these changes Sep 7, 2020

View reviewed changes

AltGr merged commit 7e4b50c into ocaml:master Sep 9, 2020

rjbou added this to the 2.1.0~beta2 milestone Sep 14, 2020

MisterDA deleted the cold-with-4.10 branch April 18, 2023 09:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bootstrap Opam with OCaml 4.11.1 #4242

Bootstrap Opam with OCaml 4.11.1 #4242

MisterDA commented Jun 23, 2020 •

edited

Loading

dra27 Jun 24, 2020 •

edited

Loading

MisterDA Jun 24, 2020

dra27 Jun 24, 2020

MisterDA Jun 24, 2020

dra27 commented Jun 24, 2020

MisterDA commented Jun 24, 2020 •

edited

Loading

dra27 commented Jun 24, 2020

MisterDA commented Jun 25, 2020

dra27 commented Jun 25, 2020

dra27 commented Jun 25, 2020

dra27 commented Jun 25, 2020

dra27 commented Jun 25, 2020

dra27 commented Jun 25, 2020

MisterDA commented Jun 25, 2020

dra27 commented Jun 25, 2020 •

edited

Loading

rjbou commented Jun 26, 2020

MisterDA commented Jul 7, 2020

dra27 left a comment

MisterDA commented Aug 17, 2020

dra27 left a comment

AltGr commented Sep 9, 2020

Bootstrap Opam with OCaml 4.11.1 #4242

Bootstrap Opam with OCaml 4.11.1 #4242

Conversation

MisterDA commented Jun 23, 2020 • edited Loading

dra27 Jun 24, 2020 • edited Loading

Choose a reason for hiding this comment

MisterDA Jun 24, 2020

Choose a reason for hiding this comment

dra27 Jun 24, 2020

Choose a reason for hiding this comment

MisterDA Jun 24, 2020

Choose a reason for hiding this comment

dra27 commented Jun 24, 2020

MisterDA commented Jun 24, 2020 • edited Loading

dra27 commented Jun 24, 2020

MisterDA commented Jun 25, 2020

dra27 commented Jun 25, 2020

dra27 commented Jun 25, 2020

dra27 commented Jun 25, 2020

dra27 commented Jun 25, 2020

dra27 commented Jun 25, 2020

MisterDA commented Jun 25, 2020

dra27 commented Jun 25, 2020 • edited Loading

rjbou commented Jun 26, 2020

MisterDA commented Jul 7, 2020

dra27 left a comment

Choose a reason for hiding this comment

MisterDA commented Aug 17, 2020

dra27 left a comment

Choose a reason for hiding this comment

AltGr commented Sep 9, 2020

MisterDA commented Jun 23, 2020 •

edited

Loading

dra27 Jun 24, 2020 •

edited

Loading

MisterDA commented Jun 24, 2020 •

edited

Loading

dra27 commented Jun 25, 2020 •

edited

Loading