Better support for PIC

[MatthewFluet]

There are a number of short- and long-term improvements to better support PIC:

• add a -native-pic {false,true} compile-time option
• determine default PIC or non-PIC for target at configure time

[xyproto]

There are issues building mlton on 64-bit Arch Linux. Even when setting CFLAGS to include -fPIC in all Makefiles, and exporting CFLAGS to include -fPIC before building with `make there are several errors like these:

/usr/bin/ld: /tmp/filepHqzpc.o: relocation R_X86_64_32S against `.data' can not be used when making a shared object; recompile with -fPIC

I tried:

• Building in a chroot and not.
• Having the binary mlton package installed at build-time, and not.
• Compiling both the latest release tag and the latest commit.

What I want to achieve is to update the official mlton package in the [community] repo to include PIC support.

[MatthewFluet]

Have you tried the very latest commit (#190)? Prior to that commit, the amd64 codegen would not generate position independent code for Linux.

And, this really is the limitation of trying to determine the PI/non-PI for the target based on the architecture/os pair. As I understand it, systems (the default behavior of the c compiler and of the linker) can be configured in many different ways. In particular, whether the default for the c compiler is to generate PIC or not and whether the linker accepts (either silently or with a non-fatal warning) mixing of PIC and non-PIC code. I think that the default on some Linux distributions (Ubuntu and Fedora) is for gcc to generate PIC, but the linker will silently allow non-PIC object files to be linked into an executable (and the resulting executable will not be PI); that's the scenario that allows MLton generated executables on those systems to work before and after merging #190. Maybe Arch is different and the linker is generating a fatal error when linking non-PIC object files into an executable; in which case, you might get a different behavior after merging #190.

You might also make progress by forcing the use of the C codegen, which would have the intermediate object files created according to the default gcc behavior. You could try make COMPILE_ARGS='-codegen c'.

[mkoloberdin]

@xyproto it is possible to bootstrap MLton with SML-NJ. I am trying to do so now. (compilation takes a while).
Apparently all you need to do is add a make smlnj-mlton (make -C mlton COMPILE_ARGS="-codegen c -cc-opt '-fPIC'" -j1 smlnj-mlton as per your PKGBUILD) before the regular make step. (which then will use the smlnj-built mlton binary to build mlton again)
Once you get the working mlton installed (or once the working binary package is in Arch's repository), you can remove the make smlnj-mlton step from PKGBUILD (and remove smlnj from makedepends), bump the pkgrel and rebuild mlton without smlnj.

Here is what I've done so far:

# Need to do this after installing smlnj if you did not have it installed before logging in / starting a session:
source /etc/profile.d/smlnj.sh
# copied from your latest PKGBUILD:
sed 's/_BSD_SOURCE/_DEFAULT_SOURCE/g' -i mlton/runtime/cenv.h
find mlton -name Makefile -type f -exec setconf {} CFLAGS "$CFLAGS -fPIC" \;
# added to bootstrap with sml-nj:
make -C mlton COMPILE_ARGS="-codegen c -cc-opt '-fPIC'" -j1 smlnj-mlton
# from your PGKBUILD:
make -C mlton COMPILE_ARGS="-codegen c -cc-opt '-fPIC'" -j1 all-no-docs
# ^^^this is taking a while, waiting for it to finish...
# here is what it echo-ed among other things:
#
# MLton starting
# MLton 20170725.130317-g159190284-dirty
#  created this file on Sun Aug  6 20:40:46 2017.
#
# So it seems to be using the smlnj-built mlton rather than the stock one I have installed on my system.

Background info:

1. http://mlton.org/SelfCompiling

Compiling with SML/NJ

2. MLton's Makefile: search for "nj"

[mkoloberdin]

@xyproto here is a patch for your PKGBUILD. (though it won't work if smlnj is not installed and source /etc/profile.d/smlnj.sh is not done beforehand)

diff --git a/PKGBUILD b/PKGBUILD
index b97db69..4488f08 100644
--- a/PKGBUILD
+++ b/PKGBUILD
@@ -6,13 +6,14 @@
 
 pkgname=mlton
 pkgver=20170725
-pkgrel=1
+pkgrel=2
 pkgdesc='Whole-program, optimizing Standard ML compiler'
 arch=('x86_64' 'i686')
 url='http://mlton.org/'
 license=('BSD' 'MIT' 'LGPL')
 depends=('gmp')
 makedepends=('git' 'setconf' 'mlton')
+makedepends_x86_64=('smlnj')
 options=('staticlibs')
 source=("git+https://github.com/MLton/mlton#commit=159190284e1268fded189104705939fe3b691b12")
 md5sums=('SKIP')
@@ -23,9 +24,13 @@ prepare() {
 }
 
 build() {
-  # Latest master does not build on x86_64. Older releases does not build on x86_64. WIP
-
   export CFLAGS="$CFLAGS -fPIC"
+  
+  # Bootstrap with SML-NJ on x86_64
+  if [ $arch = 'x86_64' ]
+  then
+    make -C mlton COMPILE_ARGS="-codegen c -cc-opt '-fPIC'" -j1 smlnj-mlton
+  fi
   make -C mlton COMPILE_ARGS="-codegen c -cc-opt '-fPIC'" -j1 all-no-docs
   #make -C mlton COMPILE_ARGS="-codegen c" all-no-docs
 }

[xyproto]

Thank you @mkoloberdin! I will try compiling with smlnj.

[mkoloberdin]

@xyproto this approach seems to be working. I've compiled mlton this way successfully. Then compiled Ur/Web with it. ( https://aur.archlinux.org/packages/urweb/ )

[xyproto]

@mkoloberdin It worked. Released new packages to [community]. Thanks!

[mkoloberdin]

@xyproto Great! You might want to do the following as well as I have mentioned above (so that when you upgrade the package next time you do not forget to remove the now unnecessary smlnj dependency/bootstrapping):

Once you get the working mlton installed (or once the working binary package is in Arch's repository), you can remove the make smlnj-mlton step from PKGBUILD (and remove smlnj from makedepends), bump the pkgrel and rebuild mlton without smlnj.

Also, you've got my email wrong. If you have to add it, pull it from the maintainer line here: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=urweb

[xyproto]

Thanks, I will consider commenting out the lines related to smlnj (might come in handy next time there is a binary incompatible upgrade). I'll also update your e-mail.

[MatthewFluet]

@mkoloberdin @xyproto Some comments:

• It shouldn't be necessary to pass COMPILE_ARGS="-codegen c -cc-opt '-fPIC'" to make smlnj-mlton; those COMPILE_ARGS are only meaningful to MLton.
• For the next bootstrap step make -C mlton COMPILE_ARGS="-codegen c -cc-opt '-fPIC'" -j1 all-no-docs, you are asking a SML/NJ built MLton to use the C codegen to build MLton. That is acceptable, but note that a program built with the C codegen is generally a little less efficient than one built with the native codegen. So, the final installed mlton will be less efficient than it could be. On the other hand, because the SML/NJ built MLton is so slow, it is probably beneficial to use the C codegen, which has lower compile times than the native codegen.
• I'm also not sure that you need -cc-opt '-fPIC' once you have the current sources of MLton built (whether with SML/NJ or via some other means). In particular, note that COMPILE_ARGS only affects the build of mlton (the compiler proper); it is not used to build the other tools (mllex, mlyacc, mlnlffigen). Assuming that the whole build completes without errors, then that means that those tools were compiled by the SML/NJ built MLton using the native codegen and without an explicit -fPIC.

You might try the following sequence:

### Build MLton with SML/NJ
make -C mlton -j1 smlnj-mlton
### An SML/NJ built MLton is slow, so use C codegen (less time in SML/NJ, more time in gcc)
make -C mlton -j1 COMPILE_ARGS='-codegen c' all-no-docs
### Clean the C-codgen built MLton
make -C mlton/mlton clean
### Use the C-codgen built MLton to build a native-codegen built MLton
make -C mlton -j1 all-no-docs

[xyproto]

@mkoloberdin Updated your e-mail address.

Updated the Arch Linux mlton package to only build with the previous version of mlton that was boostrapped with smlnj. There does not seem to be any PIC related issues with mlton on x86_64 on Arch Linux anymore.

👍