[Flang] Driver hardcodes `lib/libflang_rt.a` path that is incorrect on `lib64` systems, and does not permit multilib

[mgorny]

I'm almost ready to package Flang on Gentoo. The remaining problem is that the driver hardcodes the lib directory for libflang_rt.a:

llvm-project/clang/lib/Driver/ToolChains/CommonArgs.cpp

Lines 1361 to 1370 in 6fde8fe

	void tools::addFortranRuntimeLibraryPath(const ToolChain &TC,
	const llvm::opt::ArgList &Args,
	ArgStringList &CmdArgs) {
	// Default to the <driver-path>/../lib directory. This works fine on the
	// platforms that we have tested so far. We will probably have to re-fine
	// this in the future. In particular, on some platforms, we may need to use
	// lib64 instead of lib.
	SmallString<256> DefaultLibPath =
	llvm::sys::path::parent_path(TC.getDriver().Dir);
	llvm::sys::path::append(DefaultLibPath, "lib");

This is problematic for two reasons:

1. Gentoo is using lib64 for 64-bit libraries on platforms such on amd64, and lib for 32-bit libraries, and installing a 64-bit library into lib is generally an error.
2. The forced use of a single library directory makes cross-compilation / multilib builds impossible. I haven't tested how well Flang works with that yet, but at least a dumb flang --target=i686-pc-linux-gnu hello.f90 seems to work, modulo trying to link a 64-bit runtime.

I've tried fiddling a fair bit, and unfortunately I don't see any good way of getting the correct lib* directory in that code. Perhaps the correct approach would be to include the architecture in the filename the way compiler-rt does — and perhaps move it to clang's resource directory while at it?

[mgorny]

Oh, and for completeness: we are installing the whole LLVM into per-version prefixes, so that different versions can be used simultaneously. So we can't just rely on the linker picking up the library from /usr/lib64 but need Flang to explicitly point it to the right directory.

[llvmbot]

@llvm/issue-subscribers-flang-driver

Author: Michał Górny (mgorny)

I'm almost ready to package Flang on Gentoo. The remaining problem is that the driver hardcodes the `lib` directory for `libflang_rt.a`:

llvm-project/clang/lib/Driver/ToolChains/CommonArgs.cpp

Lines 1361 to 1370 in 6fde8fe

	void tools::addFortranRuntimeLibraryPath(const ToolChain &TC,
	const llvm::opt::ArgList &Args,
	ArgStringList &CmdArgs) {
	// Default to the <driver-path>/../lib directory. This works fine on the
	// platforms that we have tested so far. We will probably have to re-fine
	// this in the future. In particular, on some platforms, we may need to use
	// lib64 instead of lib.
	SmallString<256> DefaultLibPath =
	llvm::sys::path::parent_path(TC.getDriver().Dir);
	llvm::sys::path::append(DefaultLibPath, "lib");

This is problematic for two reasons:

1. Gentoo is using lib64 for 64-bit libraries on platforms such on amd64, and lib for 32-bit libraries, and installing a 64-bit library into lib is generally an error.
2. The forced use of a single library directory makes cross-compilation / multilib builds impossible. I haven't tested how well Flang works with that yet, but at least a dumb flang --target=i686-pc-linux-gnu hello.f90 seems to work, modulo trying to link a 64-bit runtime.

I've tried fiddling a fair bit, and unfortunately I don't see any good way of getting the correct lib* directory in that code. Perhaps the correct approach would be to include the architecture in the filename the way compiler-rt does — and perhaps move it to clang's resource directory while at it?

[tarunprabhu]

It looks like gcc installs libgfortran and libstdc++ to its equivalent of a resource directory. It seems like a reasonable place to put it. But we should try to be consistent with libc++, which is the closest equivalent in the LLVM project.

[mgorny]

Is flang_rt compatible across different Flang versions? I.e. is it fine to, say, use libflang_rt.a from Flang main with older version of Flang compiler?

[tarunprabhu]

Is flang_rt compatible across different Flang versions? I.e. is it fine to, say, use libflang_rt.a from Flang main with older version of Flang compiler?

I am not sure. Perhaps someone more knowledgeable about the runtime can give you a more definitive answer, @klausler @kiranchandramohan @jeanPerier @clementval

[jeanPerier]

Is flang_rt compatible across different Flang versions? I.e. is it fine to, say, use libflang_rt.a from Flang main with older version of Flang compiler?

There is no such guarantee. The runtime API do evolve from time to time and some may be removed/modified. The Fortran runtime is not a "standard library", it is tightly connected to the compiler, so it is more similar to compiler-rt in terms of compiler/runtime interaction, even though its features are comparable to libc++.

[mgorny]

Yeah, so I think clang resource dir would work best. FWICS the standalone runtimes build already attempts installing it somewhere like that, except that I don't think the driver actually checks that location.

[Meinersbur]

At the moment flang does not support cross-compilation. Even if flang accepts --target, it should not be used, there are many places where Flang assumes that the target system is the same as the host. Making flang-rt use of the LLVM_ENABLE_RUNTIMES system in #110217 was just the first step. Also, Flang currently only supports 64-bit, so multilib is somewhat pointless.

#110217 (LLVM_ENABLE_RUNTIMES=flang-rt) also changed the location of flang_rt.runtime.lib, compared to the old way FLANG_INCLUDE_RUNTIME=ON. I originally used CMAKE_INSTALL_LIBDIR which seemed the platform-correct way to do. Unfortunately, it did now work, see the discussion at #110217 (comment).

I see you also opened #127345, the library where libflang_rt.runtime.a and the driver searches for them must be in-sync, ideally also with clang/compiler-rt. Unfortantely, it's total chaos (together with LLVM_ENABLE_PER_TARGET_RUNTIME_DIR). There are two lib in the path, there is CMAKE_INSTALL_LIBDIR, CLANG_INSTALL_LIBDIR_BASENAME, CLANG_LIBDIR_SUFFIX, LLVM_LIBDIR_SUFFIX, ... and I have no idea what is the correct way to do it.

[Meinersbur]

Is flang_rt compatible across different Flang versions? I.e. is it fine to, say, use libflang_rt.a from Flang main with older version of Flang compiler?

There is no such guarantee. The runtime API do evolve from time to time and some may be removed/modified. The Fortran runtime is not a "standard library", it is tightly connected to the compiler, so it is more similar to compiler-rt in terms of compiler/runtime interaction, even though its features are comparable to libc++.

To elaborate, this is the reason why libflang_rt.runtime.a is put into Clang's resource dir (<clang-executable>/../lib/clang/<version>/lib/<target-triple>)

[DavidTruby]

Just to add a bit of context; flang does support cross compilation in practice between "similar" targets, ie 64 bit targets where the sizes of the types are the same. I've used it myself to cross compile from x86_64-linux to aarch64-linux, as well as from aarch64-linux to aarch64-win32, and it has worked well in those cases. It's certainly under tested though, and you have to hope that the platforms you're using have the same eg integer widths.
This could certainly be better tested, and we should look at fixing the code that assumes the build platform has the same type widths as the target platform.

I'm also not aware of anyone having tested flang on any kind of 32 bit architecture. It's almost certainly impossible to do a build of flang itself in 32 bit because of memory requirements; I suppose you could cross compile it (it's just a C++ programme after all) but I'm not aware of that having been tested by anyone.

I don't think any of this should be a mark against a distributor's desire to have lib only contain 32 bit libraries and lib64 contain only 64 bit binaries though. It does sound to me like taking the same approach as clang takes for compiler-rt would be the best path forward. Would we be able to invent a flang resource dir using the same logic as clang uses for its equivalent, and use that as the location for libflang-rt?

[Meinersbur]

Just to add a bit of context; flang does support cross compilation in practice between "similar" targets, ie 64 bit targets where the sizes of the types are the same. I've used it myself to cross compile from x86_64-linux to aarch64-linux, as well as from aarch64-linux to aarch64-win32, and it has worked well in those cases. It's certainly under tested though, and you have to hope that the platforms you're using have the same eg integer widths. This could certainly be better tested, and we should look at fixing the code that assumes the build platform has the same type widths as the target platform.

Not just type widths, but also alignment rules, which causes structs to have different sizes etc.

x86_64 and aarch64 might mostly work, since both are LP64 (sizeof(long)==8) and probably the same alignments, but win32 uses LLP64 (sizeof(long)==4) and WILL cause problems:
https://github.com/llvm/llvm-project/blob/main/flang/include/flang/Optimizer/CodeGen/DescriptorModel.h#L91-L94

I don't think we should claim any level of "supported" as long as there are such known miscompilation issues. Otherwise users/clients will use it, their program crashes unexpectedly, nobody can reproduce it until we're told that they were cross-compiling, and rightfully blame us for it. At least I want to tell them they were not supposed to do cross-compilation.

[clementval]

For 32 bits the doc is pretty clear.

Note that flang is not supported on 32 bit CPUs.

For cross compilation to other 64bits target nobody ever claimed that is was implemented and/or tested so I would not use it in production until somebody spend the time to make the adjustment to make it work 100%.