flang mingw-w64 build fails with string table overflow, file too big

[maveric2]

Hi community,

I'm able to compile clang;clang-tools-extra;compiler-rt;lld succefully. However if I add flang to the list I bump into the following problem:

FAILED: tools/flang/lib/Lower/CMakeFiles/obj.FortranLower.dir/PFTBuilder.cpp.obj
D:\csd_ntly\coside-3.2RC1_mingw-w64\tools\gcc-9.4.0\bin\g++.exe -DFLANG_LITTLE_ENDIAN=1 -D_FILE_OFFSET_BITS=64 -D__STDC_CONSTANT
_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -IC:/clang_man/llvm-project/build/tools/flang/lib/Lower -IC:/clang_man/llvm
-project/flang/lib/Lower -IC:/clang_man/llvm-project/flang/include -IC:/clang_man/llvm-project/build/tools/flang/include -IC:/cl
ang_man/llvm-project/build/include -IC:/clang_man/llvm-project/llvm/include -IC:/clang_build/msys64/mingw64/include -isystem C:/
clang_man/llvm-project/llvm/../mlir/include -isystem C:/clang_man/llvm-project/build/tools/mlir/include -isystem C:/clang_man/ll
vm-project/build/tools/clang/include -isystem C:/clang_man/llvm-project/llvm/../clang/include -Wa,-mbig-obj -Wa,-mbig-obj -Werro
r=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-
long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-redundant-move -Wno-pessimizing-move -Wno-noexcep
t-type -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wno-misleading-indentation -ffunction-sections -fdata-sections
 -Wno-deprecated-copy -Wno-ctad-maybe-unsupported -fno-strict-aliasing -fno-semantic-interposition  -O2 -DNDEBUG -std=c++17  -fn
o-exceptions -MD -MT tools/flang/lib/Lower/CMakeFiles/obj.FortranLower.dir/PFTBuilder.cpp.obj -MF tools\flang\lib\Lower\CMakeFil
es\obj.FortranLower.dir\PFTBuilder.cpp.obj.d -o tools/flang/lib/Lower/CMakeFiles/obj.FortranLower.dir/PFTBuilder.cpp.obj -c C:/c
lang_man/llvm-project/flang/lib/Lower/PFTBuilder.cpp
d:/csd_ntly/coside-3.2rc1_mingw-w64/tools/gcc-9.4.0/bin/../lib/gcc/x86_64-w64-mingw32/9.4.0/../../../../x86_64-w64-mingw32/bin/a
s.exe: tools/flang/lib/Lower/CMakeFiles/obj.FortranLower.dir/PFTBuilder.cpp.obj: section .xdata.unlikely._ZNSt8__detail9__varian
t17__gen_vtable_implILb1ENS0_12_Multi_arrayIPFvON7Fortran6common8visitorsIJZNKS3_5lower3pft20ReferenceVariantBaseILb1EJNS3_6pars
er12AllocateStmtENS9_14AssignmentStmtENS9_13BackspaceStmtENS9_8CallStmtENS9_9CloseStmtENS9_12ContinueStmtENS9_9Cycle.
...
T_IKS2O_EENS6
T_IKS2P_EENS6T_IKS2Q_EENS6T_IKS2R_EENS6T_IKS2S_EEEEEJEEESt5tupleIJSAT_EESt16integer_sequenceIyJLy32EEEE14__visit_invokeES6R_SAT_
: string table overflow at offset 10001804
C:\clang_build\msys64\tmp\cc79WZnJ.s: Assembler messages:
C:\clang_build\msys64\tmp\cc79WZnJ.s: Fatal error: tools/flang/lib/Lower/CMakeFiles/obj.FortranLower.dir/PFTBuilder.cpp.obj: fil
e too big
ninja: build stopped: subcommand failed.
[37/584] Building CXX object tools/flang/lib/Optimizer/Dialect/CMakeFiles/obj.FIRDialect.dir/FIROps.cpp.obj
ninja: build stopped: interrupted by user.

I hand over via CMAKE_C_FLAGS, CMAKE_CXX_FLAGS the big object option trying to overcome this issue, but it doesn't help. Another suggestion was to use -O2, but this is already ensured building Release.

Any suggestion on that?

KR,
Thomas

[maveric2]

In addition I see a huge memory consumption while building, thus I used these flags for restricting the core usage:
-DLLVM_PARALLEL_COMPILE_JOBS=10 -DLLVM_PARALLEL_LINK_JOBS=1 (my machine has 20threads available and 40GB RAM)

[efriedma-quic]

"String table overflow" has been reported before; see #57718/#51143 . You should be able to work around it by building with clang instead.

For memory usage, I've noticed that building flang takes a lot of memory even on other targets.

[maveric2]

Thanks @efriedma-quic - I'll try.

[carlo-bramini]

I got the same identical problem in my port of llvm-project to CYGWIN.
I did a patch that solved the problem on my platform and I posted it to D159404:
https://reviews.llvm.org/D159404
The patch adds this code to mlir/CMakeLists.txt and flang/CMakeLists.txt:

if (CYGWIN)
  # On most UNIX like platforms g++ and clang++ define _GNU_SOURCE as a
  # predefined macro, which turns on all of the wonderful libc extensions.
  # However g++ doesn't do this in Cygwin so we have to define it ourselfs
  # since we depend on GNU/POSIX/BSD extensions.
  add_definitions(-D_GNU_SOURCE=1)
  # Compiling large source files that use templates intensively, you may
  # receive the "File too big/too many sections" error. For MSVC you can
  # use the /bigobj flag while GNU Assembler provides the -mbig-obj flag.
  if( MSVC )
    append("/bigobj" CMAKE_C_FLAGS CMAKE_CXX_FLAGS)
  endif()
  if ( LLVM_COMPILER_IS_GCC_COMPATIBLE )
    append("-Wa,-mbig-obj" CMAKE_C_FLAGS CMAKE_CXX_FLAGS)
  endif( LLVM_COMPILER_IS_GCC_COMPATIBLE )
endif()

I verified that it works fine on CYGWIN.
If it will be good enough to be merged, this code may be modified in this way for fixing support also on MINGW:

if (CYGWIN)
  # On most UNIX like platforms g++ and clang++ define _GNU_SOURCE as a
  # predefined macro, which turns on all of the wonderful libc extensions.
  # However g++ doesn't do this in Cygwin so we have to define it ourselfs
  # since we depend on GNU/POSIX/BSD extensions.
  add_definitions(-D_GNU_SOURCE=1)
endif()

if (CYGWIN OR WIN32)
  # Compiling large source files that use templates intensively, you may
  # receive the "File too big/too many sections" error. For MSVC you can
  # use the /bigobj flag while GNU Assembler provides the -mbig-obj flag.
  if( MSVC )
    append("/bigobj" CMAKE_C_FLAGS CMAKE_CXX_FLAGS)
  endif()
  if ( LLVM_COMPILER_IS_GCC_COMPATIBLE )
    append("-Wa,-mbig-obj" CMAKE_C_FLAGS CMAKE_CXX_FLAGS)
  endif( LLVM_COMPILER_IS_GCC_COMPATIBLE )
endif()

Perhaps, you may try to test this fix also on your compiler.
If it will work also on your side, probably a second patch for fixing support also for WIN32 can be merged.

[kalcohol]

Hi guys, i'm trying to build mlir at 17.01 and 17.02 using mingw & ninja, both version failed(successed at 15.0.7 & 16.0.6):

E:/projects/libraries/llvm-project-llvmorg-17.0.2/build-mingw64/tools/mlir/test/lib/Dialect/Test/TestOps.h.inc:40424:19: note: use unary '+' which decays operands to pointers or '&'component_ref' not supported by dump_decl<declaration error>[0] == &'component_ref' not supported by dump_decl<declaration error>[0]' to compare the addresses
as: tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestDialect.dir/TestDialect.cpp.obj: section .data$_ZGVZN4mlir6detail14TypeIDResolverIN4test6detail34DenseArrayAttrOpGenericAdaptorBase10PropertiesEvE13resolveTypeIDEvE2id: string table overflow at offset 10000082
C:\Users\i\AppData\Local\Temp\ccdNTnVw.s: Assembler messages:
C:\Users\i\AppData\Local\Temp\ccdNTnVw.s: Fatal error: tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestDialect.dir/TestDialect.cpp.obj: file too big
ninja: build stopped: subcommand failed.

so the 'string table overflow at offset' may the same issue, and posted on here for now.

[carlo-bramini]

@kalcohol I have been able to build mlir with both CYGWIN and MinGW, although I got an issue at link time of some executables, as explained into #64150.
On CYGWIN, although I have not tried them, versions 17.0.x surely won't work, unless you import this fix applied to current 18.git:
https://reviews.llvm.org/rGe873280e614f8457ebbe2ffdee389b4e336739a6
But with MinGW, the issue was not happening here since it was already including -Wa,-mbig-obj flag.
I'm wondering if:

1. you could compile also the latest sources from git, for testing.
2. you can write the kind of environment you are using (MSys2/CYGWIN/linux/others) and the version of GCC and w32api.
3. the command line that you used when configuring with CMake.

I think that it could be useful for reproducing the issue.

[kalcohol]

@kalcohol I have been able to build mlir with both CYGWIN and MinGW, although I got an issue at link time of some executables, as explained into #64150. On CYGWIN, although I have not tried them, versions 17.0.x surely won't work, unless you import this fix applied to current 18.git: https://reviews.llvm.org/rGe873280e614f8457ebbe2ffdee389b4e336739a6 But with MinGW, the issue was not happening here since it was already including flag. I'm wondering if:-Wa,-mbig-obj

1. you could compile also the latest sources from git, for testing.
2. you can write the kind of environment you are using (MSys2/CYGWIN/linux/others) and the version of GCC and w32api.
3. the command line that you used when configuring with CMake.

I think that it could be useful for reproducing the issue.

Hi @carlo-bramini ,

1. building latest sources which cloned today was a success.
2. backport the fix https://reviews.llvm.org/rGe873280e614f8457ebbe2ffdee389b4e336739a6 to 17.0.2 then rebuilt failed same as before.
3. my building cmd:

@SETLOCAL
@ECHO OFF

@SET CMAKE_BIN_DIR=D:\Program\JetBrains\CLion\bin\cmake\win\x64\bin
@SET NINJA_BIN_DIR=D:\Program\JetBrains\CLion\bin\ninja\win\x64
@SET MINGW_BIN_DIR=D:\Program\JetBrains\CLion\bin\mingw\bin

@SET PATH=%CMAKE_BIN_DIR%;%NINJA_BIN_DIR%;%MINGW_BIN_DIR%;%PATH%

@SET LLVM_SRC_DIR=E:\projects\libraries\llvm-project
@SET LLVM_BUILD_DIR=%LLVM_SRC_DIR%\build-mingw64
@SET LLVM_INSTALL_DIR=E:\projects\libraries\libmlir-master-mingw

@MKDIR %LLVM_BUILD_DIR% && CD /D %LLVM_BUILD_DIR%
@cmake.exe -G Ninja                       ^
          -S %LLVM_SRC_DIR%\llvm          ^
          -B %LLVM_BUILD_DIR%             ^
          -DLLVM_ENABLE_PROJECTS="mlir"   ^
          -DLLVM_TARGETS_TO_BUILD="host"  ^
          -DLLVM_ENABLE_ASSERTIONS=ON     ^
          -DCMAKE_BUILD_TYPE=RELEASE      ^
          -DPython3_EXECUTABLE=D:\Program\anaconda3\envs\mlir\python3.exe ^
          -DCMAKE_INSTALL_PREFIX=%LLVM_INSTALL_DIR%
@cmake.exe --build %LLVM_BUILD_DIR% && @cmake.exe --install %LLVM_BUILD_DIR%

@ENDLOCAL

other info:
clion: 2023.2.2
cmake: 3.26.4
mingw: 308c1f397b7024a657688d42319758f2ece81f52
mingw gcc: 13.1.0