Say something in Troubleshooting about the GLIBC issue

[michaelpj]

It comes up a lot, it would be good to have it written down in one place so @hasufell doesn't have to keep repeating himself.

@hasufell could you give me a quick summary and I'll add it?

[hasufell]

The problem

For unknown linux

1. GHC 9.4.x only provide a fedora 33 bindist, not a fedora 27: https://downloads.haskell.org/~ghc/9.4.2/
2. so far we've used fedora bindists as fallback for Linux_UnknownLinux distros, because debian bindists have caused more trouble (e.g. wrt ncurses/tinfo): https://github.com/haskell/ghcup-metadata/blob/448e6b65a19b5b9b5e9d5e67d896088142d0c239/ghcup-0.0.7.yaml#L2588-L2589
3. HLS needs to be in sync with a specific GHC bindist due to ABI issues. The HLS release CI uses ghcup to install GHC and then build the release binaries with it. HLS CI builds fedora bindist on fedora 33, because otherwise it could not build the GHC-9.4.2 corresponding binary.
4. Fedora 33 glibc is very new (GLIBC_2.32) and doesn't work with a lot of distros.

For known linuxes

Every distro that has an entry for GHC (e.g. Mint, Debian, Ubuntu) also needs an explicit bindist for HLS, so that we can guarantee both correct glibc and that ABI actually matches. For linux mint this is already not the case:

1. For GHC 9.2.4 it installs ghc-9.2.4-x86_64-fedora27-linux.tar.xz
2. For GHC 9.4.2 it installs ghc-9.4.2-x86_64-deb10-linux.tar.xz
3. It installs HLS haskell-language-server-1.8.0.0-x86_64-linux-fedora33.tar.xz
   • this HLS is built with ghc-9.4.2-x86_64-fedora33-linux.tar.xz though
   • this causes both ABI mismatch and possibly glibc issues

What has been done so far

For ubuntu, we fixed this by adding another HLS release that specifically builds for ubuntu:

• haskell/ghcup-metadata@cea8655
• Add ubuntu builds to gitlab CI #3191

Solutions

For unknown linuxes

We could switch the default bindist for Linux_UnknownLinux to deb9 or deb10 for GHC. That would fix the GHC portion.

https://github.com/haskell/ghcup-metadata/blob/448e6b65a19b5b9b5e9d5e67d896088142d0c239/ghcup-0.0.7.yaml#L2588-L2589

This will cause a number of issues though:

1. it will break cabal cache (because GHC ABI is different and cabal doesn't distinguish that) if the user already has build artifacts and reinstalls GHC. This can also break github CI.
2. it will require users to re-install GHC
3. will cause further potential ABI breakage with existing HLS release (probably only 1.8.0.0, because these issues started with GHC-9.4.x)
4. Debian is less cross-distro compatible than Fedora (except for the ancient glibc), so we may be opening the door to ncurses/libtinfo issues

This however is only half of the solution. I realized that we in fact need another HLS release bindist that is built on a distro that is "unknown" in order to trigger ghcup to pick the right bindist for Linux_UnknownLinux. Currently we pick haskell-language-server-1.8.0.0-x86_64-linux-fedora33.tar.xz for unknown linuxes, which is built on Fedora 33.

That means we only get ABI matches as long as Linux_UnknownLinux and Linux_Fedora coincide (they often do) and otherwise we get fatal ABI mismatch failure. Oops.

My idea here is to actually build on Fedora 27 and do some hacks to make ghcup think it's not Fedora.

For known linuxes

Just like ubuntu, provide more bindists for known linuxes. Basically every linux distro entry that exists in GHC metadata needs a dedicated HLS release job.

Workarounds

1. compile HLS from source: https://www.haskell.org/ghcup/guide/#hls
2. or downgrade to an older HLS, e.g. via this in VScode:

    "haskell.toolchain": {
        "hls": "1.7.0.0"
    }

[hasufell]

@wz1000 I suggest we fix this with 1.8.0.1

[hasufell]

argument against deb bindists as default: https://bugzilla.redhat.com/show_bug.cgi?id=1875587

Causes lots of warnings on invoking ghc:

[ghc@1de2333cecae ~]$ ghc --version
/home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/./ghc-9.4.2: /lib64/libtinfo.so.5: no version information available (required by /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHShaskeline-0.8.2-ghc9.4.2.so)
/home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/./ghc-9.4.2: /lib64/libtinfo.so.5: no version information available (required by /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSghc-9.4.2-ghc9.4.2.so)
/home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/./ghc-9.4.2: /lib64/libtinfo.so.5: no version information available (required by /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSterminfo-0.4.1.5-ghc9.4.2.so)
The Glorious Glasgow Haskell Compilation System, version 9.4.2

[hasufell]

I'm leaning towards Cento7 bindists, although Centos7 has EOL 2024.

[ghc@1de2333cecae ~]$ objdump -T /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/ghc | grep GLIBC
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 __libc_start_main
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 memset
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 getuid
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 getlogin
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.14  memcpy
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 memchr

[ghc@1de2333cecae ~]$ ldd /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/ghc
	linux-vdso.so.1 (0x00007fff42f06000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fc0e668a000)
	libHShaskeline-0.8.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHShaskeline-0.8.2-ghc9.4.2.so (0x00007fc0e6a06000)
	libHSghc-9.4.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSghc-9.4.2-ghc9.4.2.so (0x00007fc0e0bdd000)
	libHSterminfo-0.4.1.5-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSterminfo-0.4.1.5-ghc9.4.2.so (0x00007fc0e0b9c000)
	libHSprocess-1.6.15.0-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSprocess-1.6.15.0-ghc9.4.2.so (0x00007fc0e0b5b000)
	libHShpc-0.6.1.0-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHShpc-0.6.1.0-ghc9.4.2.so (0x00007fc0e0b24000)
	libHSghci-9.4.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSghci-9.4.2-ghc9.4.2.so (0x00007fc0e08ae000)
	libHSghc-heap-9.4.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSghc-heap-9.4.2-ghc9.4.2.so (0x00007fc0e07e7000)
	libHSghc-boot-9.4.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSghc-boot-9.4.2-ghc9.4.2.so (0x00007fc0e0714000)
	libHSbinary-0.8.9.1-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSbinary-0.8.9.1-ghc9.4.2.so (0x00007fc0e064a000)
	libHSexceptions-0.10.5-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSexceptions-0.10.5-ghc9.4.2.so (0x00007fc0e060e000)
	libHSstm-2.5.1.0-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSstm-2.5.1.0-ghc9.4.2.so (0x00007fc0e05ec000)
	libHSmtl-2.2.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSmtl-2.2.2-ghc9.4.2.so (0x00007fc0e05b6000)
	libHStransformers-0.5.6.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHStransformers-0.5.6.2-ghc9.4.2.so (0x00007fc0e0468000)
	libHSdirectory-1.3.7.1-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSdirectory-1.3.7.1-ghc9.4.2.so (0x00007fc0e0403000)
	libHSunix-2.7.3-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSunix-2.7.3-ghc9.4.2.so (0x00007fc0e031d000)
	libHStime-1.12.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHStime-1.12.2-ghc9.4.2.so (0x00007fc0e0180000)
	libHSfilepath-1.4.2.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSfilepath-1.4.2.2-ghc9.4.2.so (0x00007fc0e015a000)
	libHScontainers-0.6.6-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHScontainers-0.6.6-ghc9.4.2.so (0x00007fc0dfda0000)
	libHSbytestring-0.11.3.1-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSbytestring-0.11.3.1-ghc9.4.2.so (0x00007fc0dfc2e000)
	libHStemplate-haskell-2.19.0.0-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHStemplate-haskell-2.19.0.0-ghc9.4.2.so (0x00007fc0df852000)
	libHSpretty-1.1.3.6-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSpretty-1.1.3.6-ghc9.4.2.so (0x00007fc0df7da000)
	libHSghc-boot-th-9.4.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSghc-boot-th-9.4.2-ghc9.4.2.so (0x00007fc0df79a000)
	libHSdeepseq-1.4.8.0-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSdeepseq-1.4.8.0-ghc9.4.2.so (0x00007fc0df77e000)
	libHSarray-0.5.4.0-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSarray-0.5.4.0-ghc9.4.2.so (0x00007fc0df6f6000)
	libHSbase-4.17.0.0-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSbase-4.17.0.0-ghc9.4.2.so (0x00007fc0dec3e000)
	libHSghc-bignum-1.3-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSghc-bignum-1.3-ghc9.4.2.so (0x00007fc0debeb000)
	libHSghc-prim-0.9.0-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSghc-prim-0.9.0-ghc9.4.2.so (0x00007fc0de6ee000)
	libHSrts-1.0.2_thr-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSrts-1.0.2_thr-ghc9.4.2.so (0x00007fc0de634000)
	libffi.so.8 => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libffi.so.8 (0x00007fc0de425000)
	libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007fc0de1fb000)
	librt.so.1 => /lib64/librt.so.1 (0x00007fc0ddff3000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007fc0dddf0000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fc0ddbec000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc0dd9ce000)
	libgmp.so.10 => /lib64/libgmp.so.10 (0x00007fc0dd756000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fc0dd3a0000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc0e69d5000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fc0dd187000)

[hasufell]

https://gitlab.haskell.org/ghc/ghc/-/issues/22268

[chreekat]

fedora27 bindists are now available for 9.2.5 and 9.4.4:

• https://downloads.haskell.org/~ghc/9.4.4/ghc-9.4.4-x86_64-fedora27-linux.tar.xz
• https://downloads.haskell.org/~ghc/9.2.5/ghc-9.2.5-x86_64-fedora27-linux-dwarf.tar.xz

[hasufell]

fedora27 bindists are now available for 9.2.5 and 9.4.4:

• https://downloads.haskell.org/~ghc/9.4.4/ghc-9.4.4-x86_64-fedora27-linux.tar.xz
• https://downloads.haskell.org/~ghc/9.2.5/ghc-9.2.5-x86_64-fedora27-linux-dwarf.tar.xz

We don't have corresponding HLS binaries.