GCC version in manylinux_2_24

[henryiii]

The GCC version in manylinux1 is 4.8, painfully old, unable to build C++14 code (see #118). It’s actually possible to build GCC 9 on CentOS 5, though support was removed in GCC 10.

The GCC version in manylinux2010 is 8.3, new enough for basically full C++17 support minus a few library features.

The GCC version in manylinux2014 is 9.3, full support for C++17 and even bits of C++20.

The GCC version in manylinux_2_24 is a dismal 6.3, much worse than both the year based manylinuxes and not even new enough for C++17 language support (mostly added in GCC 7)! This is a huge step backward for an image with a higher GLIBC version. (PS: listing the GLIBC versions of the older manylinuxes would be nice, IMO)

Is there some way to replicate the “RHEL dev toolset” in the Debian based images?

[henryiii]

Maybe https://stackoverflow.com/questions/61590804/how-to-install-gcc-9-on-debianlatest-docker

as long as it doesn’t update glibc by upgrading to a new os (which it might...), this should work like the dev toolset, I think?

[mayeut]

Once you install gcc-9, it will update at least libstdc++ which is a no go. RHEL dev toolset took care of that and also other libraries like libgcc_s so that binaries produced with devtoolset would still be compatible with the base image with no action.
If you have time to craft a PR installing gcc-9 and patching it so that it's still compatible with the base image, I'd be happy to take a look and merge this.

manylinux_2_24 is mostly beneficial for alternate architectures where some functions were fixed in glibc (reported by numpy).

[mattip]

Unfortunately, the 6.3.10 version of gcc is too old to compile OpenBLAS on arm64, I get a error: unknown value 'cortex-a73' for -mtune. It seems support for cortex-a73 was added in gcc7. So now we are stuck: NumPy needs both manylinux_2_24 and a modern gcc.

[henryiii]

Not at all surprised. You want the most recent GCC possible for alternative archs.

[h-vetinari]

Or go for 2_28 right away (at least for arm64)? Debian 10 has GCC 8.3.

[mayeut]

@h-vetinari
Debian 10 can't be used as a base image for manylinux_2_28. There are symbol conflicts with other 2.28 distros (at least Photon 3.0)
Furthermore, it would leave Ubuntu 18.04 users without wheels and there are still a great number of them.

@mattip, there are still a number of possibilities for OpenBLAS/numpy:

• Update the manylinux_2_24 toolchain in manylinux images.

This would be the preferred option but, as mentioned in a previous comment, I simply can't take on this one but will be happy to review a PR doing that as it would benefit the whole manylinux ecosystem (checking it respects original symbol requirements for manylinux_2_24 and reliability/maintainability of proposed packages)

• Update OpenBLAS to a version allowing to specify the DYNAMIC_LIST for aarch64 (OpenMathLib/OpenBLAS@9b2d69a)

Allows to remove CORTEXA73 tuning.
This would allow to build with the stock manylinux_2_24 image but would probably introduce a speed penalty (I don't know how much & there is probably something better to be done for dispatching over at OpenBLAS - fallback to ARMV8 where it should maybe fallback to CORTEXA72 ?)
Also, the dynamic list seems to be built depending on compiler support for some architectures. Should probably be reported over at OpenBLAS to get that feature for aarch64 also.

• Use OpenBLAS built on manylinux2014 images

It should produce a shared library that's perfectly valid on manylinux_2_24.
In fact, now that I think about this, maybe building NumPy/OpenBLAS (others?) on manylinux2014 but tagging it manylinux_2_24 would work ? (depending how the accuracy issues mentioned in #494 were fixed in glibc i.e. symbol version bumped or not ?)

• Update to ppa:ubuntu-toolchain-r/test gcc-9

While I would not recommend it as a generic toolchain for the manylinux_2_24 image (libstdc++ issue), this one might be well suited for NumPy/OpenBLAS. OpenBLAS built successfully on the aarch64 manylinux_2_24 image using this toolchain with no extra non-compliant symbols:

apt-get update
apt-get install dirmngr
apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 1E9377A2BA9EF27F
echo "deb http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu xenial main" >> /etc/apt/sources.list
apt-get update
apt-get install -y --no-install-recommends gcc-9 gfortran-9
export CC=gcc-9
export FC=gfortran-9
git clone https://github.com/xianyi/OpenBLAS.git
cd OpenBLAS/
git checkout af2b0d0
make BUFFERSIZE=20 DYNAMIC_ARCH=1 USE_OPENMP=0 NUM_THREADS=64 BINARY=64 TARGET=ARMV8

• Create & use manylinux_2_27 image based on Ubuntu 18.04

Despite Ubuntu 18.04 not being listed as a potential manylinux candidate over at https://github.com/mayeut/pep600_compliance, it's only because the symbol analysis is done on libstdc++ (gcc 8 based) rather than on the default compiler used (gcc 7)
Users of amazonlinux:2 won't be able to use wheels from this one.

[henryiii]

Or go for 2_28 right away (at least for arm64)? Debian 10 has GCC 8.3.

We need newer compilers with older images; requiring the wheel to be based on an OS newer than the compiler doesn't work well. If you want to use GCC 9 or 10, now even 2_28 would be to old. This was available with RHEL based images, so it should(*) be possible here, too. CentOS 6 has GCC 8, and that's 2_12.

*: No statement about how hard it is, just that it should be possible.

[h-vetinari]

If you want to use GCC 9 or 10

That's quite a wide gap to bridge. Scipy is just recently thinking about bumping their gcc lower bound from 4.8 to 5.5 (not least due to being buildable on more exotic platforms) - the point being that I think very few projects can assume such modern compilers (with all that entails) for their entire user base.

This was available with RHEL based images, so it should(*) be possible here, too.

RH presumably spent a non-trivial amount of time on that... Don't get me wrong, that'd be awesome to have, but I'm thinking that even "cutting-edge" packages should normally be fine with the compilers of a ~2 year old mainstream distro.

That's part of the appeal of the perennial manylinux, IMO, because maintainers can choose themselves where they want to set their cutoff (assuming pypa would provide all those images).

@mayeut:

There are symbol conflicts with other 2.28 distros (at least Photon 3.0)

Photon already has a number of compatibility issues, apparently. From the POV of a maintainer, I want to cover the largest possible portion of my user base (vis-à-vis trade-offs in terms of maintainability) - and it's IMO a legitimate choice not to care about fringe distros (where users can still build from source, and are presumably used to that anyway).

[mayeut]

Photon already has a number of compatibility issues, apparently.

Yes, it's fully headless and does not provide any libraries like X11. It's not a major issue but it's mentioned anyway. If one is using Photon, I guess they know it only works this way and won't rely on packages requiring these dependencies (e.g. use opencv-python-headless instead of opencv-python). It's more of a limitation than a compatibility issue.

From the POV of a maintainer, I want to cover the largest possible portion of my user base (vis-à-vis trade-offs in terms of maintainability) - and it's IMO a legitimate choice not to care about fringe distros (where users can still build from source, and are presumably used to that anyway).

Agreed, it's obviously up to the package maintainers to decide what they want to support. However, there's no reason for manylinux to knowingly provide images that are not compliant with PEP600.

[mayeut]

I did some experiments with PPA in the openblas PR. It seems to be the way to go to update the toolchain here.
I was able to rebuild binutils-2.36.1 & gcc-10 from the upcoming Ubuntu 21.04 but targeting Ubuntu 16.04 and that works well when installed on Debian 9.
I do not have the proper skills to do that in a clean way I think and any help on the matter would be greatly appreciated.
I will try and up my skills w.r.t. providing a PPA with an up-to-date binutils however, for gcc-10, there's still the libstdc++ compatibility issue that's really out of my league here.

1st experiments:

• build utils used to back-port binutils & gcc: https://launchpad.net/~mayeut-github/+archive/ubuntu/test2
• binutils: https://launchpad.net/~mayeut-github/+archive/ubuntu/test3
• gcc: https://launchpad.net/~mayeut-github/+archive/ubuntu/test4

[h-vetinari]

Regarding the toolchain issues, you might be interested how https://github.com/symengine/symengine-wheels handles this (because it builds based on conda-packages, which are compiled with much newer compilers).