DISCUSS: manylinux_2_24 EOL?

[h-vetinari]

manylinux_2_24 is based on Debian 9, which reaches the end of its long-term-support at the end of this month, see e.g. here.

This is obviously out-of-sync with the much longer support of the CentOS-based images, e.g. manylinux2010 being deprecated this year (#1281), and the EOL of manylinux2014 currently noted for 2024-06-30 (see link above).

manylinux_2_24 is noted as supported in the README, but not as an "Acceptable distros to build wheels" in the pep-compliance repo.

Finally, the lack of gcc-toolset (see #1012) makes 2_24 seriously unappealing, c.f. the linked issues in that thread for people staying on manylinux2014 because it has newer compilers. See the outlier in the table below the fold

manylinux vs GCC version

manylinux version	GCC version
2010	8.3
2014	10.2
2_24	6.3
2_28	11.1

All things considered, should manylinux_2_24 support expire at the end of the month? Has there been deployment in any appreciable numbers?

What do people think?

[h-vetinari]

The precarious dip in GCC version for 2_24 means that until this image is EOL (or #1012 solved, which doesn't look likely), scipy cannot move beyond GCC 6. From that POV, I wouldn't shed many tears if 2_24 already went the way of the dodo.

[mattip]

Agreed. It can be deprecated.

[hzhangxyz]

As an user, I think 2_24 is useless, I can always use 2014 in every situation that I want use 2_24, but can not vice versa.

[mayeut]

I'm okay to start a deprecation process for this one given all its shortcomings.

Using https://github.com/sethmlarson/pypi-data:
There are 275 packages out there that used or are still using manylinux_2_24.

sqlite3 'pypi.db' 'SELECT name FROM wheels WHERE platform LIKE "%manylinux_2_24%" GROUP BY name;' | wc -l
275

[mayeut]

Let's ask maintainers of project in top 5000 packages using manylinux_2_24:

• bcrypt / cryptography / pynacl: @alex, @reaperhulk
• bottleneck: @qwhelan, @rdbisme
• cvxpy / ecos: @SteveDiamond
• cython / lupa / lxml: @scoder
• httpstan: @riddell-stan
• orjson: @ijl
• pmdarima: @tgsmith61591
• psycopg-binary / psycopg2-binary: @dvarrazzo
• pyjson5: @Kijewski
• pymssql: @termim
• ruamel-yaml-clib: ?
• wasmer / wasmer-compiler-cranelift: @syrusakbary

[dvarrazzo]

psycopg-binary / psycopg2-binary: @dvarrazzo

Hi there,

For psycopg, 2_24 is the preferred build platform, because it ships with libssl 1.1.x, as opposite as Centos 7 shipping with 0.9.x, which is extremely buggy and segfaults if psycopg is used together with the Python ssl module (psycopg/psycopg2#543). It is the reason we stopped shipping wheels packages with the psycopg2 distribution and came out with the psycopg2-binary distribution hack.

Because of the above, when building on manylinux2014, we need to build from scratch several libraries (libpq, libssl, ldap, sasl).

psycopg2-binary is built with manylinux2014 on i686 and x86_64, with 2_24 on aarch64 and ppc64le platforms. Building on 2014 predates 2_24, otherwise I'd be happy to ditch 2014 altogether and just build the packages using the Debian system packages. That would make me happy for 5 minutes, after which a mob with pitchforks will start knocking at my door because of the loss of backward compatibility, of course, putting a quick end to my happiness.

psycopg-binary (which is a C speedup module for Psycopg 3, not a whole complete distribution as in psycopg2) is only build with 2_24. This is not immune to problems (psycopg/psycopg#124) however it provides a much simpler build chain. Build time is already pretty long as it is on aarch64/ppc64le (around 20 mins) and if we had to build the depending libraries will likely go up into the hours.

Dropping 2_24 will hence cause me a few days of work to port libraries building to Centos, which I'm not 100% sure will work on aarch/ppc. I will be grumpy for a few days, then probably I'll get over it.

Should it happen, I would like to know ASAP, because I would like to release psycopg 3.1 in a few weeks and it would be wise to choose which build platform to use before that rather than in a bugfix release, because the libpq has several build-time parameters and subtle incompatibilities between the packaged versions and the own-built versions are likely (see psycopg/psycopg2#1365 for example).

Thank you for the heads up!

[reaperhulk]

cryptography builds on manylinux2014 images in addition to 2_24 (on both x86_64 and aarch64) partially to see what percentages of our user base can adopt newer versions. Currently over 60% of our downloads come from the 2_24 wheels, but any platform that understands 2_24 will work with 2014 so from an operational perspective dropping 2_24 isn't a big deal to pyca projects.

That said, is the intent to have only 2014 and a new 2_28? In the past we've found that as the images age they become increasingly difficult to use sanely even if the distribution is still technically supported, and making manylinux2014 be the only option for broad compatibility across the Python user base is a bit scary. I know that volunteer resources are highly limited though, so that may be the only feasible path.

Thanks for tagging us 😄

[alex]

Yes, to me the question is what are we deprecating it in favor, something newer or something older? I think it'd be unfortunate to remove the newest manylinux image, but would be perfectly fine replacing it something newer.

[h-vetinari]

That said, is the intent to have only 2014 and a new 2_28?

I think so (the roll-out of 2_28 is almost complete, see #1282). This would also continue the pattern of the previous images being based (respectively) on RHEL 5, 6, 7, and 2_28 being based on RHEL 8 (through AlmaLinux as CentOS is now Stream).

Looking at https://mayeut.github.io/manylinux-timeline/, the "glibc readiness" per python version looked the following on 1st of June:

python version	2.24+	2.26+	2.27+	2.28+
3.7	95.7%	93.4%	62.8%	30.4%
3.8	98.9%	98.6%	95.7%	79.7%
3.9	99.3%	99.2%	98.4%	97.3%
3.10	99.6%	99.6%	99.4%	95.9%

Given that python 3.7 is already in security-only release mode (and has been dropped at end of '21 already by libraries following NEP29), this IMHO looks like jumping straight to 2_28 should be mostly feasible.**

** note, I took glibc readiness because that is harder to change. Using manylinux_2_x also needs a new enough pip which is not installed everywhere out of the box (see "policy readiness" as opposed to "glibc readiness"), but at least pip can be updated.

[Kijewski]

For pyjson5 I simply let cibuildwheel build all the wheels for "completeness". It does not need any feature not found in manylinux2014 or 2010. (Actually I did not realize that the "newer" platform tag uses an older GCC version.)

[ijl]

orjson now publishes manylinux_2_28 and manylinux2014, dropping manylinux_2_24. I initially had feedback from python3.10 users on Amazon Linux 2 (glibc 2.26?) and Ubuntu 18.04 (glibc 2.27?) that CPython wheels weren't available, so I've added manylinux2014 wheels on even the most recent Python.

I didn't know the approach was that manylinux2014 would be supported longer than manylinux_2_24. It's probably been mentioned elsewhere but I only read the README here--maybe adding it here would help others.

[riddell-stan]

httpstan is now on manylinux_2_28. Thanks for the heads up.