Tracking issue for manylinux_2_28 image

[mattip]

Due to issue #1012, the manylinux_2_24 image proposed as the first PEP600 compliant image did not take off. I think we should offer an image based on manylinux_2_28.

The big question is what base image to use. We can choose between C8S (centos:stream8), AlmaLinux, RockyLinux, or UBI. Personally I would prefer an image based entirely on open source software. There is a thread about this on discuss.python.org

base image	EOL ¹	gcc-toolset-11	aarch64	ppc64le	s390x	bug fix availability
CentOS Stream 8	2024-05-31 ²	yes	yes	yes	planned ³	1st
UBI 8	2029-05-31	NO	yes	yes	yes	2nd
AlmaLinux 8	2029-05-31	yes	yes	yes	soon ⁴	3rd
RockyLinux 8	2029-05-31	yes	yes	planned ⁵	planned ⁵	3rd

^{¹ including Maintenance Support, https://access.redhat.com/support/policy/updates/errata/
² https://wiki.centos.org/About/Product
³ #1282 (comment)
⁴ https://almalinux.discourse.group/t/plans-for-s390x/875
⁵ after RockyLinux 9 is out, https://forums.rockylinux.org/t/rocky-linux-8-4-available-now/3015/48}

The checklist is short, since wheel and pip, which both use packaging, and warehouse do not require additional PRs.

---

Publisher-side Support:

• Provide a manylinux2_28 docker environment, feature: add manylinux_2_28 image #1277 (almalinux based PoC)
• New release of auditwheel (might not be needed to get things started, c.f. test: add manylinux_2_28 tests auditwheel#369)
  • Find BETA testers to check wether changes are required immediately, especially C++ projects have a chance to get a failure on repair, knowing how much they're affected will help to prioritize work on auditwheel.
• Upload the new images to quay.io

Documentation:

• ???

Additional projects to check for support for pernnial wheels (left over from gh-542, perhaps these are already done?

• Add an option to target manylinux_2_28 in cibuildwheel feature: add manylinux_2_28 support cibuildwheel#1026
• Add an option to target manylinux_2_28 in multibuild
• Add an option to target manylinux_2_28 in maturin
• Distinguish between images in dockcross
• Develop list of further relevant helper utilities and update them

[tiran]

We can choose between AlmaLinux, RockyLinux, or UBI.

Is there any particular reason why C8S is missing from the list? centos:stream8 is an option, too. After all AlmaLinux and RockyLinux are getting RPM updates from C8S.

Personally I would prefer an image based entirely on open source software.

All images are entirely based on OSS and freely redistributable. (If you want to be really strict, then you have to exclude all three distributions, because they may contain proprietary firmware blobs.)

[mattip]

Is there any particular reason why C8S

No, I added C8S to the issue description.

All images are entirely based on OSS and freely redistributable

Cool, thanks for the clarification, sorry for any unintentional dispersion that I cast on any project.

[mattip]

@tiran, @kpfleming, @mayeut, @h-vetinari, do you know canonical links to docker images for glibc2_28 linux distributions? Any thoughts as to which distro we should use? I have really no idea how they all relate to each other, and would like to avoid any kind of minefield in our mutual goal to get to a stable base image.

[kpfleming]

We don't need to be concerned about firmware blobs, since only container images are under discussion here :-)

Also, I know it's being pedantic, but the 'container' world consists of more than just Docker in modern times. You may want to consider just referring to the images as "OCI images".

Most of the relation discussion has happened in the Discourse thread. All four image flavors you've mentioned originate from the same source packages (CentOS 8 Stream), and each packager builds their own binary packages.

The UBI8 image is available here. I don't have links for the other images.

[h-vetinari]

do you know canonical links to docker images for glibc2_28 linux distributions?

Assuming we're not talking about canonical-the-company (😉), there's https://hub.docker.com/_/almalinux and https://hub.docker.com/_/rockylinux. @mayeut seems to have experimented with almalinux already.

I don't have a strong preference regarding which distro (I'm guessing one deciding factor might be availability of architecture support like PPC, or perceived viability of the community; here the RH-sponsored flavours obviously have a big advantage...), though I don't understand one aspect about C8S that might be pertinent. Previously, updates flowed from RHEL to CentOS. With stream, this is supposed to be inverted, at least starting with RHEL9.

What I don't know if C8S now lives up- or downstream of RHEL, which - as @tiran and @kpfleming explained - probably matters quite little because the alternatives seems to be using the same sources (though given the history of COS8, I wouldn't be surprised if it's still downstream and just picked up the new "stream" name?).

Where this becomes relevant is if C8S now still is downstream of RHEL (as before), but won't be anymore as of RHEL9 - then having the same name ("centos stream") will imply we should just bump the version 8->9 at some point in the future, when in fact we'd be getting a very different thing. Is what I'm saying understandable, @tiran @kpfleming? Or completely off in some way?

[kpfleming]

If you've got a bit of time, one of our RH colleagues presented a session at FOSDEM 2022 last weekend which shows how the branches and releases relate to each other.

CentOS Stream 8 and CentOS Stream 9 will always be separate, the 'stream' name does not mean that C8S users will just magically get converted to C9S when it is released. It's not a 'rolling' distribution, if that's what you are concerned about.

[h-vetinari]

It's not a 'rolling' distribution, if that's what you are concerned about.

What I'm concerned about is C8S not being a rolling distribution (and thus a viable candidate here), but sharing the name with C9S, which is a rolling distribution (or at least, that's how I understood it so far; I don't necessarily mean "bleeding edge", but by being ahead of RHEL and possibly diverging, it flips the nature of what used to be CentOS on its head).

I would want to avoid a situation where - if we were to choose C8S now - this IMO misleading naming is then used to argue that a future manylinux image should be based on C9S.

[kpfleming]

Understood. I think you can avoid those issues entirely then by just choosing UBI instead of CentOS Stream :)

[bookwar]

What I'm concerned about is C8S not being a rolling distribution (and thus a viable candidate here), but sharing the name with C9S, which is a rolling distribution (or at least, that's how I understood it so far; I don't necessarily mean "bleeding edge", but by being ahead of RHEL and possibly diverging, it flips the nature of what used to be CentOS on its head).

I think there is still some misunderstanding.

CentOS Stream 8 and CentOS Stream 9 share name because they also share the concept. They are two branches of the same thing. So CentOS Stream 9 doesn't "roll" any more than CentOS Stream 8.

CentOS Stream 8 is the representation of RHEL 8 Nightly, CentOS Stream 9 is the representation of RHEL 9 Nightly.

The thing which is different between 8 and 9 currently is that you don't see the merge of the sources happening in 8, you only see the result of it. While in 9 you also see how exactly RHEL developer merges the code for RHEL=CentOS Stream, and how the change passes the RHEL verification stages.

The outcome is still the same: package lands in CentOS Stream 9 after it lands in RHEL 9 Nightly.

[h-vetinari]

OK, thanks for the explanation @bookwar. Can you comment if RHEL9 and C9S will be ABI-locked, or will C9S be allowed to break away and ahead?

[h-vetinari]

(I appreciate the insights; I'd be curious to know what Alma/Rocky will do once C9S rolls around, because the switch to stream created enough FUD that not one, but two projects got launched to fill the perceived void - if RHEL9 and C9S remain fully ABI-compatible, then I don't see much of a functional difference to the old way)

[duritong]

They will stay as ABI compatbile as RHEL minor versions did in the past, since CentOS Stream Y just represents RHEL Y.(N+1) where N is the last minor release of RHEL that got GA. Now when it comes to ABI compatbibility I would refer to RHEL's promise of ABI compatibility within minor releases, since this is essential what CentOS Stream Y will be.

CentOS previously picked up N after it became GA in RHEL and thus could have been lacking (sometimes for months!), now it is the other way around.

[bookwar]

OK, thanks for the explanation @bookwar. Can you comment if RHEL9 and C9S will be ABI-locked, or will C9S be allowed to break away and ahead?

CentOS Stream 9 becomes locked on RHEL 9 ABI compatibility right after RHEL 9 GA.

For pre-release development which is happening now we have the freedom of still changing and adjusting things. Though as RHEL 9 Beta has already been released and we are very close to GA dates there is not going to be any major changes. But after release rules become strict.

[h-vetinari]

OK, thanks for all that - happy to retract my concerns about centos 8 stream in the context of this discussion.

CentOS Stream Y just represents RHEL Y.(N+1) where N is the last minor release of RHEL that got GA

That's the key part IMO. Already previously, the minor version was just that - not very important. Important was (and remains) one cohesive "epoch" represented by the major number + all the ABI stability etc. To be honest, this is a TIL-moment for me, and now I understand way less what the value proposition of alma & rocky is supposed to be.

[tiran]

The official C8S OCI images is hosted on Quay as quay.io/centos/centos:stream8. The image provides developer toolsets like gcc-toolset-11-gcc-11.2.1-1.2.el8_5 out of the box.