Issues with the Stable ABI

[gvanrossum]

During PyCon, @hauntsaninja mentioned some surprising data: out of 400,000 PyPI projects, only about 300 are using the Stable ABI, and only two of those have a name anyone recognizes. (Shantanu could you link to those results and verify the numbers?) [UPDATE: There are better numbers further down in the thread.]

So I'd like to add the existence of the Stable ABI to the list of problems. It makes evolving certain APIs hard -- in particular, the ob_refcnt field is exposed through macros in the Stable ABI which means that for immortal objects in 3.12, we had to go through contortions to keep supporting wheels built with the 3.11 or older versions of Py_INCREF and friends.

Maybe the Stable ABI could at least be revised to not contain any macros that access object fields directly (replacing them with equivalent functions)?

[hauntsaninja]

Here's where I got that number: https://blog.trailofbits.com/2022/11/15/python-wheels-abi-abi3audit/

To get a list of auditable packages to feed into abi3audit, I used PyPI’s public BigQuery dataset to generate a list of every abi3-wheel-containing package downloaded from PyPI in the last 21 days:
[...]
From that query, I got 357 packages, which I’ve uploaded as a GitHub Gist.

Not sure how much the "downloaded in the last 21 days" affects things. From a single pass skim over the gist just now I recognised five packages.

Probably could twist abi3audit into telling you specifically how many packages would break from removing some macros.

[hauntsaninja]

cc @woodruffw

[woodruffw]

Thanks for the ping! Yeah, I got those numbers from the BigQuery dataset for the last 21 days (which was a purely arbitrary limitation that should be lifted for a full analysis).

Because of how weak the controls around the abi3 tag on wheels and extensions are, it's possible that there are other Stable ABI-built wheels on PyPI that are just tagged incorrectly (i.e. they use the Stable ABI but their wheel filename doesn't indicate so). These arguably don't matter since pip won't treat them as Stable ABI compatible without the abi3 filename tag, but that's another limitation of the query that's worth considering.

Probably could twist abi3audit into telling you specifically how many packages would break from removing some macros.

Yeah, this should be doable -- you can use the big JSON blob in that blog post to see which symbols are actually in use, and I'm open to further improvements to abi3audit itself around that 🙂

[gvanrossum]

Yeah, this should be doable -- you can use the big JSON blob in that blog post to see which symbols are actually in use, and I'm open to further improvements to abi3audit itself around that 🙂

How would that be able to tell you that a given binary is or isn't using Py_DECREF?

[mattip]

One of the big projects using the stable ABI is cryptography, here is their pypi download page. I think they would not be happy if they had to make wheels for all the versions of CPython they support on all the platforms they support. This is the flip-side of the faster (yearly) release cycle. If I recall correctly the stable ABI was one of the carrots offered to package maintainers worried about continually chasing new CPython versions.

Maybe the Stable ABI could at least be revised to not contain any macros that access object fields directly (replacing them with equivalent functions)?

It makes sense, I don't think the intent of the stable ABI was to freeze the ABI forever. Could there be an abi31 tag that would mark the next version of the stable C-API?

[woodruffw]

How would that be able to tell you that a given binary is or isn't using Py_DECREF?

It only works for symbols and not macros, unfortunately, so abi3audit can't reveal which extensions/wheels happen to use that macro (unless in doing so they leak an underlying symbol, as Py_DECREF does for _Py_DECREF). So in that particular case you can infer that Py_DECREF was used, but there's no guarantee in the general case.

It makes sense, I don't think the intent of the stable ABI was to freeze the ABI forever. Could there be an abi31 tag that would mark the next version of the stable C-API?

I'm biased as a package maintainer, but I'm personally a fan of something like this: the problem with the Stable ABI isn't the idea itself, but the restrictions that it's accumulated over the years.

A similar idea was floated on the forums a few months ago: https://discuss.python.org/t/lets-get-rid-of-the-stable-abi-but-keep-the-limited-api/18458

[encukou]

Stable ABI considered harmful
only two of those have a name anyone recognizes

Could we stick to facts, please?
As someone working on the stable ABI, I find the tone of this issue needlessly negative.

So I'd like to add the existence of the Stable ABI to the list of problems. It makes evolving certain APIs hard -- in particular, the ob_refcnt field is exposed through macros in the Stable ABI which means that for immortal objects in 3.12, we had to go through contortions to keep supporting wheels built with the 3.11 or older versions of Py_INCREF and friends.

AFAIK, ob_refcnt, ob_type and ob_size are the very last ones remaining. The rest of the structs are “blueprints” for initialization (not used at runtime), and the special case of Py_buffer.

This is the biggest issue in the stable ABI (which is 14 years old now). Your post suggests there are other big issues, are there? (I know about many of issues, of course, but nothing this big...)

Maybe the Stable ABI could at least be revised to not contain any macros that access object fields directly (replacing them with equivalent functions)?

Replacing them with functions would mean an abi4 -- essentially, telling everyone to recompile. It'd be a bunch of work, in CPython and packaging tools, but doable.

[gvanrossum]

In this repo we're trying to just do an inventory of problems, without immediately jumping to solutions (unless the problem is small and the solution is similarly contained).

With apologies for the provocative title, it does seem that the remaining struct fields exposed by the Stable ABI, even if they are just ob_refcnt, ob_type and ob_size are a big problem, that will require elbow grease, resolve, and probably a PEP for a solution.

Looking at the discourse thread Let's get rid of the stable ABI, but keep the limited API I actually see a spectrum of opinions on how to address the problem (and some confusion between Stable ABI and Limited API), but no additional problems, so let's just say that struct fields (including the all-important ob_refcnt) are the main problem that is exacerbated by the existence of the Stable ABI.

Another problem is that the Stable ABI doesn't provide the full functionality needed by some projects, e.g. PyObjC. (This may explain that relatively few projects use it.)

That thread also confirms the rough count of abi3-using projects (around 320).

Anyway, I suggest that we stick to this for the inventory for now, unless additional problems related to the Stable ABI are uncovered. (@encukou Feel free to add the other problems you know of, maybe we'll eventually discern a pattern leading to an innovative solution.)

[encukou]

Most other problems of the stable ABI (and limited API) are shared with the C API in general.

Let me give some thoughts, hoping that they're useful even if they are a bit off-topic.

The C API is vast, and my approach to improving it was to find a manageable subset, and then improve its quality and make it complete enough for general usage. Limited API (defined before my time, in PEP-384) was a reasonable starting point, and the stability guarantees make a good “carrot" for people to try it out (and find its shortcomings).

That thread also confirms the rough count of abi3-using projects (around 320).

Yup. It's not big. But it's now possible to use limited API in real-world project, although core dev involvement (cryptography) or lots of hacks (PySide) help. I haven't done too much marketing (e.g. adapt the tutorial), as I still see some incompleteness to fix. It's easier to iron out issues before the whole world uses it :)

As for “only two of those have a name anyone recognizes” -- that might not change much: for a popular leaf project, releasing yearly in the 2-month RC window is not that big of a problem. I expect Stable ABI to be much more useful for the long tail of smaller extensions.

---

Anyway, for the inventory, I see several issues identified here. I'm not sure how you want to organize them:

• Exposed struct fields (ob_refcnt, ob_type, ob_size) force either contortions on us or recompilations on users
• Existence of the Stable ABI (?)
• Incompleteness of the Stable ABI
• Low adoption of the Stable ABI

[alex]

👋, one of the cryptography maintainers here. Happy to answer any questions about our setup, or what's important to us.

[gvanrossum]

👋, one of the cryptography maintainers here. Happy to answer any questions about our setup, or what's important to us.

If we’re ever going to propose to deprecate the stable ABI I promise there will be a debate where you will be heard. But that’s not on the table now. We’re just trying to inventory problems with the C API here. So if anything about the C API bugs you, please create an issue for it! (Come to think of it, specific praise for the C API might also be useful.)

[gvanrossum]

(Closed by mistake, clumsy fingers. :-)