Build and test Python 3.15 wheels with cibuildwheel

[ikrommyd]

Follow-up of #927
cibuildwheel 4.0 is now released (not an RC anymore) so we update CI to that.
We also fix its config. We have to enable cpython-prerelease to build for python 3.15 (went unnoticed in the other PR) and we also fix the request for the python versions in CI and pyproject.toml. cibuildwheel 4.0 supports python 3.9 - 3.15. Asking for 3.7 or 3.8 like the CI did so far does not do anything. Looking at the existing wheels https://pypi.org/project/memray/#files, there are no wheels for 3.7 even though the CI was so far trying to ask for them. One has to use an older cibuildwheel version for older python versions.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 60.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.35%. Comparing base (5b58e31) to head (719d968).

Files with missing lines	Patch %	Lines
src/memray/commands/attach.py	33.33%	4 Missing ⚠️
tests/integration/test_tracking.py	50.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #946      +/-   ##
==========================================
- Coverage   92.49%   92.35%   -0.15%     
==========================================
  Files          99       99              
  Lines       11787    11793       +6     
  Branches      430      432       +2     
==========================================
- Hits        10902    10891      -11     
- Misses        885      902      +17     

Flag	Coverage Δ
cpp	92.35% <60.00%> (-0.15%)	⬇️
python_and_cython	92.35% <60.00%> (-0.15%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ikrommyd]

Now the 3.15 problems are being noticed cause #927 did not actually test 3.15 inside the wheel building step :(

[ikrommyd]

The cibuildwheel logs are a bit long and require some patience too but I looked into them and we're getting test failures in 3.15t and a seg fault in i686 only. I had claude parse the logs the generate a report for me which I think is generally accurate from looking at the logs briefly. Here is the report below. I can definitely tackle some of them like the abi3t injecter thing but I definitely cannot debug the i686 segfault without a 32-bit machine. I'd like to know how the maintainers want to proceed here cause 3.15t support is not there yet.

Wheels CI failure analysis (run 27154079947)

TL;DR

Every failure is isolated to the free-threaded Python 3.15 (cp315t) builds, which are newly exercised by this PR (cibuildwheel 4.0 + cpython-prerelease enables cp315/cp315t). Non-free-threaded cp315 passes in every job (628–642 passed, 0 failed). The cp315t failures are four independent
bugs, not one. The only non-cp315t failure is an unrelated flaky timeout.

Per-job breakdown

Job	Result	Failing build	Cause
manylinux_x86_64	✅ pass	—	attach tests skipped here (no debugger), so it never hit the injecter bug
manylinux_aarch64	❌ 4 failed	cp315t	1) injecter
macosx_arm64	❌ 5 failed	cp315t	1) injecter (4) + 2) native parse (1)
musllinux_x86_64	❌ 1 failed	cp315t	2) native parse
manylinux_i686	❌ segfault (exit 139)	cp315t	3) segfault
macosx_x86_64	❌ 1 failed	cp313	4) flaky timeout (unrelated to 3.15)

Root causes

1) — attach/detach can't find the injecter (abi3t)
Tests: test_basic_attach, test_aggregated_attach, test_attach_time, test_detach_without_attach.
src/memray/commands/attach.py:136 hardcodes the injecter filename:

injecter = pathlib.Path(memray.__file__).parent / "_inject.abi3.so"
assert injecter.exists()

_inject is built with py_limited_api=True. On free-threaded 3.15 the limited-API extension is tagged abi3t, so the wheel ships _inject.abi3t.so and the hardcoded _inject.abi3.so lookup fails the assert. (cp314t happened to still emit abi3.so, which is why it passed.)
Fix: locate the extension without hardcoding the ABI tag, e.g. next(package_dir.glob("_inject*.so")).

2) — native parse produces zero records of some type
Test: test_main.py::TestParseSubcommand::test_successful_parse (run with native=True).
On cp315t, memray parse of the capture file yields 0 records for at least one record type, tripping assert count > 0. The assertion iterates .values() so it doesn't report which type is missing — needs reproduction on a free-threaded 3.15 interpreter (or a temporary assert count > 0, record_type) to
identify it.

3) — segfault on 32-bit free-threaded
Job: manylinux_i686, build cp315t-manylinux_i686. pytest died with Fatal Python error: Segmentation fault (exit code 139) during the test_socket tests, so no test summary was produced. A genuine memory-safety crash specific to 32-bit + free-threaded 3.15.

4) — flaky timeout (not a 3.15 issue)
Job: macosx_x86_64, build cp313 (regular GIL). test_socket.py::...test_multi_allocation_snapshot failed with subprocess.TimeoutExpired. Because cibuildwheel aborts a job at the first version whose test phase fails, this stopped the run after cp313 — that runner never reached 3.15 at all. Almost
certainly infra flake; a re-run should clear it.

[ikrommyd]

I have fixed the injecter problem and now I'm chasing the failure in this line

memray/tests/integration/test_main.py

Lines 630 to 631 in e15c16a

	for count in record_count_by_type.values():
	assert count > 0

but I can't reproduce it locally (even by running the whole cibuildwheel chain locally).
For the i686 segfault, someone with a 32-bit machine could do or I can do it later with QEMU which will be a bit annoying.

Edit:
I added a verbose error in the above assertion and we get this. I definitely know too little to diagnose why that might be the case so any help is appreciated here 😃

            for record_type, count in record_count_by_type.items():
    >           assert (
                    count > 0
                ), f"no {record_type} records found in {record_count_by_type}"
    E           AssertionError: no FRAME_POP records found in {'ALLOCATION': 2, 'MEMORY_MAP_START': 3, 'SEGMENT_HEADER': 572, 'SEGMENT': 3047, 'NATIVE_FRAME_ID': 20, 'FRAME_PUSH': 12, 'FRAME_POP': 0, 'MEMORY_RECORD': 3, 'CONTEXT_SWITCH': 1, 'CODE_OBJECT': 11, 'TRAILER': 1}
    E           assert 0 > 0

    /Users/runner/work/memray/memray/tests/integration/test_main.py:631: AssertionError

Edit2:
I added macos-latest to the uv CI in the hopes of diagnosing something as that CI is quick but it passes (same as locally) 😞 . The above error only shows up in the wheel building CI.

[godlygeek]

The i686 segfault looks likely more likely to be a CPython bug than a Memray bug. I'll see if I can find a way to reproduce it.

The missing FRAME_POP records is... very weird. I'll probably need to reproduce that to figure out what's up. Maybe our trace function is getting uninstalled, or is somehow not firing... At least that one reproduces on x86-64! Only happening on musl is also quite odd... It might be worth adding a call to memray parse $capture_file | cat to log the contents of the file when the assertion fails. That might generate way too much data, though...

[ikrommyd]

Yeah that's what I thought about i686 as well. Perhaps Pablo is interested too if it's a cpython bug.
The FRAME_POP thing also reproduces in CI on arm64 macos but I can't reproduce locally on my m3 max :(. Also notice that the macos-latest uv CI I added also has no issue here. It feels very related to the actions environment.

[godlygeek]

Running this on an x86-64 machine reproduces the crash:

docker run -it quay.io/pypa/manylinux_2_34_i686 /usr/local/bin/python3.15t -c 'import ctypes; PyObject_Realloc = ctypes.pythonapi.PyObject_Realloc; PyObject_Realloc.argtypes = [ctypes.c_void_p, ctypes.c_size_t]; PyObject_Realloc.restype = ctypes.c_void_p; print(PyObject_Realloc(None, 984))'

[godlygeek]

I've reported the crash upstream as python/cpython#151297

[ikrommyd]

Nice, thanks! So I guess here we can just skip the i686 CI on 3.15t until there's the next python beta release with the fix?
So now only the FRAME_POP thing remains. If you manage to reproduce that, it'd be awesome.

[godlygeek]

So now only the FRAME_POP thing remains. If you manage to reproduce that, it'd be awesome.

I'm pretty sure that one's just a flaw in the tests. We only record the stacks of locations where allocations were performed. I believe we used to pop all frames at the end of the tracking run, but we definitely don't anymore. The test only directly makes allocations with one stack, but expects to see pop records for moving from one stack to another. I think it's only been working by coincidence, as we observe allocations inside the importlib machinery itself as the module is imported. With Python 3.15t, it seems like we're not seeing those, which is in fact better, but the test is failing because it no longer gets any pops. I think we just need to fix that by adding some function calls to the test to force some pops.

[ikrommyd]

If CI is happy I think this should generally be good unless you have other suggesions. --showlocals can probably be reverted cause the logs are too big now if a failure occurs unless you wanna keep it.
Would you call this a bugfix or feature in terms of news? Technically 3.15 was not being tested before so it's both?

[godlygeek]

Would you call this a bugfix or feature in terms of news? Technically 3.15 was not being tested before so it's both?

The news is always relative to the last release, and we haven't cut a release that includes support for Python 3.15 yet, so this isn't fixing a bug.

In fact, we can just add the "skip news" label to this PR, since what this PR is doing is just adding 3.15 to CI, and we already have a news entry staged for adding support for 3.15 itself. There's only two bug fixes here, one in the test suite (for the FRAME_POP thing) and one in attach (for the abi3t thing). The test suite isn't part of the released project and doesn't need news entries, and the abi3t issue doesn't affect any released version of the project and so it's not a bugfix from the PoV of our users who want to see what changed from one release to the next.

[godlygeek]

Do you want me to revert the --showlocals change or would you rather keep it? It makes the log very hard to scroll through in github actions if there is an error IMO.

I'd rather keep it. It can save us a lot of effort trying to reproduce issues in various environments if we get more info in the logs. We can always revert it later if it turns out to be more trouble than it's worth, but for now I'm thinking it'll make our lives easier.

[ikrommyd]

I think cibuildwheel still gets 3.15.0b2 until they make a release. I believe they hardcode the version and do not get the latest pre-release. I had tried it a couple of commits ago and the hack was still needed.

[godlygeek]

I think cibuildwheel still gets 3.15.0b2 until they make a release.

Ah - I checked their main and saw that they updated to a manylinux image that has 3.15.0b3, but I didn't check whether or not that had been released...

OK, presumably that'll fail. If so, I'll revert that commit and leave the hack in for now.

Other than that, I'm gonna revert the change to test all of 3.1{4,5}{t,} with uv. That job is just a quick smoke test, we're not trying to be exhaustive - the types of failures we think we might catch with that test should appear for any one of the uv interpreters, so there's no reason to test multiple.

I'm also gonna add a news entry that announces we're dropping support for 3.8, too.

After that, this is good to go!

[godlygeek]

Thanks for the contribution, @ikrommyd!

[ikrommyd]

I'm also gonna add a news entry that announces we're dropping support for 3.8, too.

Yeah on the readme badges you also claim to support 3.7 and also the readme says "Memray requires Python 3.7+". You don't have wheels in the latest release naturally due to cibuildwheel having dropped it: https://pypi.org/project/memray/#files
So I don't know if support means "wheels are provided" or "memray should theoretically compile and run" regarding 3.7 and 3.8 now . You are the judge of that so I guess if you want to keep support for those at the sdist level, you can have just CIs that compile it from source and test it outside of cibuildwheel. Or remove all mentions of 3.7 and 3.8 entirely I guess and make all pre-commit checks and linting and everything 3.9+

[godlygeek]

Yeah on the readme badges you also claim to support 3.7 and also the readme says "Memray requires Python 3.7+".

Thanks, I've updated those as well to say 3.9+ (though the badges won't update until we cut the next release, since they're based on the PyPI metadata).

Or remove all mentions of 3.7 and 3.8 entirely I guess and make all pre-commit checks and linting and everything 3.9+

I've removed a few more references to 3.7 and 3.8, though I left a few in the "attaching" docs. In theory attaching to an old version of Python might still work, so long as (an older version of) memray is installed in the environment of the process you're attaching to.

[ikrommyd]

Absolutely yeah. Just deal with my branch however you want and modify title + description accordingly :)