Add support for 3.14

[ZeroIntensity]

Issue number of the reported bug or feature request: #229

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 27.65957% with 34 lines in your changes missing coverage. Please review.

Project coverage is 82.27%. Comparing base (ca83721) to head (f1cb0a0).

Files with missing lines	Patch %	Lines
src/pystack/_pystack/pycode.cpp	18.18%	18 Missing ⚠️
src/pystack/_pystack/process.cpp	30.00%	7 Missing ⚠️
src/pystack/_pystack/pyframe.cpp	54.54%	5 Missing ⚠️
src/pystack/_pystack/version.cpp	0.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #230      +/-   ##
==========================================
- Coverage   82.66%   82.27%   -0.40%     
==========================================
  Files          46       46              
  Lines        6249     6286      +37     
  Branches      458      461       +3     
==========================================
+ Hits         5166     5172       +6     
- Misses       1083     1114      +31     

Flag	Coverage Δ
cpp	82.27% <27.65%> (-0.40%)	⬇️
python_and_cython	82.27% <27.65%> (-0.40%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[godlygeek]

@ZeroIntensity I've managed to get CI working for 3.14 in #237, and rebased this PR on top of those changes. After getting CI running, there are quite a few failing tests here. Would you mind taking a look and trying to fix things?

At a glance, all of the failures seem to be related to reading local variables from Python 3.14 processes.

[ZeroIntensity]

Will take a look soon.

[ZeroIntensity]

@godlygeek Is CI setting up 3.14b0, or 3.14 straight off the CPython repo? Mark changed the tag bits for stackrefs a few days ago, so that could definitely be screwing things up.

[godlygeek]

CI is using 3.14.0 beta 1

[ZeroIntensity]

Sorry for the delay! I can't reproduce the CI failures locally on 3.14b1, but I can reproduce off CPython main, and changing the bits doesn't seem to fix it. How sure are we that it's 3.14b1?

[godlygeek]

How sure are we that it's 3.14b1?

Pretty close to 100%... The failing run logs:

platform linux -- Python 3.14.0b1, pytest-8.3.5, pluggy-1.6.0 -- /opt/hostedtoolcache/Python/3.14.0-beta.1/x64/bin/python3.14

as well as

WARNING  /opt/hostedtoolcache/Python/3.14.0-beta.1/x64/lib/python3.14/site-packages/pystack/_pystack.cpython-314-x86_64-linux-gnu.so:test_local_variables.py:18 Failed to read typeobject from address 0xb00007f8a8b71f3

and

python = ((3, 14), PosixPath('/opt/hostedtoolcache/Python/3.14.0-beta.1/x64/bin/python3.14'))

I guess there's some small chance that /opt/hostedtoolcache/Python/3.14.0-beta.1 isn't beta1, but that'd be pretty surprising...

[ZeroIntensity]

Beta 2 is out anyway, I guess. Since I can repro on that, let's switch to it.

[godlygeek]

Beta 2 is out anyway

Indeed, if I rerun the failing CI job it now uses beta2 - seems like actions/setup-python now supports it.

And with beta2 there's a bunch more failures - 118 tests fail, instead of the previous 18 failures. So it definitely does seem like something changed between beta1 and beta2 that we'll need to adjust for.

[ZeroIntensity]

There's definitely the stackref bits, but not too sure what else changed. Will investigate tomorrow.

[godlygeek]

That must be python/cpython#134286 - adding the offset of the thread-local bytecode offsets to _Py_DebugOffsets (which lets us remove the hack where we assume the offset relative to co_code_adaptive)

[ZeroIntensity]

Oh, yeah, that'd do it!

[ZeroIntensity]

Hmm, fixing that and changing the stackref bits didn't seem to fix it. Everything is failing with NotEnoughInformation: Could not gather enough information to extract the Python frame information, so I'm assuming something changed with frames between b1 and b2.

I am seeing this on the tests though: [method=ELF_DATA, blocking=True, python=3.13]. Is that python=3.13 just a flaw with the test suite, or a clue to the bug here?

[godlygeek]

Is that python=3.13 just a flaw with the test suite, or a clue to the bug here?

Hm, neither - the test suite by default will test debugging every version of the Python interpreter it can find. If you want to test with only the Python version that you're running pytest with, you can export PYTHON_TEST_VERSION=auto

But if pystack running under 3.14 fails to successfully debug Python applications running under 3.13, that would mean that things are broken in such a way that the offsets for 3.13 are wrong as well, not just the offsets for 3.14... that's concerning...

[godlygeek]

Hmm, fixing that and changing the stackref bits didn't seem to fix it.

Push your changes and I can take a look tomorrow, if you don't beat me to figuring it out.

Did you notice there were two changes to the debug offsets in python/cpython#134286 - adding tlbc_index to interpreter_frame and co_tlbc to code_object?

Does anything interesting get logged if you pystack -v remote $PID to attach to a running Python 3.14 process? If nothing jumps out with one -v, try -vv - that'll be real noisy though.

[ZeroIntensity]

The most interesting thing is:

INFO(process_remote): Failing to resolve PyInterpreterState based on PyRuntime address 0x5f88df219bc0
INFO(process_remote): Address of PyInterpreterState not found by using ELF data
INFO(process_remote): Trying to find PyInterpreterState with symbols
INFO(process_remote): Searching for PyInterpreterState based on PyRuntime address 0x5f88df219bc0
INFO(process_remote): Failing to resolve PyInterpreterState based on PyRuntime address 0x5f88df219bc0
INFO(process_remote): Address of PyInterpreterState not found by using symbols
INFO(process_remote): Scanning BSS section for PyInterpreterState
INFO(process_remote): Searching for PyInterpreterState in memory area spanning from 0x5f88df267000 to 0x5f88df2d9000
INFO(process_remote): Could not find a valid PyInterpreterState in memory area spanning from 0x5f88df267000 to 0x5f88df2d9000
INFO(process_remote): Address of PyInterpreterState not found by scanning the BSS
INFO(process_remote): Address of PyInterpreterState could not be found

[godlygeek]

But if pystack running under 3.14 fails to successfully debug Python applications running under 3.13, that would mean that things are broken in such a way that the offsets for 3.13 are wrong as well, not just the offsets for 3.14... that's concerning...

You'll kick yourself: your latest commit, eb5c104, changes the layout of the _Py_DebugOffsets in the Python3_13 namespace, instead of the layout of the _Py_DebugOffsets in the Python3_14 namespace - that's why it both broke 3.13 and didn't fix 3.14 😄

[ZeroIntensity]

Ugh, yeah, that fixed a lot of the failures. On the remaining failures, I'm seeing these two warnings:

• Ignoring debug offsets because py_code.size (64) - py_code.o_name.offset (216) < the field's size (8)
• The interpreter is shutting itself down so it is possible that no Python stack trace is available for inspection. You can still use --native-all to force displaying all the threads.

[godlygeek]

python/cpython@1822f33 affected the debug offsets, too. Adding those gets us back down to 18 failing tests - probably the same ones that were failing with beta 1... All of the failing tests are reading local variables.

[godlygeek]

Seems like that's just because the locals are now PyStackRef's too, and so we need to mask off the low bits.

[ZeroIntensity]

Thanks! Looks like we're down to 2 failed tests.

[godlygeek]

I took another swing at dealing with the layout changes to PyTupleObject, and now everything is passing except for the tests running pystack against a 3.14t interpreter. That's pretty badly broken, it seems.

[ZeroIntensity]

Which objects does pystack need to know the layout of? Free-threaded objects have a lot of internal layout changes.

[godlygeek]

ints, floats, strings, bytes, dicts, lists, tuples, code objects, frames, and a handful of others.

The first issue I tracked down is that the state field of PyASCIIObject has a different layout in free-threaded. We'll need to update AbstractProcessManager::getStringFromAddress to cope with that, but we can get past it temporarily by just doing:

diff --git a/src/pystack/_pystack/process.cpp b/src/pystack/_pystack/process.cpp
index b48a225..8c3715c 100644
--- a/src/pystack/_pystack/process.cpp
+++ b/src/pystack/_pystack/process.cpp
@@ -540,10 +540,10 @@ AbstractProcessManager::getStringFromAddress(remote_addr_t addr) const
                    << addr;
         Structure<py_unicode_v> unicode(shared_from_this(), addr);

-        Python3::_PyUnicode_State state = unicode.getField(&py_unicode_v::o_state);
-        if (state.kind != 1 || state.compact != 1) {
-            throw InvalidRemoteObject();
-        }
+        //Python3::_PyUnicode_State state = unicode.getField(&py_unicode_v::o_state);
+        //if (state.kind != 1 || state.compact != 1) {
+        //    throw InvalidRemoteObject();
+        //}

         len = unicode.getField(&py_unicode_v::o_length);
         buffer.resize(len);

That'll lead to problems if there's any non-ASCII strings, but we can circle back to that later. After applying that diff, I get a segfault (!!) in code that you changed in pystack::getLocationInfo. Adding this patch:

diff --git a/src/pystack/_pystack/pycode.cpp b/src/pystack/_pystack/pycode.cpp
index fc490e3..874fa25 100644
--- a/src/pystack/_pystack/pycode.cpp
+++ b/src/pystack/_pystack/pycode.cpp
@@ -133,6 +133,7 @@ getLocationInfo(
                 tlbc_entries + sizeof(tlbc_size),
                 tlbc_size * sizeof(uintptr_t),
                 vec.data());
+        LOG(DEBUG) << "tlbc_index=" << tlbc_index << " tlbc_size=" << tlbc_size;
         uintptr_t code_adaptive_actual = vec[tlbc_index];
         ptrdiff_t addrq =
                 (reinterpret_cast<uint16_t*>(last_instruction_index)

prints this out right before the crash:

DEBUG(process_remote): tlbc_index=-738076544 tlbc_size=0

Both of those look wrong, but it's not crashing on the first frame it processes, it's crashing on the 6th... Ah, it looks like we're trying to get the code object for a shim frame, which is just nonsense. This patch fixes that:

diff --git a/src/pystack/_pystack/pyframe.cpp b/src/pystack/_pystack/pyframe.cpp
index 3b81beb..6276952 100644
--- a/src/pystack/_pystack/pyframe.cpp
+++ b/src/pystack/_pystack/pyframe.cpp
@@ -29,10 +29,10 @@ FrameObject::FrameObject(
     if (d_is_shim) {
         LOG(DEBUG) << "Skipping over a shim frame inserted by the interpreter";
         next_frame_no = frame_no;
+    } else {
+        d_code = getCode(manager, frame);
     }

-    d_code = getCode(manager, frame);
-
     auto prev_addr = frame.getField(&py_frame_v::o_back);
     LOG(DEBUG) << std::hex << std::showbase << "Previous frame address: " << prev_addr;
     if (prev_addr) {

OK, so with those patches applied, things seem to actually work, but we will need to refactor things to allow for a different unicode state layout in free threaded vs non free threaded.

[ZeroIntensity]

IIRC, dictionaries have a different layout in FT vs non-FT too.

[godlygeek]

IIRC, dictionaries have a different layout in FT vs non-FT too.

Looks like any differences there are covered by the debug offsets. After making the two fixes I identified yesterday (not trying to extract code objects from shim frames, and handling the state flags of PyUnicode having a different layout in 3.14t) the test suite passes.

We do probably need to add some more tests to prove that the thread-local bytecode stuff is working properly, though. I doubt that any of our existing tests force specialization in a non-main thread.

[ZeroIntensity]

It shouldn't matter what thread it's in, right? As long as PyStack is testing for specializations, then it should be covering the TLBC code.

[godlygeek]

The main thread's TLBC pointer points to the co_adaptive in the code object. If we're only testing in the main thread, there's no proof that we're not just using the bytecode for the main thread unconditionally. In other words, we need a test that fails without your changes to use the thread-local bytecode array, and succeeds with them.