Compile fails on 3.14 no-GIL

[clin1234]

  
  × Building wheel for zstandard (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [25 lines of output]
      <string>:41: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
      <string>:42: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
      <frozen importlib._bootstrap>:488: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module '_cffi_backend', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
      not modified: 'build/zstandard/_cffi.c'
      generating build/zstandard/_cffi.c
      (already up-to-date)
      running bdist_wheel
      running build
      running build_py
      creating build/lib.linux-x86_64-cpython-314
      creating build/lib.linux-x86_64-cpython-314/zstandard
      copying zstandard/backend_cffi.py -> build/lib.linux-x86_64-cpython-314/zstandard
      copying zstandard/__init__.py -> build/lib.linux-x86_64-cpython-314/zstandard
      copying zstandard/__init__.pyi -> build/lib.linux-x86_64-cpython-314/zstandard
      copying zstandard/py.typed -> build/lib.linux-x86_64-cpython-314/zstandard
      running build_ext
      building 'zstandard.backend_c' extension
      creating build/temp.linux-x86_64-cpython-314
      creating build/temp.linux-x86_64-cpython-314/c-ext
      x86_64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fPIC -Ic-ext -Izstd -I/tmp/venv/include -I/usr/include/python3.14t -c c-ext/backend_c.c -o build/temp.linux-x86_64-cpython-314/c-ext/backend_c.o -DZSTD_SINGLE_FILE -DZSTDLIB_VISIBLE= -DZDICTLIB_VISIBLE= -DZSTDERRORLIB_VISIBLE= -fvisibility=hidden
      c-ext/backend_c.c: In function ‘safe_pybytes_resize’:
      c-ext/backend_c.c:316:15: error: ‘PyObject’ {aka ‘struct _object’} has no member named ‘ob_refcnt’
        316 |     if ((*obj)->ob_refcnt == 1) {
            |               ^~
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for zstandard
  Building wheel for cryptography (pyproject.toml) ... error
  error: subprocess-exited-with-error

[jimkring]

I worked around this quite a while back.

#201

I'm able to pip install git+https://github.com/jimkring/python-zstandard-nogil@676226d8adb6db384bf945ffb7ed015ba0295d4d (zstandard-0.22.0.dev0) into Python 3.13rc1 nogil (running the python3.13t.exe optional binary that the Windows Installer offers in the install options).

I tried to sync the latest indygreg/python-zstandard changes into my branch but it's not working out of the box -- it probably requires a few more tweaks to get things working again.

[ngoldbaum]

@jimkring thanks for this!

@indygreg do you think you'd be willing to publish free-threaded wheels for zstandard (assuming there are no thread-safety issues you're aware of)?

You can follow https://py-free-threading.github.io/porting/ for porting instructions.

[geertj]

Would love to see this as well. Note that a free threading build is now available in Python 3.13 which was released on October 7. Free threading is still experimental and not the default. For my specific use case it performs extremely well and so I'd like to see zstandard support this.

[dpdani]

Hi there 👋

Is there any blocker other than using Py_REFCNT?
Anything I can help with?

[ngoldbaum]

Is there any blocker other than using Py_REFCNT?

I did a short grep through the codebase and don't see any obvious issues in the C extensions like mutable static global variables.

The zstd.h header talks about using one context per thread in multithreaded environments. It looks like the ZstdDecompressor extension type stores a ZSTD_DCtx on the struct for the object, so I guess that might be possibly problematic if someone shares a decompressor across threads? I didn't look at the code carefully enough to see if this would be problematic in the GIL-enabled build as well, but assuming the extension drops the GIL it probably is an issue.

Not sure if there are other thread safety issues. Running something like pytest-run-parallel on the test suite would probably catch issues.

[ofek]

@indygreg Would you be open to one of us adding support for this and you publishing a release?

[dpdani]

I did some tests with the pytest-run-parallel package and there was only one failed test, I'll post a PR when I get the chance, probably this week.