Open
Description
Now that #1767 is closed, tokenizers builds under the free-threaded build.
aiohttp
is an (indirect) dev dependency, and they're still working on support, so that might be a complication.
The extension module also still needs to be marked as compatible with the free-threaded build:
<frozen importlib._bootstrap>:488
<frozen importlib._bootstrap>:488: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'tokenizers.tokenizers', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
If I do PYTHON_GIL=0 make test
, the tests all pass. That said, I don't see any explicitly multithreaded tests, so it might be a good idea to extend the multiprocessing tests to also use thread pools, to the extent that's feasible. It might also be a good idea to try pytest-run-parallel to shake out any use of global state in the tests or implementation.
Metadata
Metadata
Assignees
Labels
No labels