Skip to content

Add cp313t tests, mark extension modules as compatible, and ship free-threaded wheels #1784

Open
@ngoldbaum

Description

@ngoldbaum

Now that #1767 is closed, tokenizers builds under the free-threaded build.

aiohttp is an (indirect) dev dependency, and they're still working on support, so that might be a complication.

The extension module also still needs to be marked as compatible with the free-threaded build:

<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'tokenizers.tokenizers', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.

If I do PYTHON_GIL=0 make test, the tests all pass. That said, I don't see any explicitly multithreaded tests, so it might be a good idea to extend the multiprocessing tests to also use thread pools, to the extent that's feasible. It might also be a good idea to try pytest-run-parallel to shake out any use of global state in the tests or implementation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions