Skip to content

feature(threading): use explicit/configurable thread pool executor #2327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 10, 2024

Conversation

jhamman
Copy link
Member

@jhamman jhamman commented Oct 10, 2024

Previously, Zarr was using a home rolled to_thread function to offload work to an implicit executor. This PR changes this in three main ways:

  1. Use asyncio.to_thread instead of the home rolled version
  2. Setup a configurable ThreadPoolExecutor alongside the IOthread and IOloop. The number of threads can be configured using threading.max_workers.
  3. Provide a teardown function (zarr.core.sync.cleanup_resources) to explicitly cleanup resources created by Zarr at runtime. This function is executed at shutdown (via atexit) and can also be triggered manually (e.g. in the dask test suite)

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/tutorial.rst
  • Changes documented in docs/release.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

Copy link
Contributor

@d-v-b d-v-b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like a nice improvement!

@jhamman jhamman merged commit b8f6cb9 into zarr-developers:v3 Oct 10, 2024
20 checks passed
@jhamman jhamman deleted the fix/thread-cleanup branch October 10, 2024 20:09
@jhamman jhamman added the V3 label Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants