Move cache into the resource dict

[jan-janssen]

Summary by CodeRabbit

Release Notes

• New Features

  • Enhanced resource management in the executor framework, allowing users to specify resource requirements through a resource_dict.
  • Streamlined cache management by integrating cache directory handling within the resource dictionary.

• Bug Fixes

  • Improved cancellation of futures in the shutdown process for better task management.

• Documentation

  • Updated documentation strings to reflect changes in the Executor class and related functionalities.

• Tests

  • Adjusted tests to incorporate the new resource management approach, ensuring continued functionality and coverage.

[coderabbitai]

Caution

Review failed

The pull request is closed.

Walkthrough

The changes in this pull request involve the removal of the cache_directory parameter from several classes and functions within the executor library. The Executor class, FileExecutor class, and related functions have been updated to utilize a resource_dict for managing resource-related parameters, including cache directories. This adjustment simplifies the constructor signatures and internal logic, ensuring that caching is handled dynamically through the resource dictionary. Additionally, documentation has been updated to reflect these changes.

Changes

File Path	Change Summary
executorlib/init.py	Removed cache_directory from Executor class __init__ and __new__ methods; updated signatures accordingly.
executorlib/cache/executor.py	Updated FileExecutor to remove cache_directory; now uses resource_dict for cache management.
executorlib/interactive/executor.py	Removed cache_directory from create_executor function; updated logic for handling resource dictionaries.
executorlib/interactive/shared.py	Enhanced task execution and resource management; replaced cache_directory with resource_dict in several methods.
tests/test_cache_executor_pysqa_flux.py	Updated TestCacheExecutorPysqa to reflect changes in Executor instantiation, adding cache to resource_dict.
tests/test_executor_backend_mpi.py	Modified TestExecutorBackendCache to include cache in resource_dict for Executor instantiation.

Possibly related PRs

• Automatically set hostname_localhost #433: Changes in the Executor class constructor and __new__ method involve modifications to optional parameters.
• Move cache functionality to shared #434: Removal of the cache_directory parameter in the FileExecutor constructor aligns with the main PR's changes.
• Split shared cache in backend and frontend #443: Introduction of a resource_dict parameter in the ExecutorWithDependencies class relates to simplifying parameter management.
• Cache: Implement resource dict for submit() #451: The addition of a resource_dict parameter in the FileExecutor constructor directly relates to the main PR's changes.
• Use resource dict for internal communication #455: Consolidation of parameters into a resource_dict in the Executor class connects to the main PR's focus.
• Implement resource_dict for file executor #456: Modifications to the FileExecutor constructor to include a resource_dict parameter relate to the main PR's changes.
• Add option to disable_dependencies for cache #477: Addition of the disable_dependencies parameter in the Executor class enhances functionality and configurability.
• Set max_cores and max_workers to None #478: Changes to make max_workers and max_cores optional in the Executor class emphasize flexible parameter handling.
• Fix resource dict in interactive jobs #482: Adjustments in how the resource_dict is accessed in the execute_task_with_cache function relate to improving resource management.
• Fix interference bug #485: Modification to ensure a copy of the resource_dict is used in the _submit_function_to_separate_process method emphasizes robust handling of resource parameters.

🐰 "In the land of code where bunnies hop,
We’ve trimmed the clutter, made it stop!
No more cache paths to confuse the way,
Resource dicts now lead the play.
Simplified, streamlined, oh what a sight,
Hopping along, everything feels right!" 🐇

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.