Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Caching of file-set hashes by local path and mtimes #700

Merged
merged 36 commits into from
Mar 17, 2024
Merged

Caching of file-set hashes by local path and mtimes #700

merged 36 commits into from
Mar 17, 2024

Commits on Feb 24, 2024

  1. added code to handle "locally-persistent-ids"

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    45117ef View commit details
    Browse the repository at this point in the history

  2. implemented persistent hash cache to avoid rehashing files

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    2b7ca50 View commit details
    Browse the repository at this point in the history

  3. touched up persistent_hash_cache test

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    04b95ff View commit details
    Browse the repository at this point in the history

  4. replaced Cache({}) with Cache() to match new proper class

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    0c865f4 View commit details
    Browse the repository at this point in the history

  5. upped resolution of mtime to nanoseconds

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    3b3fdb7 View commit details
    Browse the repository at this point in the history

  6. added sleep to various tests to ensure file mtimes are different

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    81a5108 View commit details
    Browse the repository at this point in the history

  7. added more sleeps to ensure mtimes of input files are different in tests

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    0c4b179 View commit details
    Browse the repository at this point in the history

  8. debugged setting hash cache via env var and added clean up of directory

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    615d590 View commit details
    Browse the repository at this point in the history

  9. mock mtime writing instead of adding sleeps to ensure mtimes are diff… 

    …erent
    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    55b660e View commit details
    Browse the repository at this point in the history

  10. undid overzealous black

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    5d51736 View commit details
    Browse the repository at this point in the history

  11. added missing import

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    0421f85 View commit details
    Browse the repository at this point in the history

  12. Adds platformdirs dependency and use it to store the hash cache within

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    a864b32 View commit details
    Browse the repository at this point in the history

  13. added unittests to hit exceptions in persistentcache init

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    05ca695 View commit details
    Browse the repository at this point in the history

  14. added mkdir to location converter

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    52ef03f View commit details
    Browse the repository at this point in the history

  15. debugged mkdir of persistent cache

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    0216236 View commit details
    Browse the repository at this point in the history

  16. bug fixes in persistentcache location init

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    bad261b View commit details
    Browse the repository at this point in the history

  17. Revert "mock mtime writing instead of adding sleeps to ensure mtimes … 

    …are different"

This reverts commit 0895965.
    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    2fbee2b View commit details
    Browse the repository at this point in the history

  18. skip lock files in directory clean up

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    91948f0 View commit details
    Browse the repository at this point in the history

  19. fixed clean-up bug

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    e058408 View commit details
    Browse the repository at this point in the history

  20. added mock import

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    f1ded7a View commit details
    Browse the repository at this point in the history

  21. added another sleep to trigger atime change

    @tclose
    tclose committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    bb11067 View commit details
    Browse the repository at this point in the history

Commits on Feb 29, 2024

  1. implementing @effigies suggestions

    @tclose
    tclose committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    a031ea5 View commit details
    Browse the repository at this point in the history

  2. added comments and doc strings to explain the use of the persistent c… 

    …ache
    @tclose
    tclose committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    f2f70a6 View commit details
    Browse the repository at this point in the history

  3. touched up comments

    @tclose
    tclose committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    191aa9c View commit details
    Browse the repository at this point in the history

  4. another comment touch up

    @tclose
    tclose committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    3076fea View commit details
    Browse the repository at this point in the history

  5. touch up comments again

    @tclose
    tclose committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    a094fbc View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2024

  1. Merge branch 'master' into local-cache-ids

    @tclose
    tclose authored Mar 8, 2024
    Configuration menu
    Copy the full SHA
    d27201f View commit details
    Browse the repository at this point in the history

  2. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    291f29f View commit details
    Browse the repository at this point in the history

  3. added in @djarecka's test for moving file cache locations

    @tclose
    tclose committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    0a10f6c View commit details
    Browse the repository at this point in the history

  4. updated cache initialisation

    @tclose
    tclose committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    311e3dd View commit details
    Browse the repository at this point in the history

  5. switched to use blake2b isntead of blake2s

    @tclose
    tclose committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    4827365 View commit details
    Browse the repository at this point in the history

  6. [skip ci] deleted already commented-out code

    @tclose
    tclose committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    b6799b6 View commit details
    Browse the repository at this point in the history

  7. additional doc strings for hash cache objects

    @tclose
    tclose committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    2bb86fe View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2024

  1. added test to see that persistent cache is used in the running of tasks

    @tclose
    tclose committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    1f601e1 View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2024

  1. moved persistent hash cache within "hash_cache" subdirectory of the p… 

    …ydra user cache dir
    @tclose
    tclose committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    7e60c41 View commit details
    Browse the repository at this point in the history

  2. fixed import issue

    @tclose
    tclose committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    921979c View commit details
    Browse the repository at this point in the history