Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

only load collections once #15344

Merged
merged 11 commits into from
Sep 13, 2024
Merged

only load collections once #15344

merged 11 commits into from
Sep 13, 2024

Conversation

abrookins
Copy link
Collaborator

@abrookins abrookins commented Sep 11, 2024

Any time we look up a block from a block document, we load all collections from disk, even if we've already loaded them. It seems unlikely that these will change while the process is running or that we need to support collections changing out from under a running process, so this PR loads them only once. A micro-optimization? Maybe.

Checklist

  • This pull request references any related issue by including "closes <link to issue>"
    • If no issue exists and your change is not a small fix, please create an issue first.
  • If this pull request adds new functionality, it includes unit tests that cover the changes
  • If this pull request removes docs files, it includes redirect settings in mint.json.
  • If this pull request adds functions or classes, it includes helpful docstrings.

Copy link

codspeed-hq bot commented Sep 11, 2024

CodSpeed Performance Report

Merging #15344 will not alter performance

Comparing cache-collections (78d158a) with main (c846de0)

Summary

✅ 3 untouched benchmarks

@abrookins abrookins marked this pull request as ready for review September 11, 2024 22:24
Copy link
Member

@cicdw cicdw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a style nit, otherwise love it! thank you

src/prefect/blocks/core.py Outdated Show resolved Hide resolved
@desertaxle
Copy link
Member

Would it make sense to put this optimization in the load_collections function directly? That way other consumers share the optimization.

@abrookins
Copy link
Collaborator Author

Would it make sense to put this optimization in the load_collections function directly? That way other consumers share the optimization.

It might! We'd need to cache the output of the function and return that instead. I'll throw up a version like that for you guys to check out. Nothing crazy, but also that object will live in memory, of course.

@abrookins
Copy link
Collaborator Author

Delayed a bit while I add a test suite for the plugins module but should be ready this morning PT.

@abrookins
Copy link
Collaborator Author

I pushed up some commits with the changes we discussed. However, GitHub must be "having a moment" because while the commits appear in the branch, this PR isn't reflecting them. 🙄 A problem for tomorrow.

@abrookins
Copy link
Collaborator Author

@desertaxle @cicdw This is ready again. 👍

Copy link
Member

@cicdw cicdw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@abrookins abrookins merged commit 6e9adb8 into main Sep 13, 2024
30 checks passed
@abrookins abrookins deleted the cache-collections branch September 13, 2024 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants