Skip to content

Conversation

@ellisms
Copy link
Contributor

@ellisms ellisms commented Mar 4, 2024


Closes #35449
Cached an S3 resource property for functions still using resources. Ran a simple loop of hook.get_bucket() for 5 buckets and captured memory data with memray. Memory usage went from 34.5mb without caching to 11.5mb with cached property. This PR doesn't address removing the use of Resources, since it would cause a breaking change for anyone directly calling hook.get_bucket() or hook.get_key().
Cleaned up some of the other typing code and imports.

@boring-cyborg boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Mar 4, 2024
@eladkal
Copy link
Contributor

eladkal commented Mar 4, 2024

This PR doesn't address removing the use of Resources, since it would cause a breaking change for anyone directly calling hook.get_bucket() or hook.get_key().

If we think it's right to remove we can introduce a breaking change release.

@eladkal eladkal requested review from ferruzzi and vincbeck March 4, 2024 20:48
Copy link
Contributor

@ferruzzi ferruzzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a good start. I'd love to see Resources replaced with Clients some day to bring it in line with the other hooks, but that caching is nice and I like the cleanup work.

@Taragolis
Copy link
Contributor

It's a good start. I'd love to see Resources replaced with Clients

As far a I could see this PR do not replace boto3.resource by a boto3.client, in general it should reduce a bit of memory usage

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reduce memory usage in S3Hook

7 participants