Skip to content

refactor: reduce s3 listing#173

Merged
CL-SHLOMIKONCHA merged 8 commits intomasterfrom
reduce-s3-listing
Jul 22, 2025
Merged

refactor: reduce s3 listing#173
CL-SHLOMIKONCHA merged 8 commits intomasterfrom
reduce-s3-listing

Conversation

@shimoncohen
Copy link
Collaborator

When dealing with S3 buckets that contain a huge amount of objects, each list command takes a big toll on the servers.
This PR reduces the use of object listing when using an S3 client. Instead of lising, we now attempt to get the relevant object and handle the case when it doesn't exist.

@shimoncohen shimoncohen self-assigned this Jun 24, 2025
@shimoncohen shimoncohen added the enhancement New feature or request label Jun 24, 2025
@shimoncohen shimoncohen requested a review from asafMasa June 24, 2025 09:15
@shimoncohen shimoncohen requested review from asafMasa and razbroc June 30, 2025 13:54
Copy link
Contributor

@razbroc razbroc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Shimon.
Looks well, just a few questions.
Thank you for your service.

Copy link
Contributor

@CL-SHLOMIKONCHA CL-SHLOMIKONCHA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good - going to test it before merge on dev depoyment,
what about all those "possibly null reference." warnings?

Copy link
Contributor

@razbroc razbroc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image_720

@CL-SHLOMIKONCHA CL-SHLOMIKONCHA merged commit f47758c into master Jul 22, 2025
2 checks passed
@CL-SHLOMIKONCHA CL-SHLOMIKONCHA deleted the reduce-s3-listing branch July 22, 2025 08:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants