Ability to pick S3 download multipart threshold, and other settings as well. #968
Labels
feature-request
A feature should be added or improved.
high-level-library
p3
This is a minor priority issue
Describe the feature
In python with Boto3, I can do the following:
A way to do this from aws-rust-sdk, or at least use the locally configured rules, for example
Use Case
Common Crawl's bucket is always rate limited (on requests) but not bandwidth, so avoiding multipart downloads is the only way to reliably download it. The time difference is between 10 seconds and 10 minutes.
Proposed Solution
No response
Other Information
No response
Acknowledgements
A note for the community
Community Note
The text was updated successfully, but these errors were encountered: