Replace minio client with aws client-s3 + lib-storage for multi-part upload #943
Conversation
…ature/replace-minio-client-with-aws-sdk
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
@mguella @wvengen @HeliosLHC can you test to see if this implementation meets your needs and works with the different S3 providers that you were using, and hopefully addresses the issues in #479 also (if its still relevant)? |
|
Tested latest changes with VersityGW, RustFS and SeaweedFS, Garage and existing Minio. |
…ucket public for now
update get-folder-size to have latest types
| async downloadFile(srcFilename: string, destFilename: string) { | ||
| let count = 0; | ||
| logger.debug("Downloading profile", { srcFilename }, "storage"); | ||
| while (true) { |
There was a problem hiding this comment.
Retries handled by the S3Client itself, with maxAttempts set in constructor
This resolves an issue where large multi-part uploads weren't working with local Minio. Bumps the version of client-s3 and lib-storage to ensure we're on a version that supports requestChecksumCalculation.
|
Tested locally and on dev with large crawls. After adding |
Agreed! It seems given that most S3-provider try to match AWS compatibility, at least there is a higher chance of things being compatible, than with existing Minio client, given where Minio is at this point. |
Extends work in #547 adds Upload via @aws-sdk/lib-storage library:
Testing: