-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: use multipart upload method to put files larger than 5Gi to OSS. Fixes #12877 #12897
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some comments here。
Co-authored-by: AlbeeSo <suyashi1321@163.com> Co-authored-by: shuangkun <tsk2013uestc@163.com> Signed-off-by: AlbeeSo <suyashi1321@163.com>
d5edf57
to
403b39b
Compare
Signed-off-by: AlbeeSo <suyashi1321@163.com>
Done, PTAL. :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
// OSS multipart upload code reference: https://www.alibabacloud.com/help/en/oss/user-guide/multipart-upload?spm=a2c63.p38356.0.0.4ebe423fzsaPiN#section-trz-mpy-tes | ||
func multipartUpload(bucket *oss.Bucket, objectName, path string, objectSize int64) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know this is mostly taken from the Alibaba docs, but I wonder if the chunking / UploadPart
section could be parallelized?
If so, could potentially contribute that back upstream if Alibaba's docs are open source (idk?)
Backported cleanly into |
…ixes argoproj#12877 (argoproj#12897) Signed-off-by: AlbeeSo <suyashi1321@163.com> Co-authored-by: shuangkun <tsk2013uestc@163.com>
…ixes argoproj#12877 (argoproj#12897) Signed-off-by: AlbeeSo <suyashi1321@163.com> Co-authored-by: shuangkun <tsk2013uestc@163.com>
Fixes #12877
Motivation
argo-workflows/workflow/artifacts/oss/oss.go
Line 342 in 0b625a6
OSS SDK client offers several functions help upload local files to server. And PutObjectFromFile uses Simple Upload which usually causes (or always from my experience) InvalidArgument or EntityTooLarge error when uploading files larger than 5Gi.
More info plz see: https://www.alibabacloud.com/help/en/oss/you-cannot-upload-large-objects-by-using-simple-upload?spm=a3c0i.23458820.2359477120.55.7ef56e9boHbYeC
Multipart Upload is the recommended way.
Modifications
When trying to save files in OSS by Save Func of ArtifactDriver Interface:
size of capacity / 5Gi +1
parts and upload by Multipart Upload method.Verification
Plz test with a simple workflow like