Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support bucket lifecycle for OSS artifact driver #5731

Merged
merged 3 commits into from
May 3, 2021
Merged

feat: Support bucket lifecycle for OSS artifact driver #5731

merged 3 commits into from
May 3, 2021

Conversation

terrytangyuan
Copy link
Member

Signed-off-by: terrytangyuan terrytangyuan@gmail.com

Checklist:

Signed-off-by: terrytangyuan <terrytangyuan@gmail.com>
@codecov
Copy link

codecov bot commented Apr 21, 2021

Codecov Report

Merging #5731 (d8bd442) into master (2b87409) will decrease coverage by 0.11%.
The diff coverage is 0.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #5731      +/-   ##
==========================================
- Coverage   46.93%   46.81%   -0.12%     
==========================================
  Files         244      244              
  Lines       15220    15344     +124     
==========================================
+ Hits         7143     7184      +41     
- Misses       7171     7253      +82     
- Partials      906      907       +1     
Impacted Files Coverage Δ
pkg/apis/workflow/v1alpha1/workflow_types.go 46.04% <ø> (ø)
workflow/artifacts/oss/oss.go 5.60% <0.00%> (-2.29%) ⬇️
workflow/metrics/server.go 12.50% <0.00%> (-4.17%) ⬇️
cmd/argoexec/commands/emissary.go 48.43% <0.00%> (-1.57%) ⬇️
workflow/controller/operator.go 71.07% <0.00%> (-0.06%) ⬇️
cmd/argo/commands/terminate.go 0.00% <0.00%> (ø)
...kg/apiclient/http1/cron-workflow-service-client.go 0.00% <0.00%> (ø)
...piclient/argo-kube-cron-workflow-service-client.go 0.00% <0.00%> (ø)
...lient/error-translating-workflow-service-client.go 0.00% <0.00%> (ø)
...ient/argo-kube-workflow-template-service-client.go 0.00% <0.00%> (ø)
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2b87409...d8bd442. Read the comment docs.

pkg/apis/workflow/v1alpha1/workflow_types.go Outdated Show resolved Hide resolved
// LifecycleRule specifies how to manage bucket's lifecycle
type LifecycleRule struct {
// MarkInfrequentAccessAfterDays is the number of days before we convert the objects in the bucket to Infrequent Access (IA) storage type
MarkInfrequentAccessAfterDays *int32 `json:"markInfrequentAccessAfterDays,omitempty" protobuf:"bytes,1,opt,name=markInfrequentAccessAfterDays"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why use pointer type?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying to make this consistent with other types in workflow.

Changed to int32.

workflow/artifacts/oss/oss.go Outdated Show resolved Hide resolved
workflow/artifacts/oss/oss.go Outdated Show resolved Hide resolved
versionExpiration := oss.LifecycleVersionExpiration{
NoncurrentDays: markDeletionAfterDays,
}
versionTransitionRule := oss.LifecycleRule{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might it be good idea to have a 1-2-1 structural mapping between wfv1.LifecycleRule and oss.LifecycleRule? that way you could have copy the fields

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's intentional so it's easier for users to just specify the number of days. This already supports the most popular use cases to save cloud storage expenses for users.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can it be extended to more complex cases easily?

Copy link
Member Author

@terrytangyuan terrytangyuan Apr 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be unnecessary to support more complex cases at this point. This should cover most of the common use cases when handling 0SS objects created by Argo. Other complicated usages such as tagging and matching on objects should be done using OSS CLI or SDK (we could easily introduce new fields in OSSLifecycleRule to support those as well when needed).

Signed-off-by: terrytangyuan <terrytangyuan@gmail.com>
Signed-off-by: terrytangyuan <terrytangyuan@gmail.com>
@alexec alexec merged commit 4a55e6f into argoproj:master May 3, 2021
@alexec alexec added this to the v3.1 milestone May 3, 2021
@terrytangyuan terrytangyuan deleted the dev-oss-lifecycle-rule branch May 3, 2021 16:09
@sarabala1979 sarabala1979 mentioned this pull request May 4, 2021
33 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants