-
Notifications
You must be signed in to change notification settings - Fork 1.8k
feat: override task timeouts in pipelineruns #8636
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
/kind feature |
The following is the coverage report on the affected files.
|
41cbb8b
to
1cb87f9
Compare
The following is the coverage report on the affected files.
|
1cb87f9
to
d5ca69c
Compare
The following is the coverage report on the affected files.
|
d5ca69c
to
0418ce7
Compare
0418ce7
to
5b082b1
Compare
38db1f0
to
440d481
Compare
The following is the coverage report on the affected files.
|
The following is the coverage report on the affected files.
|
The following is the coverage report on the affected files.
|
9c66620
to
ee773fc
Compare
The following is the coverage report on the affected files.
|
ee773fc
to
c19714c
Compare
The following is the coverage report on the affected files.
|
The following is the coverage report on the affected files.
|
cc @vdemeester @afrittoli @aThorp96 ready for review |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cc @tektoncd/core-maintainers @afrittoli
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: vdemeester The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good job @waveywaves ! 😸
My comments are nice to haves, its all implemented and tested well.
I think this PR would benefit from a documentation entry in "Specifying taskRunSpecs". You could reuse the explanation from #7752.
Let me know if I can support.
@@ -87,6 +87,14 @@ func (ps *PipelineRunSpec) Validate(ctx context.Context) (errs *apis.FieldError) | |||
// Validate propagated parameters | |||
errs = errs.Also(ps.validateInlineParameters(ctx)) | |||
|
|||
// Validate task run specs first | |||
for idx, trs := range ps.TaskRunSpecs { | |||
if trs.Timeout != nil && trs.Timeout.Duration < 0 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this check would fit very well into validateTaskRunSpec()
in line 338 after the trs.PodTeplate
check. Then you would not need the loop and the call to validateTaskRunSpec
in line 95 here.
I think you can also reuse validateTimeoutDuration
and remove the if statement.
@@ -103,6 +111,20 @@ func (ps *PipelineRunSpec) Validate(ctx context.Context) (errs *apis.FieldError) | |||
defaultTimeout := time.Duration(config.FromContextOrDefaults(ctx).Defaults.DefaultTimeoutMinutes) | |||
errs = errs.Also(ps.validatePipelineTimeout(defaultTimeout, "should be <= default timeout duration")) | |||
} | |||
|
|||
if len(ps.TaskRunSpecs) > 0 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would extract this block into a function like the entries above and below do, it could be named: validateTaskRunSpecTimeouts
and use the guard pattern, e.g.:
if len(ps.TaskRunSpecs) == 0 {
return errs
}
//...
pipelineTimeout = &metav1.Duration{Duration: time.Duration(config.FromContextOrDefaults(ctx).Defaults.DefaultTimeoutMinutes) * time.Minute} | ||
} | ||
for _, taskRunSpec := range ps.TaskRunSpecs { | ||
if taskRunSpec.Timeout != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This would also be a good place for a guard clause (or two), then the next if statement does not need to be nested :)
if taskRunSpec.Timeout == nil {
continue
}
if pipelineTimeout.Duration == config.NoTimeoutDuration {
continue
}
if taskRunSpec.Timeout.Duration > pipelineTimeout.Duration {
errs = ...
}
// ...
@@ -1352,6 +1391,26 @@ func TestPipelineRun_InvalidTimeouts(t *testing.T) { | |||
}, | |||
}, | |||
want: apis.ErrInvalidValue(`0s (no timeout) should be <= pipeline duration`, "spec.timeouts.finally"), | |||
}, { | |||
name: "task-specific timeout exceeds pipeline timeout", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You could add another valid test covering the case when the pipelineTimeout.Duration == config.NoTimeoutDuration
, then there should be no error when task timeout exceeds pipeline timeout.
298fd76
to
7e0706d
Compare
The following is the coverage report on the affected files.
|
The following is the coverage report on the affected files.
|
The following is the coverage report on the affected files.
|
The following is the coverage report on the affected files.
|
The following is the coverage report on the affected files.
|
/retest |
``` | ||
{{% /tab %}} | ||
{{< /tabs >}} | ||
|
||
If used with this `Pipeline`, `build-task` will use the task specific `PodTemplate` (where `nodeSelector` has `disktype` equal to `ssd`) | ||
along with `securityContext` from the `pipelineRun.spec.podTemplate`. | ||
along with `securityContext` from the `pipelineRun.spec.podTemplate`. The task will also have a specific timeout of 1 hour and 30 minutes. This overrides any existing timeout already defined by the pipelineTask as well. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
along with `securityContext` from the `pipelineRun.spec.podTemplate`. The task will also have a specific timeout of 1 hour and 30 minutes. This overrides any existing timeout already defined by the pipelineTask as well. | |
along with `securityContext` from the `pipelineRun.spec.podTemplate`. The task will also have a specific timeout of 1 hour and 30 minutes. This overrides any existing timeout already defined by the pipelineTask as well, though the specified `pipelineRun.spec.timeouts.tasks` will still take precedence. |
@@ -656,6 +656,11 @@ type PipelineTaskRunSpec struct { | |||
|
|||
// Compute resources to use for this TaskRun | |||
ComputeResources *corev1.ResourceRequirements `json:"computeResources,omitempty"` | |||
|
|||
// Time after which the TaskRun times out. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// Time after which the TaskRun times out. | |
// Time after which the TaskRun times out. Overrides the timeout specified | |
// on the Task's spec if specified. Takes lower precedence to PipelineRun's | |
// `spec.timeouts.tasks` |
pipelineTimeout := ps.Timeouts.Pipeline | ||
if pipelineTimeout == nil { | ||
pipelineTimeout = &metav1.Duration{Duration: time.Duration(config.FromContextOrDefaults(ctx).Defaults.DefaultTimeoutMinutes) * time.Minute} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe this should be comparing ps.Timeouts.Tasks
, and if that is not specified then falling back to ps.Timeouts.Pipeline
. Since ps.Timeouts.Tasks
is already enforced to be less than or equal to ps.Timeouts.Pipeline
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you think about something like this? This implementation adds a couple things:
- Creates a central timeout validation function which falls back to the default if no timeout is supplied, and if the timeout is negative then it of course fails. It DRYs up the validation for both TaskRuns schema versions and could also be used on the PipelineRun.spec.timeouts as well
- PipelineRunTask timeout check now ensures that the timeout is
0 < timeout < maxTimeout
wheremaxTimeout
is the first present in:pr.spec.timeouts.tasks || pr.spec.timeouts.pipeline || defaultTimeout
,
// in the PipelineValidation
func validatePipelineTaskRunSpecTimeouts(ps *PipelineRunSpec, ctx context.Context) *apis.FieldError {
cfg := config.FromContextOrDefaults(ctx)
var errs *apis.FieldError
maxTimeout := &metav1.Duration{Duration: time.Duration(cfg.Defaults.DefaultTimeoutMinutes) * time.Minute}
if ps.Timeouts != nil {
// Loop over max-timeouts in order of precedence to find most applicable
for _, timeout := range []*metav1.Duration{ps.Timeouts.Tasks, ps.Timeouts.Pipeline} {
if timeout != nil {
timeout, err := validateTimeout(ps.Timeouts.Tasks, cfg.Defaults.DefaultTimeoutMinutes)
if err != nil {
return errs.Also(err)
}
maxTimeout = timeout
break
}
}
}
if maxTimeout.Duration == config.NoTimeoutDuration || len(ps.TaskRunSpecs) == 0 {
return errs
}
for _, taskRunSpec := range ps.TaskRunSpecs {
taskRunTimeout, err := validateTimeout(taskRunSpec.Timeout, cfg.Defaults.DefaultTimeoutMinutes)
if err != nil {
errs = errs.Also(err)
} else if taskRunTimeout.Duration > maxTimeout.Duration {
errs = errs.Also(apis.ErrInvalidValue(fmt.Sprintf("%s should be <= pipeline tasks duration %s", &taskRunTimeout.Duration, maxTimeout.Duration), "taskRunSpecs["+taskRunSpec.PipelineTaskName+"].timeout"))
}
}
return errs
}
// In the TaskRun validation
func (ts *TaskRunSpec) Validate(ctx context.Context) (errs *apis.FieldError) {
cfg := config.FromContextOrDefaults(ctx)
// ...
_, timeoutErr := validateTimeout(ts.Timeout, cfg.Defaults.DefaultTimeoutMinutes)
errs = errs.Also(timeoutErr)
// ...
}
// In some resource-agnostic location
func validateTimeout(timeout *metav1.Duration, defaultTimeoutMinutes int) (*metav1.Duration, *apis.FieldError) {
if timeout == nil {
return &metav1.Duration{Duration: time.Duration(defaultTimeoutMinutes) * time.Minute}, nil
}
if timeout.Duration < 0 {
return nil, apis.ErrInvalidValue(timeout.Duration.String()+" should be >= 0", "timeout")
}
return timeout, nil
}
@@ -4474,6 +4474,11 @@ spec: | |||
description: The name of the Step to override. | |||
type: string | |||
x-kubernetes-list-type: atomic | |||
timeout: | |||
description: |- | |||
Time after which the TaskRun times out. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Time after which the TaskRun times out. | |
Time after which the TaskRun times out. | |
Must be less than or equal to `pipelineRun.spec.timeouts.tasks`. |
// validateTaskRunSpecTimeouts validates that the timeout is a valid duration and is not negative. | ||
func validateTaskRunSpecTimeouts(timeout *metav1.Duration) (errs *apis.FieldError) { | ||
if timeout == nil { | ||
return nil | ||
} | ||
|
||
if timeout.Duration < 0 { | ||
return errs.Also(apis.ErrInvalidValue(timeout.Duration.String()+" should be >= 0", "timeout")) | ||
} | ||
|
||
return nil | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be an antipattern to use the v1
implementation of validateTaskRunSpecTimeouts
here?
@@ -331,3 +341,33 @@ func validateTaskRunSpec(ctx context.Context, trs PipelineTaskRunSpec) (errs *ap | |||
} | |||
return errs | |||
} | |||
|
|||
// validatePipelineTaskRunSpecTimeouts validates timeouts for TaskRunSpecs in a PipelineRun | |||
func validatePipelineTaskRunSpecTimeouts(ps *PipelineRunSpec, ctx context.Context) *apis.FieldError { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: conventionally the context is usually the first argument
pipelineTimeout := ps.Timeouts.Pipeline | ||
if pipelineTimeout == nil { | ||
pipelineTimeout = &metav1.Duration{Duration: time.Duration(config.FromContextOrDefaults(ctx).Defaults.DefaultTimeoutMinutes) * time.Minute} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you think about something like this? This implementation adds a couple things:
- Creates a central timeout validation function which falls back to the default if no timeout is supplied, and if the timeout is negative then it of course fails. It DRYs up the validation for both TaskRuns schema versions and could also be used on the PipelineRun.spec.timeouts as well
- PipelineRunTask timeout check now ensures that the timeout is
0 < timeout < maxTimeout
wheremaxTimeout
is the first present in:pr.spec.timeouts.tasks || pr.spec.timeouts.pipeline || defaultTimeout
,
// in the PipelineValidation
func validatePipelineTaskRunSpecTimeouts(ps *PipelineRunSpec, ctx context.Context) *apis.FieldError {
cfg := config.FromContextOrDefaults(ctx)
var errs *apis.FieldError
maxTimeout := &metav1.Duration{Duration: time.Duration(cfg.Defaults.DefaultTimeoutMinutes) * time.Minute}
if ps.Timeouts != nil {
// Loop over max-timeouts in order of precedence to find most applicable
for _, timeout := range []*metav1.Duration{ps.Timeouts.Tasks, ps.Timeouts.Pipeline} {
if timeout != nil {
timeout, err := validateTimeout(ps.Timeouts.Tasks, cfg.Defaults.DefaultTimeoutMinutes)
if err != nil {
return errs.Also(err)
}
maxTimeout = timeout
break
}
}
}
if maxTimeout.Duration == config.NoTimeoutDuration || len(ps.TaskRunSpecs) == 0 {
return errs
}
for _, taskRunSpec := range ps.TaskRunSpecs {
taskRunTimeout, err := validateTimeout(taskRunSpec.Timeout, cfg.Defaults.DefaultTimeoutMinutes)
if err != nil {
errs = errs.Also(err)
} else if taskRunTimeout.Duration > maxTimeout.Duration {
errs = errs.Also(apis.ErrInvalidValue(fmt.Sprintf("%s should be <= pipeline tasks duration %s", &taskRunTimeout.Duration, maxTimeout.Duration), "taskRunSpecs["+taskRunSpec.PipelineTaskName+"].timeout"))
}
}
return errs
}
// In the TaskRun validation
func (ts *TaskRunSpec) Validate(ctx context.Context) (errs *apis.FieldError) {
cfg := config.FromContextOrDefaults(ctx)
// ...
_, timeoutErr := validateTimeout(ts.Timeout, cfg.Defaults.DefaultTimeoutMinutes)
errs = errs.Also(timeoutErr)
// ...
}
// In some resource-agnostic location
func validateTimeout(timeout *metav1.Duration, defaultTimeoutMinutes int) (*metav1.Duration, *apis.FieldError) {
if timeout == nil {
return &metav1.Duration{Duration: time.Duration(defaultTimeoutMinutes) * time.Minute}, nil
}
if timeout.Duration < 0 {
return nil, apis.ErrInvalidValue(timeout.Duration.String()+" should be >= 0", "timeout")
}
return timeout, nil
}
docs: timeout definition per PipelineRun taskRunSpecs cleanup: refactor timeouts validation for pipelineRun taskRunSpecs test: 0s pipeline timeout but long taskrun timeout
7e0706d
to
437a2bd
Compare
The following is the coverage report on the affected files.
|
Such a useful feature, should we add a release note to recommend it to everyone? 😆 |
Changes
/fixes #7752
Submitter Checklist
As the author of this PR, please check off the items in this checklist:
/kind <type>
. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tepRelease Notes