Skip to content

Conversation

@l-qing
Copy link
Member

@l-qing l-qing commented Sep 2, 2025

Apply SetDefaults to both old and new specs in ValidateUpdate methods to handle cases where default field values have changed between API versions. This prevents upgrade scenarios from being incorrectly flagged as user modifications when only default values have changed.

Fixes issue where v1beta1 PipelineRun/TaskRun resources in production fail validation after Tekton upgrade due to timeout field structure differences between old and new default values.

  • Apply SetDefaults to old spec copy before comparison
  • Apply SetDefaults to new spec copy before comparison
  • Add comprehensive test cases for upgrade scenarios
  • Ensure actual user modifications are still caught correctly

Affects both v1 and v1beta1 API versions for PipelineRun and TaskRun validation methods.


I encountered this problem when upgrading Pipelines in my environment from version `0.56` to `0.65`.

Previously completed v1beta1 PipelineRuns in the cluster were unable to have their finalizers removed, with the error message Once the PipelineRun is complete, no updates are allowed This prevented these resources from being properly cleaned up. I wasn't actually modifying any Spec fields.

After applying this fix, the issue was resolved.

Changes

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

  • Has Docs if any changes are user facing, including updates to minimum requirements e.g. Kubernetes version bumps
  • Has Tests included if any functionality added or changed
  • pre-commit Passed
  • Follows the commit message standard
  • Meets the Tekton contributor standards (including functionality, content, code)
  • Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
  • Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings). See some examples of good release notes.
  • Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes

fix: avoid validation errors during API version upgrades where SetDefaults changes were incorrectly flagged as user modifications for PipelineRun and TaskRun resources.

Apply SetDefaults to both old and new specs in ValidateUpdate methods
to handle cases where default field values have changed between API
versions. This prevents upgrade scenarios from being incorrectly
flagged as user modifications when only default values have changed.

Fixes issue where v1beta1 PipelineRun/TaskRun resources in production
fail validation after Tekton upgrade due to timeout field structure
differences between old and new default values.

- Apply SetDefaults to old spec copy before comparison
- Apply SetDefaults to new spec copy before comparison
- Add comprehensive test cases for upgrade scenarios
- Ensure actual user modifications are still caught correctly

Affects both v1 and v1beta1 API versions for PipelineRun and TaskRun
validation methods.
@tekton-robot tekton-robot added the release-note-none Denotes a PR that doesnt merit a release note. label Sep 2, 2025
@tekton-robot tekton-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesnt merit a release note. labels Sep 2, 2025
@l-qing
Copy link
Member Author

l-qing commented Sep 2, 2025

/kind bug

@tekton-robot tekton-robot added the kind/bug Categorizes issue or PR as related to a bug. label Sep 2, 2025
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/pipelinerun_validation.go 93.8% 93.9% 0.1
pkg/apis/pipeline/v1/taskrun_validation.go 97.4% 97.5% 0.0
pkg/apis/pipeline/v1beta1/pipelinerun_validation.go 95.7% 95.7% 0.1
pkg/apis/pipeline/v1beta1/taskrun_validation.go 98.1% 98.1% 0.0

@tekton-robot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: danielfbm
To complete the pull request process, please assign afrittoli after the PR has been reviewed.
You can assign the PR to them by writing /assign @afrittoli in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@l-qing l-qing closed this Sep 3, 2025
@github-project-automation github-project-automation bot moved this from Todo to Done in Tekton Community Roadmap Sep 3, 2025
@l-qing l-qing reopened this Sep 3, 2025
@github-project-automation github-project-automation bot moved this from Done to In Progress in Tekton Community Roadmap Sep 3, 2025
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/pipelinerun_validation.go 93.8% 93.9% 0.1
pkg/apis/pipeline/v1/taskrun_validation.go 97.4% 97.5% 0.0
pkg/apis/pipeline/v1beta1/pipelinerun_validation.go 95.7% 95.7% 0.1
pkg/apis/pipeline/v1beta1/taskrun_validation.go 98.1% 98.1% 0.0

@l-qing
Copy link
Member Author

l-qing commented Sep 3, 2025

Does it seem like failure is related to the environment?

It looks like the gitea failure to start, and consequently the e2e failure, are entirely due to pod/gitea-memcached being stuck in an ImagePullBackOff state.

2025-09-03T07:27:37.4834234Z     resolvers_test.go:598: Error waiting for gitea-0 pod to be running: timed out waiting for the condition
2025-09-03T07:27:37.4834572Z     panic.go:636: ############################################
2025-09-03T07:27:37.4834793Z     panic.go:636: ### Dumping objects from arendelle-vb6vl ###
2025-09-03T07:27:37.4834982Z     panic.go:636: ############################################
2025-09-03T07:27:37.4835092Z     panic.go:636: 
2025-09-03T07:27:37.4835424Z     panic.go:636: #####################################################
2025-09-03T07:27:37.4835677Z     panic.go:636: ### Dumping logs from Pods in the arendelle-vb6vl ###
2025-09-03T07:27:37.4835877Z     panic.go:636: #####################################################
2025-09-03T07:27:37.4836090Z     panic.go:636: Not deleting namespace arendelle-vb6vl
2025-09-03T07:27:37.4836226Z --- FAIL: TestGitResolver_API_Identifier (606.14s)
2025-09-03T07:27:37.4836326Z === NAME  TestGitResolver_API
2025-09-03T07:27:37.4836733Z     resolvers_test.go:536: Error waiting for gitea-0 pod to be running: timed out waiting for the condition
2025-09-03T07:27:37.4836932Z     panic.go:636: ############################################
2025-09-03T07:27:37.4837158Z     panic.go:636: ### Dumping objects from arendelle-8w5kj ###
2025-09-03T07:27:37.4837354Z     panic.go:636: ############################################
2025-09-03T07:27:37.4837571Z     panic.go:636: 
2025-09-03T07:27:37.4837830Z     panic.go:636: #####################################################
2025-09-03T07:27:37.4838166Z     panic.go:636: ### Dumping logs from Pods in the arendelle-8w5kj ###
2025-09-03T07:27:37.4838454Z     panic.go:636: #####################################################
2025-09-03T07:27:37.4838692Z     panic.go:636: Not deleting namespace arendelle-8w5kj
2025-09-03T07:27:37.4838887Z --- FAIL: TestGitResolver_API (606.32s)
2025-09-03T07:27:37.4838996Z === NAME  TestGitResolver_HTTPAuth
2025-09-03T07:27:37.4839515Z     resolvers_test.go:483: Error waiting for gitea-0 pod to be running: timed out waiting for the condition
2025-09-03T07:27:37.4839813Z     panic.go:636: ############################################
2025-09-03T07:27:37.4840032Z     panic.go:636: ### Dumping objects from arendelle-f84mz ###
2025-09-03T07:27:37.4840214Z     panic.go:636: ############################################
2025-09-03T07:27:37.4840325Z     panic.go:636: 
2025-09-03T07:27:37.4840645Z     panic.go:636: #####################################################
2025-09-03T07:27:37.4840889Z     panic.go:636: ### Dumping logs from Pods in the arendelle-f84mz ###
2025-09-03T07:27:37.4841089Z     panic.go:636: #####################################################
2025-09-03T07:27:37.4841286Z     panic.go:636: Not deleting namespace arendelle-f84mz
2025-09-03T07:27:37.4841416Z --- FAIL: TestGitResolver_HTTPAuth (606.29s)
2025-09-03T07:27:37.4841493Z FAIL
2025-09-03T07:36:32.6526101Z ***************************************
2025-09-03T07:36:32.6526524Z ***         E2E TEST FAILED         ***
2025-09-03T07:36:32.6527174Z ***    Start of information dump    ***
2025-09-03T07:36:32.6527654Z ***************************************
2025-09-03T07:36:32.6528216Z >>> All resources:
2025-09-03T07:36:32.8444245Z NAMESPACE                    NAME                                                    READY   STATUS             RESTARTS        AGE
2025-09-03T07:36:32.8444901Z arendelle-8w5kj              pod/gitea-0                                             0/1     CrashLoopBackOff   8 (2m27s ago)   19m
2025-09-03T07:36:32.8445722Z arendelle-8w5kj              pod/gitea-memcached-599fcbdf58-z4qds                    0/1     ImagePullBackOff   0               19m
2025-09-03T07:36:32.8446438Z arendelle-8w5kj              pod/gitea-postgresql-0                                  1/1     Running            0               19m
2025-09-03T07:36:32.8447039Z arendelle-f84mz              pod/gitea-0                                             0/1     CrashLoopBackOff   8 (118s ago)    19m
2025-09-03T07:36:32.8447824Z arendelle-f84mz              pod/gitea-memcached-599fcbdf58-mkf7z                    0/1     ImagePullBackOff   0               19m
2025-09-03T07:36:32.8448454Z arendelle-f84mz              pod/gitea-postgresql-0                                  1/1     Running            0               19m
2025-09-03T07:36:32.8449243Z arendelle-q8ps4              pod/authenticating-git-commands-pod                     1/4     NotReady           0               11s
2025-09-03T07:36:32.8449933Z arendelle-rxjjh              pod/authenticating-git-commands-pod                     1/4     Terminating        0               19s
2025-09-03T07:36:32.8451263Z arendelle-vb6vl              pod/gitea-0                                             0/1     CrashLoopBackOff   8 (2m40s ago)   19m
2025-09-03T07:36:32.8452031Z arendelle-vb6vl              pod/gitea-memcached-599fcbdf58-s2ktc                    0/1     ImagePullBackOff   0               19m
2025-09-03T07:36:32.8452934Z arendelle-vb6vl              pod/gitea-postgresql-0                                  1/1     Running            0               19m

@l-qing
Copy link
Member Author

l-qing commented Sep 3, 2025

Duplicate of #9011

@l-qing l-qing closed this Sep 3, 2025
@github-project-automation github-project-automation bot moved this from In Progress to Done in Tekton Community Roadmap Sep 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Categorizes issue or PR as related to a bug. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants