podSpecPatch not applied when configured in workflowDefaults and workflow is submitted from template

[tczhao]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

workflowDefaults + podSpecPatch https://github.com/argoproj/argo-workflows/blob/a514a42/test/e2e/manifests/mixins/workflow-controller-configmap.yaml (make start also uses this config)

• works for direct submitted workflow, e.g. you can find terminationGracePeriodSeconds: 3s in the created pod
• doesn't work when workflow is submitted from template, e.g. use templateref, the terminationGracePeriodSeconds is missing, or go back to default 30s

Version

latest(a514a42)

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

any

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
time="2024-05-30T15:08:16.315Z" level=info msg="Processing workflow" Phase= ResourceVersion=536481 namespace=argo workflow=hello-wtbcr
time="2024-05-30T15:08:16.317Z" level=debug msg="Get workflowtemplates 200"
time="2024-05-30T15:08:16.318Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=hello-wtbcr)" tmpl="*v1alpha1.WorkflowStep (hello/p0#false)"
time="2024-05-30T15:08:16.318Z" level=debug msg="Found stored template" base="*v1alpha1.Workflow (namespace=argo,name=hello-wtbcr)" tmpl="*v1alpha1.WorkflowStep (hello/p0#false)"
time="2024-05-30T15:08:16.320Z" level=debug msg="Get workflowtemplates 200"
time="2024-05-30T15:08:16.320Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=hello-wtbcr)" tmpl="*v1alpha1.WorkflowStep (hello/p0#false)"
time="2024-05-30T15:08:16.320Z" level=debug msg="Found stored template" base="*v1alpha1.Workflow (namespace=argo,name=hello-wtbcr)" tmpl="*v1alpha1.WorkflowStep (hello/p0#false)"
time="2024-05-30T15:08:16.321Z" level=debug msg="Get workflowtemplates 200"
time="2024-05-30T15:08:16.321Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=hello-wtbcr)" tmpl="*v1alpha1.WorkflowStep (hello/p1#false)"
time="2024-05-30T15:08:16.322Z" level=debug msg="Found stored template" base="*v1alpha1.Workflow (namespace=argo,name=hello-wtbcr)" tmpl="*v1alpha1.WorkflowStep (hello/p1#false)"
time="2024-05-30T15:08:16.324Z" level=debug msg="Get workflowtemplates 200"
time="2024-05-30T15:08:16.326Z" level=debug msg="Get configmaps 200"
time="2024-05-30T15:08:16.326Z" level=info msg="resolved artifact repository" artifactRepositoryRef="argo/#"
time="2024-05-30T15:08:16.326Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=hello-wtbcr
time="2024-05-30T15:08:16.326Z" level=debug msg="Skipping artifact GC" namespace=argo workflow=hello-wtbcr
time="2024-05-30T15:08:16.326Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=hello-wtbcr
time="2024-05-30T15:08:16.326Z" level=debug msg="Evaluating node hello-wtbcr: template: *v1alpha1.WorkflowStep (p0), boundaryID: " namespace=argo workflow=hello-wtbcr
time="2024-05-30T15:08:16.326Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=hello-wtbcr
time="2024-05-30T15:08:16.326Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (p0)"
time="2024-05-30T15:08:16.326Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (p0)"
time="2024-05-30T15:08:16.326Z" level=debug msg="Getting the template by name: p0" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (p0)"
time="2024-05-30T15:08:16.326Z" level=debug msg="Inject a retry node for node hello-wtbcr" namespace=argo workflow=hello-wtbcr
time="2024-05-30T15:08:16.326Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo\", Name:\"hello-wtbcr\", UID:\"96fbb953-c5fd-4752-9160-0a36538ecdc2\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"536481\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowRunning' Workflow Running"
time="2024-05-30T15:08:16.326Z" level=debug msg="Initializing node hello-wtbcr: template: *v1alpha1.WorkflowStep (p0), boundaryID: " namespace=argo workflow=hello-wtbcr
time="2024-05-30T15:08:16.326Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=hello-wtbcr
time="2024-05-30T15:08:16.326Z" level=info msg="Retry node hello-wtbcr initialized Running" namespace=argo workflow=hello-wtbcr
time="2024-05-30T15:08:16.326Z" level=debug msg="Initializing node hello-wtbcr(0): template: *v1alpha1.WorkflowStep (p0), boundaryID: " namespace=argo workflow=hello-wtbcr
time="2024-05-30T15:08:16.326Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=hello-wtbcr
time="2024-05-30T15:08:16.326Z" level=info msg="Pod node hello-wtbcr-3688744399 initialized Pending" namespace=argo workflow=hello-wtbcr
time="2024-05-30T15:08:16.326Z" level=debug msg="Executing node hello-wtbcr(0) with container template: p0\n" namespace=argo workflow=hello-wtbcr
time="2024-05-30T15:08:16.327Z" level=debug namespace=argo needLocation=true workflow=hello-wtbcr
time="2024-05-30T15:08:16.327Z" level=warning msg="couldn't get boundaryTemplate through nodeName hello-wtbcr(0)" namespace=argo workflow=hello-wtbcr
time="2024-05-30T15:08:16.329Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.NodeStatus (p0)"
time="2024-05-30T15:08:16.329Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.NodeStatus (p0)"
time="2024-05-30T15:08:16.329Z" level=debug msg="Getting the template by name: p0" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.NodeStatus (p0)"
time="2024-05-30T15:08:16.329Z" level=debug msg="Creating Pod: hello-wtbcr(0) (hello-wtbcr-p0-3688744399)" namespace=argo workflow=hello-wtbcr
time="2024-05-30T15:08:16.329Z" level=debug msg="Create events 201"
time="2024-05-30T15:08:16.335Z" level=debug msg="Create pods 201"
time="2024-05-30T15:08:16.335Z" level=info msg="Created pod: hello-wtbcr(0) (hello-wtbcr-p0-3688744399)" namespace=argo workflow=hello-wtbcr
time="2024-05-30T15:08:16.335Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-wtbcr
time="2024-05-30T15:08:16.335Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-wtbcr
time="2024-05-30T15:08:16.335Z" level=debug msg="Log changes patch: {\"metadata\":{\"annotations\":{\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"labels\":{\"workflows.argoproj.io/completed\":\"false\",\"workflows.argoproj.io/phase\":\"Running\"}},\"status\":{\"artifactGCStatus\":{\"notSpecified\":true},\"artifactRepositoryRef\":{\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"accessKeySecret\":{\"key\":\"accesskey\",\"name\":\"my-minio-cred\"},\"bucket\":\"my-bucket\",\"endpoint\":\"minio:9000\",\"insecure\":true,\"secretKeySecret\":{\"key\":\"secretkey\",\"name\":\"my-minio-cred\"}}},\"configMap\":\"artifact-repositories\",\"key\":\"default-v1\",\"namespace\":\"argo\"},\"nodes\":{\"hello-wtbcr\":{\"children\":[\"hello-wtbcr-3688744399\"],\"displayName\":\"hello-wtbcr\",\"finishedAt\":null,\"id\":\"hello-wtbcr\",\"name\":\"hello-wtbcr\",\"phase\":\"Running\",\"startedAt\":\"2024-05-30T15:08:16Z\",\"templateName\":\"p0\",\"templateScope\":\"local/\",\"type\":\"Retry\"},\"hello-wtbcr-3688744399\":{\"displayName\":\"hello-wtbcr(0)\",\"finishedAt\":null,\"id\":\"hello-wtbcr-3688744399\",\"name\":\"hello-wtbcr(0)\",\"nodeFlag\":{\"retried\":true},\"phase\":\"Pending\",\"startedAt\":\"2024-05-30T15:08:16Z\",\"templateName\":\"p0\",\"templateScope\":\"local/\",\"type\":\"Pod\"}},\"phase\":\"Running\",\"startedAt\":\"2024-05-30T15:08:16Z\",\"storedWorkflowTemplateSpec\":{\"activeDeadlineSeconds\":300,\"arguments\":{},\"entrypoint\":\"p0\",\"podSpecPatch\":\"terminationGracePeriodSeconds: 3\\n\",\"templates\":[{\"container\":{\"command\":[\"sleep\",\"1\"],\"image\":\"alpine:3.7\",\"name\":\"\",\"resources\":{}},\"inputs\":{},\"metadata\":{},\"name\":\"p0\",\"outputs\":{},\"retryStrategy\":{\"limit\":3}},{\"container\":{\"command\":[\"sleep\",\"100000000\"],\"image\":\"alpine:3.7\",\"name\":\"\",\"resources\":{}},\"inputs\":{},\"metadata\":{},\"name\":\"p1\",\"outputs\":{},\"retryStrategy\":{\"limit\":3}}],\"workflowTemplateRef\":{\"name\":\"hello\"}}}}"
time="2024-05-30T15:08:16.336Z" level=info msg="Workflow to be dehydrated" Workflow Size=1522

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

[jswxstw]

Similar to #12729.