Variables not re-calculated in pod spec patch when re-submitting workflow

[RoryDoherty]

Checklist

• Double-checked my configuration.
• Tested using the latest version.
• Used the Emissary executor.

Summary

What happened/what you expected to happen?
Ran a workflow which uses {{workflow.name}} to upload artifacts to a unique location
Workflow name is stored in an env var in podSpecPatch so that it is available to all pods that are created

I then resubmitted the same workflow and noticed the env var with the workflow name was not updated in the new pod

What version are you running?
v3.3.5

Diagnostics

Paste the smallest workflow that reproduces the bug. We must be able to run the workflow.

If you run the below workflow once and then resubmit it via the UI you will see the issue

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: demo-
spec:
  entrypoint: demo
  templateDefaults:
    podSpecPatch: |-
      containers:
        - name: main
          env:
            - name: workflow_name
              value: {{workflow.name}}
  templates:
    - name: demo
      dag:
        tasks:
          - name: workflow-test
            template: run-test

    - name: run-test
      script:
        image: ubuntu:18.04
        imagePullPolicy: Always
        command: [bash]
        workingDir: "/src"
        source: |
          echo ${workflow_name}

Logs from the workflow controller:

Log from first run:

time="2022-05-24T17:23:25.547Z" level=info msg="Processing workflow" namespace=argoci workflow=demo-7vm76
time="2022-05-24T17:23:25.554Z" level=info msg="Updated phase  -> Running" namespace=argoci workflow=demo-7vm76
time="2022-05-24T17:23:25.554Z" level=info msg="DAG node demo-7vm76 initialized Running" namespace=argoci workflow=demo-7vm76
time="2022-05-24T17:23:25.554Z" level=info msg="All of node demo-7vm76.workflow-test dependencies [] completed" namespace=argoci workflow=demo-7vm76
time="2022-05-24T17:23:25.554Z" level=info msg="Pod node demo-7vm76-3435674468 initialized Pending" namespace=argoci workflow=demo-7vm76
time="2022-05-24T17:23:25.582Z" level=info msg="Created pod: demo-7vm76.workflow-test (demo-7vm76-3435674468)" namespace=argoci workflow=demo-7vm76
time="2022-05-24T17:23:25.582Z" level=info msg="TaskSet Reconciliation" namespace=argoci workflow=demo-7vm76
time="2022-05-24T17:23:25.582Z" level=info msg=reconcileAgentPod namespace=argoci workflow=demo-7vm76
time="2022-05-24T17:23:25.595Z" level=info msg="Workflow update successful" namespace=argoci phase=Running resourceVersion=387173309 workflow=demo-7vm76
time="2022-05-24T17:23:35.584Z" level=info msg="Processing workflow" namespace=argoci workflow=demo-7vm76
time="2022-05-24T17:23:35.584Z" level=info msg="Task-result reconciliation" namespace=argoci numObjs=0 workflow=demo-7vm76
time="2022-05-24T17:23:35.585Z" level=info msg="node changed" new.message= new.phase=Succeeded new.progress=0/1 nodeID=demo-7vm76-3435674468 old.message= old.phase=Pending old.progress=0/1
time="2022-05-24T17:23:35.585Z" level=info msg="Outbound nodes of demo-7vm76 set to [demo-7vm76-3435674468]" namespace=argoci workflow=demo-7vm76
time="2022-05-24T17:23:35.585Z" level=info msg="node demo-7vm76 phase Running -> Succeeded" namespace=argoci workflow=demo-7vm76
time="2022-05-24T17:23:35.585Z" level=info msg="node demo-7vm76 finished: 2022-05-24 17:23:35.585469398 +0000 UTC" namespace=argoci workflow=demo-7vm76
time="2022-05-24T17:23:35.585Z" level=info msg="Checking daemoned children of demo-7vm76" namespace=argoci workflow=demo-7vm76
time="2022-05-24T17:23:35.585Z" level=info msg="TaskSet Reconciliation" namespace=argoci workflow=demo-7vm76
time="2022-05-24T17:23:35.585Z" level=info msg=reconcileAgentPod namespace=argoci workflow=demo-7vm76
time="2022-05-24T17:23:35.585Z" level=info msg="Updated phase Running -> Succeeded" namespace=argoci workflow=demo-7vm76
time="2022-05-24T17:23:35.585Z" level=info msg="Marking workflow completed" namespace=argoci workflow=demo-7vm76
time="2022-05-24T17:23:35.585Z" level=info msg="Checking daemoned children of " namespace=argoci workflow=demo-7vm76
time="2022-05-24T17:23:35.591Z" level=info msg="cleaning up pod" action=deletePod key=argoci/demo-7vm76-1340600742-agent/deletePod
time="2022-05-24T17:23:35.598Z" level=info msg="Workflow update successful" namespace=argoci phase=Succeeded resourceVersion=387173484 workflow=demo-7vm76
time="2022-05-24T17:23:35.608Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argoci/demo-7vm76-3435674468/labelPodCompleted

Log from rerun:

time="2022-05-24T17:23:57.305Z" level=info msg="Processing workflow" namespace=argoci workflow=demo-mvdl5
time="2022-05-24T17:23:57.313Z" level=info msg="Updated phase  -> Running" namespace=argoci workflow=demo-mvdl5
time="2022-05-24T17:23:57.313Z" level=info msg="DAG node demo-mvdl5 initialized Running" namespace=argoci workflow=demo-mvdl5
time="2022-05-24T17:23:57.313Z" level=info msg="All of node demo-mvdl5.workflow-test dependencies [] completed" namespace=argoci workflow=demo-mvdl5
time="2022-05-24T17:23:57.313Z" level=info msg="Pod node demo-mvdl5-4274864227 initialized Pending" namespace=argoci workflow=demo-mvdl5
time="2022-05-24T17:23:57.349Z" level=info msg="Created pod: demo-mvdl5.workflow-test (demo-mvdl5-4274864227)" namespace=argoci workflow=demo-mvdl5
time="2022-05-24T17:23:57.350Z" level=info msg="TaskSet Reconciliation" namespace=argoci workflow=demo-mvdl5
time="2022-05-24T17:23:57.350Z" level=info msg=reconcileAgentPod namespace=argoci workflow=demo-mvdl5
time="2022-05-24T17:23:57.365Z" level=info msg="Workflow update successful" namespace=argoci phase=Running resourceVersion=387173887 workflow=demo-mvdl5
time="2022-05-24T17:24:07.349Z" level=info msg="Processing workflow" namespace=argoci workflow=demo-mvdl5
time="2022-05-24T17:24:07.349Z" level=info msg="Task-result reconciliation" namespace=argoci numObjs=0 workflow=demo-mvdl5
time="2022-05-24T17:24:07.349Z" level=info msg="node changed" new.message= new.phase=Succeeded new.progress=0/1 nodeID=demo-mvdl5-4274864227 old.message= old.phase=Pending old.progress=0/1
time="2022-05-24T17:24:07.350Z" level=info msg="Outbound nodes of demo-mvdl5 set to [demo-mvdl5-4274864227]" namespace=argoci workflow=demo-mvdl5
time="2022-05-24T17:24:07.350Z" level=info msg="node demo-mvdl5 phase Running -> Succeeded" namespace=argoci workflow=demo-mvdl5
time="2022-05-24T17:24:07.350Z" level=info msg="node demo-mvdl5 finished: 2022-05-24 17:24:07.350116856 +0000 UTC" namespace=argoci workflow=demo-mvdl5
time="2022-05-24T17:24:07.350Z" level=info msg="Checking daemoned children of demo-mvdl5" namespace=argoci workflow=demo-mvdl5
time="2022-05-24T17:24:07.350Z" level=info msg="TaskSet Reconciliation" namespace=argoci workflow=demo-mvdl5
time="2022-05-24T17:24:07.350Z" level=info msg=reconcileAgentPod namespace=argoci workflow=demo-mvdl5
time="2022-05-24T17:24:07.350Z" level=info msg="Updated phase Running -> Succeeded" namespace=argoci workflow=demo-mvdl5
time="2022-05-24T17:24:07.350Z" level=info msg="Marking workflow completed" namespace=argoci workflow=demo-mvdl5
time="2022-05-24T17:24:07.350Z" level=info msg="Checking daemoned children of " namespace=argoci workflow=demo-mvdl5
time="2022-05-24T17:24:07.355Z" level=info msg="cleaning up pod" action=deletePod key=argoci/demo-mvdl5-1340600742-agent/deletePod
time="2022-05-24T17:24:07.363Z" level=info msg="Workflow update successful" namespace=argoci phase=Succeeded resourceVersion=387174057 workflow=demo-mvdl5
time="2022-05-24T17:24:07.374Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argoci/demo-mvdl5-4274864227/labelPodCompleted

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[alexec]

I'm not sure what the semantics should be for "resubmit" vs "resubmit + memoized" should be. See #2320

"memoized" blurs "resubmit" and "retry".

I thought that resubmit should nil the status and start again, but that's not what the code does:

argo-workflows/workflow/util/util.go

Line 672 in d8aa467

for _, node := range wf.Status.Nodes {

I think that loop should only run in memoized == true.

Could you like to submit a PR to fix?

[RoryDoherty]

Thanks Alex, was about to tackle this but then I noticed something:
If memoized isn't true would I not return before reaching that loop so that loop is already only run if memoized is true

argo-workflows/workflow/util/util.go

Lines 660 to 662 in d8aa467

	if !memoized {
	return &newWF, nil
	}

Or am I missing something?

[alexec]

Hmm. You're correct.

[alexec]

Are you re-submitting with memoized?

[RoryDoherty]

I'm not using memoized anywhere in the workflow that I submit, The demo workflow above also doesn't use it

I was just clicking resubmit from the Web UI, do you know if it defaults to memoized?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

[RoryDoherty]

This issue still exists

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

[RoryDoherty]

issue still exists
@alexec do you know if clicking resubmit via the UI defaults to memoized=true?

[stale]

This issue has been closed due to inactivity. Feel free to re-open if you still encounter this issue.

[RoryDoherty]

This issue still exists

[RoryDoherty]

@alexec Can this be reopened? I'm still seeing this behaviour in v3.4.3