Unable to view artifacts of archived workflows in Workflows UI (v3.5.4)

[talebzeghmi]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issue exists when I tested with :latest
  • note that this reproduces with v3.5.4 but NOT latest with latest as of 2/20/2024 https://quay.io/repository/argoproj/argocli/manifest/sha256:bb3257c799cd7a4007e9f7b8845661da65d22989e6a92011b78eb4d6e0db45a5
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

What happened

From a user's perspective this seems like a reproduction of #9968

We have a ttlStrategy as follows

        ttlStrategy:
          # Pod/workflow cleanup of any completed run
          secondsAfterFailure: 3600
          secondsAfterSuccess: 60

• Screenshot before the Workflow k8s resource is deleted and ttl-ed
  

• Screenshot after the Workflow k8s resource is ttl-ed within 60s
  

argo-server logs

kubectl logs -n argo argo-server-5f755dc54f-86tnk | grep artifact-passing-8scp9

time="2024-02-20T17:15:55.426Z" level=info msg="Get artifact file" artifactName=hello-art namespace=aip-playground-sandbox nodeId=artifact-passing-8scp9-621443663 workflowName=artifact-passing-8scp9
time="2024-02-20T17:15:55.532Z" level=info msg="Check if directory" artifactName=hello-art duration=88.393446ms error="<nil>" key=zillow/ai-platform/mlpipeline-store/sandbox/artifacts/artifact-passing-8scp9/artifact-passing-8scp9-621443663/hello_world.txt
time="2024-02-20T17:15:55.532Z" level=info msg="S3 OpenStream: key: zillow/ai-platform/mlpipeline-store/sandbox/artifacts/artifact-passing-8scp9/artifact-passing-8scp9-621443663/hello_world.txt"
time="2024-02-20T17:15:55.537Z" level=info msg="Opening file from s3" bucket=zg-ai-platform-sandbox endpoint=s3.amazonaws.com key=zillow/ai-platform/mlpipeline-store/sandbox/artifacts/artifact-passing-8scp9/artifact-passing-8scp9-621443663/hello_world.txt
time="2024-02-20T17:15:55.603Z" level=info msg="Stream artifact" artifactName=hello-art duration=70.408905ms error="<nil>" key=zillow/ai-platform/mlpipeline-store/sandbox/artifacts/artifact-passing-8scp9/artifact-passing-8scp9-621443663/hello_world.txt
time="2024-02-20T17:15:55.616Z" level=info duration=190.146953ms method=GET path=/artifact-files/aip-playground-sandbox/workflows/artifact-passing-8scp9/artifact-passing-8scp9-621443663/outputs/hello-art size=399 status=200
time="2024-02-20T17:17:23.180Z" level=info msg="Get artifact file" artifactName=hello-art namespace=aip-playground-sandbox nodeId=artifact-passing-8scp9-621443663 workflowName=artifact-passing-8scp9
time="2024-02-20T17:17:23.188Z" level=error msg="Artifact Server returned internal error" error="workflows.argoproj.io \"artifact-passing-8scp9\" not found"
time="2024-02-20T17:17:23.188Z" level=info duration=8.412262ms method=GET path=/artifact-files/aip-playground-sandbox/workflows/artifact-passing-8scp9/artifact-passing-8scp9-621443663/outputs/hello-art size=22 status=500

What did you expect to happen?

For the artifact to be viewable after the workflow ttls.

Version

v3.5.4

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifact-passing-
spec:
  entrypoint: artifact-example
  templates:
  - name: artifact-example
    steps:
    - - name: generate-artifact
        template: whalesay
    - - name: consume-artifact
        template: print-message
        arguments:
          artifacts:
          # bind message to the hello-art artifact
          # generated by the generate-artifact step
          - name: message
            from: "{{steps.generate-artifact.outputs.artifacts.hello-art}}"

  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [sh, -c]
      args: ["cowsay hello world | tee /tmp/hello_world.txt"]
      resources:
        limits:
          cpu: "100m"
          memory: "100Mi"
    outputs:
      artifacts:
      # generate hello-art artifact from /tmp/hello_world.txt
      # artifacts can be directories as well as files
      - name: hello-art
        path: /tmp/hello_world.txt
        archive:
          none: {}

  - name: print-message
    inputs:
      artifacts:
      # unpack the message input artifact
      # and put it at /tmp/message
      - name: message
        path: /tmp/message
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["cat /tmp/message"]
      resources:
        limits:
          cpu: "100m"
          memory: "100Mi"

Logs from the workflow controller

No error is shown here, please see the `argo-server` logs


kubectl logs -n argo workflow-controller-58f4ccfcff-drjtk | grep artifact-passing-8jskk


time="2024-02-20T18:31:03.403Z" level=info msg="Processing workflow" Phase= ResourceVersion=2527021884 namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:03.410Z" level=info msg="Task-result reconciliation" namespace=aip-playground-sandbox numObjs=0 workflow=artifact-passing-8jskk
time="2024-02-20T18:31:03.410Z" level=info msg="Updated phase  -> Running" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:03.410Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:03.411Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:03.411Z" level=info msg="Steps node artifact-passing-8jskk initialized Running" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:03.411Z" level=info msg="StepGroup node artifact-passing-8jskk-618532745 initialized Running" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:03.411Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:03.411Z" level=info msg="Pod node artifact-passing-8jskk-20788667 initialized Pending" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:03.539Z" level=info msg="Created pod: artifact-passing-8jskk[0].generate-artifact (artifact-passing-8jskk-20788667)" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:03.539Z" level=info msg="Workflow step group node artifact-passing-8jskk-618532745 not yet completed" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:03.539Z" level=info msg="TaskSet Reconciliation" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:03.539Z" level=info msg=reconcileAgentPod namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:03.557Z" level=info msg="Workflow update successful" namespace=aip-playground-sandbox phase=Running resourceVersion=2527021889 workflow=artifact-passing-8jskk
time="2024-02-20T18:31:13.403Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2527021889 namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:13.404Z" level=info msg="Task-result reconciliation" namespace=aip-playground-sandbox numObjs=1 workflow=artifact-passing-8jskk
time="2024-02-20T18:31:13.404Z" level=info msg="task-result changed" namespace=aip-playground-sandbox nodeID=artifact-passing-8jskk-20788667 workflow=artifact-passing-8jskk
time="2024-02-20T18:31:13.404Z" level=info msg="node changed" namespace=aip-playground-sandbox new.message= new.phase=Running new.progress=0/1 nodeID=artifact-passing-8jskk-20788667 old.message= old.phase=Pending old.progress=0/1 workflow=artifact-passing-8jskk
time="2024-02-20T18:31:13.404Z" level=info msg="Workflow step group node artifact-passing-8jskk-618532745 not yet completed" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:13.404Z" level=info msg="TaskSet Reconciliation" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:13.404Z" level=info msg=reconcileAgentPod namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:13.410Z" level=info msg="cleaning up pod" action=terminateContainers key=aip-playground-sandbox/artifact-passing-8jskk-20788667/terminateContainers
time="2024-02-20T18:31:13.417Z" level=info msg="Workflow update successful" namespace=aip-playground-sandbox phase=Running resourceVersion=2527022184 workflow=artifact-passing-8jskk
time="2024-02-20T18:31:23.418Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2527022184 namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:23.419Z" level=info msg="Task-result reconciliation" namespace=aip-playground-sandbox numObjs=1 workflow=artifact-passing-8jskk
time="2024-02-20T18:31:23.419Z" level=info msg="task-result changed" namespace=aip-playground-sandbox nodeID=artifact-passing-8jskk-20788667 workflow=artifact-passing-8jskk
time="2024-02-20T18:31:23.419Z" level=info msg="node changed" namespace=aip-playground-sandbox new.message= new.phase=Succeeded new.progress=0/1 nodeID=artifact-passing-8jskk-20788667 old.message= old.phase=Running old.progress=0/1 workflow=artifact-passing-8jskk
time="2024-02-20T18:31:23.419Z" level=info msg="Step group node artifact-passing-8jskk-618532745 successful" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:23.419Z" level=info msg="node artifact-passing-8jskk-618532745 phase Running -> Succeeded" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:23.419Z" level=info msg="node artifact-passing-8jskk-618532745 finished: 2024-02-20 18:31:23.419491513 +0000 UTC" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:23.419Z" level=info msg="StepGroup node artifact-passing-8jskk-3772769044 initialized Running" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:23.419Z" level=info msg="SG Outbound nodes of artifact-passing-8jskk-20788667 are [artifact-passing-8jskk-20788667]" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:23.419Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:23.419Z" level=info msg="Pod node artifact-passing-8jskk-1269863105 initialized Pending" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:23.547Z" level=info msg="Created pod: artifact-passing-8jskk[1].consume-artifact (artifact-passing-8jskk-1269863105)" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:23.547Z" level=info msg="Workflow step group node artifact-passing-8jskk-3772769044 not yet completed" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:23.547Z" level=info msg="TaskSet Reconciliation" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:23.547Z" level=info msg=reconcileAgentPod namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:23.561Z" level=info msg="Workflow update successful" namespace=aip-playground-sandbox phase=Running resourceVersion=2527022438 workflow=artifact-passing-8jskk
time="2024-02-20T18:31:28.563Z" level=info msg="cleaning up pod" action=deletePod key=aip-playground-sandbox/artifact-passing-8jskk-20788667/deletePod
time="2024-02-20T18:31:33.550Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2527022438 namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.550Z" level=info msg="Task-result reconciliation" namespace=aip-playground-sandbox numObjs=2 workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.550Z" level=info msg="task-result changed" namespace=aip-playground-sandbox nodeID=artifact-passing-8jskk-20788667 workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.550Z" level=info msg="task-result changed" namespace=aip-playground-sandbox nodeID=artifact-passing-8jskk-1269863105 workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.550Z" level=info msg="node changed" namespace=aip-playground-sandbox new.message= new.phase=Succeeded new.progress=0/1 nodeID=artifact-passing-8jskk-1269863105 old.message= old.phase=Pending old.progress=0/1 workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.551Z" level=info msg="SG Outbound nodes of artifact-passing-8jskk-20788667 are [artifact-passing-8jskk-20788667]" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.551Z" level=info msg="Step group node artifact-passing-8jskk-3772769044 successful" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.551Z" level=info msg="node artifact-passing-8jskk-3772769044 phase Running -> Succeeded" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.551Z" level=info msg="node artifact-passing-8jskk-3772769044 finished: 2024-02-20 18:31:33.551401531 +0000 UTC" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.551Z" level=info msg="Outbound nodes of artifact-passing-8jskk-1269863105 is [artifact-passing-8jskk-1269863105]" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.551Z" level=info msg="Outbound nodes of artifact-passing-8jskk is [artifact-passing-8jskk-1269863105]" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.551Z" level=info msg="node artifact-passing-8jskk phase Running -> Succeeded" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.551Z" level=info msg="node artifact-passing-8jskk finished: 2024-02-20 18:31:33.55147447 +0000 UTC" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.551Z" level=info msg="TaskSet Reconciliation" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.551Z" level=info msg=reconcileAgentPod namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.551Z" level=info msg="Updated phase Running -> Succeeded" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.551Z" level=info msg="Marking workflow completed" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.551Z" level=info msg="Marking workflow as pending archiving" namespace=aip-playground-sandbox workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.557Z" level=info msg="cleaning up pod" action=deletePod key=aip-playground-sandbox/artifact-passing-8jskk-1340600742-agent/deletePod
time="2024-02-20T18:31:33.567Z" level=info msg="Workflow update successful" namespace=aip-playground-sandbox phase=Succeeded resourceVersion=2527022723 workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.602Z" level=info msg="archiving workflow" namespace=aip-playground-sandbox uid=9ffcd93b-40e2-4f6f-86ca-baed3a4faebd workflow=artifact-passing-8jskk
time="2024-02-20T18:31:33.637Z" level=info msg="Queueing Succeeded workflow aip-playground-sandbox/artifact-passing-8jskk for delete in 1m0s due to TTL"
time="2024-02-20T18:31:38.603Z" level=info msg="cleaning up pod" action=deletePod key=aip-playground-sandbox/artifact-passing-8jskk-1269863105/deletePod
time="2024-02-20T18:31:43.410Z" level=info msg="cleaning up pod" action=killContainers key=aip-playground-sandbox/artifact-passing-8jskk-20788667/killContainers
time="2024-02-20T18:32:34.000Z" level=info msg="Deleting garbage collected workflow 'aip-playground-sandbox/artifact-passing-8jskk'"
time="2024-02-20T18:32:34.012Z" level=info msg="Successfully request 'aip-playground-sandbox/artifact-passing-8jskk' to be deleted"

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

[agilgur5]

Follow-up to #9968 (comment) it would seem?

As I wrote there, this would be duplicative of #12331, which was fixed in #12397, but has not yet been released. It is slated for the next patch release (v3.5.5): #11997 (comment)

• note that this reproduces with v3.5.4 but not latest with latest as of 2/20/2024 https://quay.io/repository/argoproj/argocli/manifest/sha256:bb3257c799cd7a4007e9f7b8845661da65d22989e6a92011b78eb4d6e0db45a5

That would indeed confirm that #12397 has fixed this