Artifact in workflow diagram not working when workflow is archived and deleted from cluster

[dmarquez-splunk]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

Artifact URLs used in the workflow diagram's artifact icons seem to no longer work once a workflow is archived and deleted from the cluster (Returns Internal Server Error). However, if you click on the node from which the artifact is output from and attempt to download the artifact from the right-hand bar's input/output's menu then the artifact successfully downloads.

Difference in the 2 links:

link from workflow diagram's artifact icon: https://<argo-host>/artifact-files/<namespace>/workflows/<workflow-name>/<workflow-name>-1855980756/outputs/result_html

link from node's input/output menu: https://<argo-host>/artifact-files/<namespace>/archived-workflows/f7655950-a426-4fb3-a82d-eb1f53dfb948/<workflow-name>-1855980756/outputs/result_html

As you can see the link in the workflow diagram's artifact icon is still trying to reference the worklow as a non-archived workflow. Whereas the link in the node's input/output menu is now referencing it as an archived-workflow in the API. I suspect this is causing the argo-server to return Internal Server Error when clicking on the link from the workflow diagram's artifact icon.

NOTE: In order to reproduce the workflow and its artifacts must be archived AND deleted from the cluster in order to reproduce

Version

v3.5.2

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifact-passing-
spec:
  entrypoint: artifact-example
  templates:
  - name: artifact-example
    steps:
    - - name: generate-artifact
        template: whalesay
    - - name: consume-artifact
        template: print-message
        arguments:
          artifacts:
          # bind message to the hello-art artifact
          # generated by the generate-artifact step
          - name: message
            from: "{{steps.generate-artifact.outputs.artifacts.hello-art}}"

  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [sh, -c]
      args: ["cowsay hello world | tee /tmp/hello_world.txt"]
    outputs:
      artifacts:
      # generate hello-art artifact from /tmp/hello_world.txt
      # artifacts can be directories as well as files
      - name: hello-art
        path: /tmp/hello_world.txt

  - name: print-message
    inputs:
      artifacts:
      # unpack the message input artifact
      # and put it at /tmp/message
      - name: message
        path: /tmp/message
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["cat /tmp/message"]

Logs from the workflow controller

N/A

Logs from in your workflow's wait container

N/A

[Garett-MacGowan]

I may be able to assist here. I'm looking at a potentially related issue. @agilgur5 can you point me in the right direction for enabling persistence when developing locally?

[Joibel]

make start should install minio and enable artifact storage in it.

[Garett-MacGowan]

make start should install minio and enable artifact storage in it.

Thanks @Joibel, I'm refering to workflow persistence. I'm running the TestArtifactServer test suite and the logs are indicating

time="2023-12-21T20:59:33.141Z" level=info msg="Persistence configuration disabled"

Minio working just fine.

[Garett-MacGowan]

make start should install minio and enable artifact storage in it.

Thanks @Joibel, I'm refering to workflow persistence. I'm running the TestArtifactServer test suite and the logs are indicating

time="2023-12-21T20:59:33.141Z" level=info msg="Persistence configuration disabled"

Minio working just fine.

I think I resolved this - need to supply a PROFILE which is not "minimal" in make.

As usual, I skipped right over this in the docs: https://argoproj.github.io/argo-workflows/running-locally/#developing-locally

[Joibel]

Yeah, sorry, I meant you get the basics for storage and the config is there for artifacts - I was hoping that would be enough to show you the way forward. If you'd like to contribute some developer documentation about things like this that would be great.

[Garett-MacGowan]

NOTE: In order to reproduce the workflow and its artifacts must be archived AND deleted from the cluster in order to reproduce

Based on the above, I'm assuming that this issue is specific to the workflow archive. This is likely related to changes I made in #11947.

Here's my hypothesis:

It appears that the workflow archive is running too early.

In controller.go, we attach an event listener to wfc.wfInformer which triggers wfc.archiveWorkflow when

un.GetLabels()[common.LabelKeyWorkflowArchivingStatus] == "Pending"

The Pending status is set when case wfv1.WorkflowSucceeded, wfv1.WorkflowFailed, wfv1.WorkflowError: in operator.go. I think the problem is that artifacts haven't necessarily finished reconciling from the wait container at this point.

The solution may be to simply add a check in the event listener filter function to trigger when LabelKeyReportOutputsCompleted

un.GetLabels()[common.LabelKeyWorkflowArchivingStatus] == "Pending" && 
(
    un.GetLabels()[common.LabelKeyReportOutputsCompleted] == "true" || 
    un.GetAnnotations()[common.AnnotationKeyReportOutputsCompleted] == "true"
)

I'm going to try and get a failing test case for this & see if the above works.

@juliev0 tagging you here for feedback & thoughts.

EDIT 1

This change had no affect for this particular issue, although I still think it might be important.

EDIT 2

Although the details of the solution above aren't quite right, the idea that the workflow archive was running too early was correct. I have fixed this issue in #12402

[Garett-MacGowan]

I have the issue reproduced and I am working towards a solution.

[Garett-MacGowan]

@dmarquez-splunk I'm getting the following link for the artifact icon:

http://localhost:8080/artifact-files/argo/workflows/artifact-passing-98kwp/artifact-passing-98kwp-4182999709/inputs/message

This differs from what you have above:

https://<argo-host>/artifact-files/<namespace>/workflows/<workflow-name>/<workflow-name>-1855980756/outputs/result_html

Not sure why mine (same workflow as what you posted) is different in terms of outputs vs inputs and result_html vs message

Can you confirm that's what you're seeing?

[Garett-MacGowan]

This particular issue seems to be related to UI only. The ArtifactPanel component is not being passed the archived props.

[Garett-MacGowan]

I've confirmed the fix is the missing archived props.