Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ArtifactGC fails when workflows are retried #13161

Closed
3 of 4 tasks
mstrange96 opened this issue Jun 10, 2024 · 4 comments · Fixed by #13066
Closed
3 of 4 tasks

ArtifactGC fails when workflows are retried #13161

mstrange96 opened this issue Jun 10, 2024 · 4 comments · Fixed by #13066
Labels
area/artifacts S3/GCP/OSS/Git/HDFS etc area/gc Garbage collection, such as TTLs, retentionPolicy, delays, and more P2 Important. All bugs with >=3 thumbs up that aren’t P0 or P1, plus: Any other bugs deemed important type/bug
Milestone
v3.5.x patches

Comments

@mstrange96
Copy link

mstrange96 commented Jun 10, 2024 •
edited
Loading

Pre-requisites

  • I have double-checked my configuration
  • I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
  • I have searched existing issues and could not find a match for this bug
  • I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

When running a workflow with retries using artifact storage, the artifact garbage collection fails if there are any failed runs. While I haven't done a deep dive into the code, my theory is that the WorkflowArtifactGCTask is generated incorrectly. Failed runs have an artifact listed but no storage location - this is only found with the successful run. The artifact gc pod deletes logs and artifacts until an artifact without a storage location is encountered where it produces the error shown from the logs below:
Artifact gc task:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowArtifactGCTask
metadata:
  creationTimestamp: "2024-06-10T12:29:43Z"
  generation: 1
  labels:
    workflows.argoproj.io/artifact-gc-pod: "2436631062"
  name: artifact-gc-test-lpghp-artgc-wfcomp-1518482309-0
  namespace: argo-artifactgc-test
  ownerReferences:
  - apiVersion: argoproj.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Workflow
    name: artifact-gc-test-lpghp
    uid: 6fdb728e-1d54-4aa0-8587-6551e6904591
  resourceVersion: "187816230"
  uid: 0874cb15-96ac-4c4c-a887-0e4a2c0cc1e6
spec:
  artifactsByNode:
    artifact-gc-test-lpghp-1178359935:
      archiveLocation:
        archiveLogs: true
        s3:
          bucket: argo-workflow-artifact-storage-playground
          endpoint: s3.amazonaws.com
          key: '{{workflow.namespace}}/{{workflow.name}}/{{pod.name}}/{{workflow.uid}}'
          useSDKCreds: true
      artifacts:
        file:
          name: file
          path: /tmp/file.txt
          s3:
            key: argo-artifactgc-test/artifact-gc-test-lpghp/artifact-gc-test-lpghp-source-1178359935/6fdb728e-1d54-4aa0-8587-6551e6904591/file.tgz
        main-logs:
          name: main-logs
          s3:
            key: argo-artifactgc-test/artifact-gc-test-lpghp/artifact-gc-test-lpghp-source-1178359935/6fdb728e-1d54-4aa0-8587-6551e6904591/main.log
    artifact-gc-test-lpghp-1782501314:
      archiveLocation:
        archiveLogs: true
        s3:
          bucket: argo-workflow-artifact-storage-playground
          endpoint: s3.amazonaws.com
          key: '{{workflow.namespace}}/{{workflow.name}}/{{pod.name}}/{{workflow.uid}}'
          useSDKCreds: true
      artifacts:
        file:
          name: file
          path: /tmp/file.txt
        main-logs:
          name: main-logs
          s3:
            key: argo-artifactgc-test/artifact-gc-test-lpghp/artifact-gc-test-lpghp-source-1782501314/6fdb728e-1d54-4aa0-8587-6551e6904591/main.log

Logs from the artifact gc pod:

$ kubectl logs artifact-gc-test-lpghp-artgc-wfcomp-1518482309 -n argo-artifactgc-test
time="2024-06-10T12:29:45.481Z" level=info msg="S3 Delete artifact: key: argo-artifactgc-test/artifact-gc-test-lpghp/artifact-gc-test-lpghp-source-1178359935/6fdb728e-1d54-4aa0-8587-6551e6904591/file.tgz"
time="2024-06-10T12:29:45.481Z" level=info msg="Creating minio client using AWS SDK credentials"
time="2024-06-10T12:29:45.599Z" level=info msg="Deleting object from s3" bucket=argo-workflow-artifact-storage-playground endpoint=s3.amazonaws.com key=argo-artifactgc-test/artifact-gc-test-lpghp/artifact-gc-test-lpghp-source-1178359935/6fdb728e-1d54-4aa0-8587-6551e6904591/file.tgz
time="2024-06-10T12:29:45.816Z" level=info msg="S3 Delete artifact: key: argo-artifactgc-test/artifact-gc-test-lpghp/artifact-gc-test-lpghp-source-1178359935/6fdb728e-1d54-4aa0-8587-6551e6904591/main.log"
time="2024-06-10T12:29:45.816Z" level=info msg="Creating minio client using AWS SDK credentials"
time="2024-06-10T12:29:45.824Z" level=info msg="Deleting object from s3" bucket=argo-workflow-artifact-storage-playground endpoint=s3.amazonaws.com key=argo-artifactgc-test/artifact-gc-test-lpghp/artifact-gc-test-lpghp-source-1178359935/6fdb728e-1d54-4aa0-8587-6551e6904591/main.log
Error: You need to configure artifact storage. More information on how to do this can be found in the docs: https://argo-workflows.readthedocs.io/en/release-3.5/configure-artifact-repository/
You need to configure artifact storage. More information on how to do this can be found in the docs: https://argo-workflows.readthedocs.io/en/release-3.5/configure-artifact-repository/

To my knowledge of how argo workflows work, I'd expect there to be some kind of check/guard on the status of the workflow/container before including its artifacts in garbage collection (although I don't know how that would work with templates producing multiple artifacts)

Version

v3.5.5 - Bitnami helm chart v6.7.2

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

### Assuming the default service account has the required
### permissions to run the workflow and the garbage collection
###
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: artifact-gc-test
spec:
  arguments:
    parameters:
    - name: random_ceil
      value: 2
  entrypoint: source
  artifactGC:
    strategy: OnWorkflowCompletion
  retryStrategy:
    limit: 10
    retryPolicy: OnFailure
    backoff:
      duration: "1"
      factor: 1
      maxDuration: "1m"
  templates:
  - name: source
    inputs:
      parameters:
      - name: random_ceil
      - name: seed
        value: '{{ retries }}'
    outputs:
      artifacts:
      - name: file
        path: /tmp/file.txt
    script:
      image: python:3.11
      command:
      - python
      source: |-
        import os
        import sys
        sys.path.append(os.getcwd())
        import json
        try: random_ceil = json.loads(r'''{{inputs.parameters.random_ceil}}''')
        except: random_ceil = r'''{{inputs.parameters.random_ceil}}'''
        try: seed = json.loads(r'''{{inputs.parameters.seed}}''')
        except: seed = r'''{{inputs.parameters.seed}}'''

        import random

        random.seed(int(seed))

        exit_code = random.randint(0,int(random_ceil))
        if exit_code == 0:
          with open('/tmp/file.txt', 'w+') as f:
              f.write("Hello, world!\n")
        
        sys.exit(exit_code)

Logs from the workflow controller

$ kubectl logs -n argo-workflows argo-workflows-controller-785d899d59-s7b4t | grep artifact-gc-test-lpghp
time="2024-06-10T12:29:18.690Z" level=info msg="Processing workflow" Phase= ResourceVersion=187815854 namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:18.706Z" level=info msg="adding artifact GC finalizer" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:18.706Z" level=info msg="Task-result reconciliation" namespace=argo-artifactgc-test numObjs=0 workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:18.706Z" level=info msg="Updated phase  -> Running" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:18.706Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:18.706Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:18.706Z" level=info msg="Retry node artifact-gc-test-lpghp initialized Running" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:18.706Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:18.706Z" level=info msg="Pod node artifact-gc-test-lpghp-1782501314 initialized Pending" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:18.739Z" level=info msg="Created pod: artifact-gc-test-lpghp(0) (artifact-gc-test-lpghp-source-1782501314)" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:18.739Z" level=info msg="TaskSet Reconciliation" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:18.739Z" level=info msg=reconcileAgentPod namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:18.754Z" level=info msg="Workflow update successful" namespace=argo-artifactgc-test phase=Running resourceVersion=187815858 workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:28.735Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=187815858 namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:28.736Z" level=info msg="Task-result reconciliation" namespace=argo-artifactgc-test numObjs=1 workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:28.736Z" level=info msg="task-result changed" namespace=argo-artifactgc-test nodeID=artifact-gc-test-lpghp-1782501314 workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:28.736Z" level=info msg="Pod failed: Error (exit code 1)" displayName="artifact-gc-test-lpghp(0)" namespace=argo-artifactgc-test pod=artifact-gc-test-lpghp-source-1782501314 templateName=source workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:28.736Z" level=info msg="node changed" namespace=argo-artifactgc-test new.message="Error (exit code 1)" new.phase=Failed new.progress=0/1 nodeID=artifact-gc-test-lpghp-1782501314 old.message= old.phase=Pending old.progress=0/1 workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:28.736Z" level=info msg="node has maxDuration set, setting executionDeadline to: Mon Jun 10 12:30:18 +0000 (49 seconds from now)" namespace=argo-artifactgc-test node=artifact-gc-test-lpghp workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:28.736Z" level=info msg="Retry Policy: OnFailure (onFailed: true, onError false)" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:28.736Z" level=info msg="1 child nodes of artifact-gc-test-lpghp failed. Trying again..." namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:28.736Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:28.736Z" level=info msg="Pod node artifact-gc-test-lpghp-1178359935 initialized Pending" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:28.768Z" level=info msg="Created pod: artifact-gc-test-lpghp(1) (artifact-gc-test-lpghp-source-1178359935)" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:28.768Z" level=info msg="TaskSet Reconciliation" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:28.768Z" level=info msg=reconcileAgentPod namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:28.778Z" level=info msg="Workflow update successful" namespace=argo-artifactgc-test phase=Running resourceVersion=187816011 workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:28.783Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-artifactgc-test/artifact-gc-test-lpghp-source-1782501314/labelPodCompleted
time="2024-06-10T12:29:38.769Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=187816011 namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:38.769Z" level=info msg="Task-result reconciliation" namespace=argo-artifactgc-test numObjs=2 workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:38.769Z" level=info msg="task-result changed" namespace=argo-artifactgc-test nodeID=artifact-gc-test-lpghp-1782501314 workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:38.769Z" level=info msg="task-result changed" namespace=argo-artifactgc-test nodeID=artifact-gc-test-lpghp-1178359935 workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:38.769Z" level=info msg="node changed" namespace=argo-artifactgc-test new.message= new.phase=Succeeded new.progress=0/1 nodeID=artifact-gc-test-lpghp-1178359935 old.message= old.phase=Pending old.progress=0/1 workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:38.770Z" level=info msg="node artifact-gc-test-lpghp phase Running -> Succeeded" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:38.770Z" level=info msg="node artifact-gc-test-lpghp finished: 2024-06-10 12:29:38.770059355 +0000 UTC" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:38.770Z" level=info msg="TaskSet Reconciliation" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:38.770Z" level=info msg=reconcileAgentPod namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:38.770Z" level=info msg="Updated phase Running -> Succeeded" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:38.770Z" level=info msg="Marking workflow completed" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:38.775Z" level=info msg="cleaning up pod" action=deletePod key=argo-artifactgc-test/artifact-gc-test-lpghp-1340600742-agent/deletePod
time="2024-06-10T12:29:38.781Z" level=info msg="Workflow update successful" namespace=argo-artifactgc-test phase=Succeeded resourceVersion=187816162 workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:38.782Z" level=info msg="Queueing Succeeded workflow argo-artifactgc-test/artifact-gc-test-lpghp for delete in 336h0m0s due to TTL"
time="2024-06-10T12:29:38.808Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-artifactgc-test/artifact-gc-test-lpghp-source-1178359935/labelPodCompleted
time="2024-06-10T12:29:43.804Z" level=info msg="Processing workflow" Phase=Succeeded ResourceVersion=187816162 namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:43.804Z" level=info msg="Task-result reconciliation" namespace=argo-artifactgc-test numObjs=0 workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:43.804Z" level=info msg="Creating Artifact GC Task artifact-gc-test-lpghp-artgc-wfcomp-1518482309-0" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:43.812Z" level=info msg="creating pod to delete artifacts: artifact-gc-test-lpghp-artgc-wfcomp-1518482309" namespace=argo-artifactgc-test strategy=OnWorkflowCompletion workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:43.840Z" level=info msg="Workflow update successful" namespace=argo-artifactgc-test phase=Succeeded resourceVersion=187816232 workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:43.846Z" level=info msg="Queueing Succeeded workflow argo-artifactgc-test/artifact-gc-test-lpghp for delete in 335h59m55s due to TTL"
time="2024-06-10T12:29:48.849Z" level=info msg="Processing workflow" Phase=Succeeded ResourceVersion=187816232 namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:48.849Z" level=info msg="Task-result reconciliation" namespace=argo-artifactgc-test numObjs=0 workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:48.849Z" level=info msg="reconciling artifact-gc pod" message= namespace=argo-artifactgc-test phase=Failed pod=artifact-gc-test-lpghp-artgc-wfcomp-1518482309 workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:48.849Z" level=info msg="processing completed Artifact GC Pod \"artifact-gc-test-lpghp-artgc-wfcomp-1518482309\"" namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:48.866Z" level=info msg="Workflow update successful" namespace=argo-artifactgc-test phase=Succeeded resourceVersion=187816305 workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:48.868Z" level=info msg="Queueing Succeeded workflow argo-artifactgc-test/artifact-gc-test-lpghp for delete in 335h59m50s due to TTL"
time="2024-06-10T12:29:53.870Z" level=info msg="Processing workflow" Phase=Succeeded ResourceVersion=187816305 namespace=argo-artifactgc-test workflow=artifact-gc-test-lpghp
time="2024-06-10T12:29:53.871Z" level=info msg="Task-result reconciliation" namespace=argo-artifactgc-test numObjs=0 workflow=artifact-gc-test-lpghp

Logs from in your workflow's wait container

kubectl logs -n argo-artifactgc-test artifact-gc-test-lpghp-source-1782501314 -c wait
time="2024-06-10T12:29:20.510Z" level=info msg="Starting Workflow Executor" version=v3.5.7
time="2024-06-10T12:29:20.514Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2024-06-10T12:29:20.514Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-artifactgc-test podName=artifact-gc-test-lpghp-source-1782501314 templateName=source version="&Version{Version:v3.5.7,BuildDate:2024-06-04T19:41:01Z,GitCommit:503eef1357ebc9facc3f463708031441072ef7c2,GitTag:v3.5.7,GitTreeState:clean,GoVersion:go1.21.11,Compiler:gc,Platform:linux/amd64,}"
time="2024-06-10T12:29:20.541Z" level=info msg="Starting deadline monitor"
time="2024-06-10T12:29:22.541Z" level=info msg="Main container completed" error="<nil>"
time="2024-06-10T12:29:22.541Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2024-06-10T12:29:22.541Z" level=info msg="No output parameters"
time="2024-06-10T12:29:22.542Z" level=info msg="Saving output artifacts"
time="2024-06-10T12:29:22.542Z" level=info msg="Staging artifact: file"
time="2024-06-10T12:29:22.542Z" level=info msg="Copying /tmp/file.txt from container base image layer to /tmp/argo/outputs/artifacts/file.tgz"
time="2024-06-10T12:29:22.542Z" level=info msg="/var/run/argo/outputs/artifacts/tmp/file.txt.tgz -> /tmp/argo/outputs/artifacts/file.tgz"
time="2024-06-10T12:29:22.542Z" level=error msg="executor error: open /var/run/argo/outputs/artifacts/tmp/file.txt.tgz: no such file or directory"
time="2024-06-10T12:29:22.542Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: argo-artifactgc-test/artifact-gc-test-lpghp/artifact-gc-test-lpghp-source-1782501314/6fdb728e-1d54-4aa0-8587-6551e6904591/main.log"
time="2024-06-10T12:29:22.542Z" level=info msg="Creating minio client using AWS SDK credentials"
time="2024-06-10T12:29:22.627Z" level=info msg="Saving file to s3" bucket=argo-workflow-artifact-storage-playground endpoint=s3.amazonaws.com key=argo-artifactgc-test/artifact-gc-test-lpghp/artifact-gc-test-lpghp-source-1782501314/6fdb728e-1d54-4aa0-8587-6551e6904591/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-06-10T12:29:22.684Z" level=info msg="Save artifact" artifactName=main-logs duration=141.753964ms error="<nil>" key=argo-artifactgc-test/artifact-gc-test-lpghp/artifact-gc-test-lpghp-source-1782501314/6fdb728e-1d54-4aa0-8587-6551e6904591/main.log
time="2024-06-10T12:29:22.684Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-06-10T12:29:22.684Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-06-10T12:29:22.699Z" level=info msg="Alloc=11640 TotalAlloc=17023 Sys=28517 NumGC=4 Goroutines=12"
time="2024-06-10T12:29:22.705Z" level=fatal msg="open /var/run/argo/outputs/artifacts/tmp/file.txt.tgz: no such file or directory"
@agilgur5
Copy link
Member

agilgur5 commented Jun 10, 2024 •
edited
Loading

Failed runs have an artifact listed but no storage location - this is only found with the successful run. The artifact gc pod deletes logs and artifacts until an artifact without a storage location is encountered where it produces the error shown from the logs below:

Error: You need to configure artifact storage. More information on how to do this can be found in the docs: https://argo-workflows.readthedocs.io/en/release-3.5/configure-artifact-repository/

This looks like a duplicate of #12845, which has a fix out in #13066

  • I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.

Version

v3.5.5

Also, v3.5.5 is neither :latest nor latest stable. Please fill out the issue template accurately.

@agilgur5 agilgur5 linked a pull request Jun 10, 2024 that will close this issue
fix: don't necessarily include all artifacts from templates in node outputs #13066
Merged
@agilgur5 agilgur5 added this to the v3.5.x patches milestone Jun 10, 2024
@agilgur5 agilgur5 added area/artifacts S3/GCP/OSS/Git/HDFS etc area/gc Garbage collection, such as TTLs, retentionPolicy, delays, and more P2 Important. All bugs with >=3 thumbs up that aren’t P0 or P1, plus: Any other bugs deemed important labels Jun 10, 2024
@mstrange96
Copy link
Author

mstrange96 commented Jun 11, 2024 •
edited by agilgur5
Loading

Also, v3.5.5 is neither :latest nor latest stable. Please fill out the issue template accurately.

All logs etc. are from our v3.5.5 installation, hence why I put it there. I tested :latest as outlined in the template, the bug was still present.

This looks like a duplicate of #12845, which has a fix out in #13066

Indeed it does. I have clearly missed it when searching through passed issues for this bug. Thank you so much, happy to hear you are already working on a fix!

We are enjoying working with Argo-Workflows, it's a great tool 👏

@agilgur5
Copy link
Member

All logs etc. are from our v3.5.5 installation, hence why I put it there. I tested :latest as outlined in the template, the bug was still present.

Oh, gotcha, sorry about that then. It might be more clear to write "v3.5.5 + :latest" in the "Version" section. Or some folks put a comma separated list of versions they tried.
There are unfortunately many users who report bugs on outdated versions (often much more outdated than v3.5.5) so it's something we watch out for 😕

I have clearly missed it when searching through passed issues for this bug.

That's because it was mistakenly filed as a discussion by the user, so it wouldn't pop up on issue search 😅
That's also why I left your issue open, since technically one doesn't exist.

Also GH didn't unify search for Discussions when they made that feature, so things do get lost and duplicated between them 😕 That's one of the reasons I don't turn on Discussions in my personal repos

@mstrange96
Copy link
Author

mstrange96 commented Jun 12, 2024 •
edited
Loading

Oh, gotcha, sorry about that then. It might be more clear to write "v3.5.5 + :latest" in the "Version" section. Or some folks put a comma separated list of versions they tried. There are unfortunately many users who report bugs on outdated versions (often much more outdated than v3.5.5) so it's something we watch out for 😕

Of course, no worries! I can see how it was not clear to you. I'll keep that in mind in case I report anything in the future 🙂

That's because it was mistakenly filed as a discussion by the user, so it wouldn't pop up on issue search 😅 That's also why I left your issue open, since technically one doesn't exist.

Also GH didn't unify search for Discussions when they made that feature, so things do get lost and duplicated between them 😕 That's one of the reasons I don't turn on Discussions in my personal repos

That makes sense, it's neither the first nor last time this has happened 😅

@agilgur5 agilgur5 mentioned this issue Jul 9, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/artifacts S3/GCP/OSS/Git/HDFS etc area/gc Garbage collection, such as TTLs, retentionPolicy, delays, and more P2 Important. All bugs with >=3 thumbs up that aren’t P0 or P1, plus: Any other bugs deemed important type/bug
Projects
None yet
Milestone
v3.5.x patches
Development

Successfully merging a pull request may close this issue.

fix: don't necessarily include all artifacts from templates in node outputs juliev0/argo-workflows
2 participants
@agilgur5 @mstrange96