-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Linter fails on templateRef
for WorkflowTemplate
#5173
Comments
Is your template on the cluster? That checks verifies that the reference workflow template is accessible from the cluster, not just the file. If not, can you try adding the workflow template to the cluster first (using |
Here are the steps λ argo-windows-amd64-2.12.9.exe template create ref-example.yaml
2021/02/23 18:33:28 Failed to create workflow template: templates.whalesay.steps[0].call-whalesay-template template reference workflow-template-1.whalesay-template not found => error when creating the template whith a single file
argo-windows-amd64-2.12.9.exe template create ref-example-part1.yaml
Name: workflow-template-1
Namespace: mine
Created: Tue Feb 23 18:35:20 +0100 (now) => creation ok λ argo-windows-amd64-2.12.9.exe template lint ref-example.yaml
WorkflowTemplate manifests validated
WorkflowTemplate manifests validated => linter ok So your guess is right the linter seems failing due to the missing template in the cluster. So what is this the correct workflow for deployment ?
I was expecting to be able to lint everything before deployment and deploying everything on the cluster in one go. |
Yes, I agree that that is desired behavior. I'll work on a fix |
@bmaggi Another workaround: you don't actually need two files, you can do it on the same file as long as the WorkflowTemplate that is being referenced goes before (i.e. on top) of the WorkflowTemplate that is doing the referencing |
Thanks for your answer, maybe I missed something but I still have the linter error with apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: github-issue-example-lib
spec:
entrypoint: whalesay-template # Fields other than "arguments" and "templates" not supported in v2.4 - v2.6
arguments:
parameters:
- name: message
value: hello world
templates:
- name: whalesay-template
inputs:
parameters:
- name: message
container:
image: docker/whalesay
command: [cowsay]
args: ["{{inputs.parameters.message}}"]
---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate # used to be Workflow
metadata:
generateName: github-issue-example-main
spec:
entrypoint: whalesay
templates:
- name: whalesay
steps: # You should only reference external "templates" in a "steps" or "dag" "template".
- - name: call-whalesay-template
templateRef: # You can reference a "template" from another "WorkflowTemplate" using this field
name: github-issue-example-lib # This is the name of the "WorkflowTemplate" CRD that contains the "template" you want
template: whalesay-template # This is the name of the "template" you want to reference
arguments: # You can pass in arguments as normal
parameters:
- name: message
value: "hello world" λ argo-windows-amd64-2.12.9.exe template lint github-example.yaml
time="2021-02-25T17:04:44.704Z" level=error msg="templates.whalesay.steps[0].call-whalesay-template template reference github-issue-example-lib.whalesay-template not found"
time="2021-02-25T17:04:44.744Z" level=fatal msg="Errors encountered in validation" |
Low priority, but #5198 stands as a PoC to fix this issue |
This comment was marked as resolved.
This comment was marked as resolved.
@simster7 - is this still considered a low priority issue / wontfix? It's a showstopper for us since we like to do our tests vs a clean environment. Would you consider a partial fix whereby so long as the template files are specified in order such that no template is referred to before it is defined then it would pass? That would be immensely helpful! |
broken argo lint makes troubleshooting argo workflows really really hard for us. It is like looking for needle in haystack. Would it be possible to bump up the priority? Thanks |
@tvalasek I am trying to understand your use-case and how this issue will be a showstopper. If you split templates into separate files it will work what you expect like lint and create each resource. I understand keep everything in one file is kind of convenient and maintenance-free. But currently, our team has under-resource to work on this kind of issue. But we will try to prioritize but you or @simster7 like to submit the PR, we are happy to review it.
|
@sarabala1979 I've found the opposite. If I glue the scripts into a single file in an appropriate order it works ok [workaround], but if you split the files it never works because the templates are never defined. I image if I were to do it like so:
It would work - but that complicates our CI! @tvalasek as an ugly workaround, we rename all our templates to start with # Use awk to glue all the ago files into a single yaml file to work around argo bug
awk 'FNR==1 && NR!=1 {print "----"}{print}' argo/* > combined.yaml
argo lint --all-kinds combined.yaml I'm not proud! |
This comment was marked as resolved.
This comment was marked as resolved.
Hi. I tried your workaround but it does not work for me in v3.1.12. We use following reference chain: Workflow (wrapper) -> WorkflowTemplate (logical groups) -> WorkflowTemplate (pod specs). This article explains our use case: https://tomas-valasek.medium.com/building-kubernetes-clusters-using-kubernetes-dde332c77d65 |
I read "If you split templates into separate files it will work what you expect like lint and create each resource" as "lint one file, if OK, push it to the cluster". This would be inconvenient with a VCS-based flow that want to ensure that workflows (and WorkflowTemplates) are well-formed at the PR stage. I suspect it would require loading all referenced files before the linting starts, and use internally-stored Workflows and WorkflowTemplates before the linting starts, so as it avoid ordering issues between them. |
+1 to all of the above. We're trying to use the argo linting in our CI pipeline to catch errors in the manifests before we apply resources to our cluster. The linting fails for any references to workflow templates even if they are defined within the same file. This isn't actually an error and shouldn't be caught by the linting tool as one, given that the template will exist if it were applied above. It would be very helpful if we could get a fix in that allows the linting to check above in the file for resources and pass if they exist so that references to them won't error out. Or, at the very least switch these errors to warnings instead of errors. |
I'm running into the same issue but the workaround of combining them into one yaml in dependency order doesn't work for me. In this case, they are all |
This seems like a big problem. I'd like to see a mode that did not communicate with the cluster. However, I did discover that if I added the namespace where the interdependent wftmpls are deployed, then the lint passed! argo lint myfolder -n mynamespace |
I consider to up the minikube in the CI/CD to apply templates and lint it. |
templateRef
for WorkflowTemplate
This is still a large issue in 2024. argo lint should be linting against static files. Combining multiple files into one, results in the same behavior still of not being able to find If these For example the workflow-template example in this repository, with the independent resources put in their dependent order in ONE file: # WorkflowTemplate for printing a message
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: workflow-template-print-message
spec:
entrypoint: print-message
templates:
- container:
args:
- '{{inputs.parameters.message}}'
command:
- echo
image: busybox
inputs:
parameters:
- name: message
name: print-message
---
# WorkflowTemplate for random fail template
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: workflow-template-random-fail-template
spec:
templates:
- container:
args:
- import random; import sys; exit_code = random.choice([0, 1, 1]); sys.exit(exit_code)
command:
- python
- -c
image: python:alpine3.6
name: random-fail-template
retryStrategy:
limit: 10
---
# WorkflowTemplate for inner DAG
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: workflow-template-inner-dag
spec:
templates:
- container:
args:
- '{{inputs.parameters.message}}'
command:
- echo
image: busybox
inputs:
parameters:
- name: message
name: print-message
- dag:
tasks:
- arguments:
parameters:
- name: message
value: inner-A
name: inner-A
templateRef:
name: workflow-template-print-message
template: print-message
- arguments:
parameters:
- name: message
value: inner-B
depends: inner-A
name: inner-B
template: print-message
- arguments:
parameters:
- name: message
value: inner-C
depends: inner-A
name: inner-C
template: print-message
- arguments:
parameters:
- name: message
value: inner-D
depends: inner-B && inner-C
name: inner-D
templateRef:
name: workflow-template-print-message
template: print-message
name: inner-diamond
---
# WorkflowTemplate for inner steps
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: workflow-template-inner-steps
spec:
templates:
- container:
args:
- '{{inputs.parameters.message}}'
command:
- echo
image: busybox
inputs:
parameters:
- name: message
name: print-message
- name: inner-steps
steps:
- - arguments:
parameters:
- name: message
value: inner-hello1
name: inner-hello1
templateRef:
name: workflow-template-print-message
template: print-message
- - arguments:
parameters:
- name: message
value: inner-hello2a
name: inner-hello2a
templateRef:
name: workflow-template-print-message
template: print-message
- arguments:
parameters:
- name: message
value: inner-hello2b
name: inner-hello2b
template: print-message
---
# WorkflowTemplate for submittable workflow
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: workflow-template-submittable
spec:
arguments:
parameters:
- name: message
value: hello world
entrypoint: print-message
templates:
- container:
args:
- '{{inputs.parameters.message}}'
command:
- echo
image: busybox
inputs:
parameters:
- name: message
name: print-message
---
# Now the Workflows that use the templates
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: archive-location
spec:
archiveLogs: true
entrypoint: hello-world
templates:
- container:
args:
- hello world
command:
- echo
image: busybox
name: hello-world
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: workflow-template-dag-diamond
spec:
entrypoint: diamond
templates:
- dag:
tasks:
- arguments:
parameters:
- name: message
value: A
name: A
templateRef:
name: workflow-template-print-message
template: print-message
- arguments:
parameters:
- name: message
value: B
depends: A
name: B
templateRef:
name: workflow-template-print-message
template: print-message
- depends: A
name: C
templateRef:
name: workflow-template-inner-dag
template: inner-diamond
- arguments:
parameters:
- name: message
value: D
depends: B && C
name: D
templateRef:
name: workflow-template-print-message
template: print-message
name: diamond
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: workflow-template-hello-world
spec:
entrypoint: hello-world-from-templateRef
templates:
- name: hello-world-from-templateRef
steps:
- - arguments:
parameters:
- name: message
value: hello world
name: call-print-message
templateRef:
name: workflow-template-print-message
template: print-message
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: workflow-template-hello-world-1
spec:
arguments:
parameters:
- name: message
value: hello world
entrypoint: print-message
workflowTemplateRef:
name: workflow-template-print-message
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: workflow-template-retry-with-steps
spec:
entrypoint: retry-with-steps
templates:
- name: retry-with-steps
steps:
- - name: hello1
templateRef:
name: workflow-template-random-fail-template
template: random-fail-template
- - name: hello2a
templateRef:
name: workflow-template-random-fail-template
template: random-fail-template
- name: hello2b
templateRef:
name: workflow-template-random-fail-template
template: random-fail-template
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: workflow-template-steps
spec:
entrypoint: hello-hello-hello
templates:
- name: hello-hello-hello
steps:
- - arguments:
parameters:
- name: message
value: hello1
name: hello1
templateRef:
name: workflow-template-print-message
template: print-message
- - arguments:
parameters:
- name: message
value: hello2a
name: hello2a
templateRef:
name: workflow-template-inner-steps
template: inner-steps
- arguments:
parameters:
- name: message
value: hello2b
name: hello2b
templateRef:
name: workflow-template-print-message
template: print-message Results in:
This is a big blocker using argo-lint, it's hard to find good static analysis tools and this isn't usable. |
They are all separate files in the examples, as you noted.
I don't think any contributor would disagree with that. There's a reason This is listed as a P1 bug with workarounds and Argo is lacking contributors, per the Sustainability Effort. Anyone is welcome to contribute fixes, improvements to CI, etc |
Summary
I got a "template reference not found" when using the linter on the following example.
I was expecting a valid workflowTemplate, so I suspect an error in the linter.
Diagnostics
What Kubernetes provider are you using?
Version 1.16.15 Self Hosted
What version of Argo Workflows are you running?
2.12.9
Example
Result in the command line :
Notes :
(Should be correct according to Workflow template Spec)
Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.
The text was updated successfully, but these errors were encountered: