expressions and labels

[static-moonlight]

I don't know how this is supposed to be used. Kubernetes labels usually have the format x/y=value, with x often being a fq domain name.

An then there are the workflow variables workflow.labels.<NAME> and workflow.labels.json.

They don't seem to mix very well. How on earth is this supposed to be used to end up with a valid expression? The closest I got to work was this:

- when: "{{ workflow.labels.x/y }} == 'value'"

And this only works when that label exists. If it doesn't exist (which in my case can happen):

 Bad Request: templates.my-workflow.tasks.my-task failed to resolve {{ workflow.labels.x/y )}}

or

Invalid 'when' expression [...]: Invalid token: '{{=' (hint: try wrapping the affected expression in quotes ("))

I also tried these:

• when: "{{ workflow.labels['x/y'] )}} == 'value'" (this should be possible, according to the Argo documentation)
• when: "{{ workflow.labels.['x/y'] }} == 'value'"
• when: "{{ workflow.labels?.x/y}} == 'value'"
• when: "{{=jsonpath(workflow.labels, '$.x/y')}} == 'value'"
• ... and lots of others ... (in brute force mode ... pure desperation/frustration ...)

So, how do I build a condition using when which checks if a Kubernetes label, which may or may not exists, has a specific value?

[agilgur5]

when: "{{ workflow.labels['x/y'] )}} == 'value'" (this should be possible, according to the Argo documentation)

You're missing an equals sign -- try when: "{{=workflow.labels['x/y'] == 'value'}}"

No equals sign in when uses the legacy govaluate syntax, which does not work with the brackets, which are an expr syntax. See also #7831 and #7576. Per my last comment there (#7576 (comment)), would like to deprecate it, but there's several steps involved in that since it would break existing conditionals.

• when: "{{ workflow.labels?.x/y}} == 'value'"
• when: "{{=jsonpath(workflow.labels, '$.x/y')}} == 'value'"

This one is pretty confusing, detailed in #11549 (comment). Basically Workflows often replaces the full, primitive variable and there are no data structures with "nested variables" -- i.e. I'm not sure if workflow.labels would even work, it may just replace workflow.labels.mylabel. Or it might work with workflow.labels as that's an array, but not with workflow, for instance.
I believe we can make this work with expr these days though; I think that logic predates expr's usage. Also requires a non-trivial refactor.

[static-moonlight]

You're missing an equals sign -- try when: "{{=workflow.labels['x/y'] == 'value'}}"

Thank you. I missed that one.

I got a little bit further, but not much. What I am trying to do is this:

  metrics:
    prometheus:
      - name: status_warning
        help: workflow warnings
        when: "{{=workflow.labels['something/status']}} == warning"
        labels:
          - key: name
            value: some-name
        counter:
          value: "1"

Within my workflow there are multiple (dag) tasks, gathering data. Some of them can fail because the data is just not there, and nobody can do anything about it. It's considered to be a soft error, more like a warning. One part of making that visible, would be to just count them.

The easiest thing would be if I could just access the exit code of those (dag) task directly within the metrics when expression. But I don't see a way to actually to that here. So I am currently trying to just set a label to the workflow (basically flagging it into the warning state) and then check in the expression for the metric, if the counter needs to be updated or not.

However, the workflow controller still don't like that expression:

[...] level=error msg="unable to substitute parameters for metric 'result': failed to evaluate expression \"workflow.labels['something/status']\"" namespace=[...] workflow=[...]

What am I missing? do I still have some syntax error in here?

EDIT:

Alternatively, I was playing around with this:

      - name: result
        help: workflow result
        labels:
          - key: name
            value: some-name
          - key: status
            value: "{{=workflow.labels['something/status']}} == 'warning' ? 'warning' : {{=lower(workflow.status)}}"

But again, the workflow controller doesn't like the expression {{=workflow.labels['something/status']}}

EDIT2: I think the expression, even if correct syntax-wise, fails when the label doesn't exit.

[agilgur5]

        when: "{{=workflow.labels['something/status']}} == warning"

Note that this is not quite the same as what I wrote; I wrapped the entire expression in braces. That way it uses expr entirely. Whereas your variant uses both expr and legacy govaluate.

The way I wrote it was:

when: "{{=workflow.labels['something/status'] == warning}}"

That might work better when the label doesn't exist, although I'm not 100% sure.

The easiest thing would be if I could just access the exit code of those (dag) task directly within the metrics when expression.

Metrics do have an exitCode variable. You didn't post your full Workflow though so I'm not sure if there's some other missing context

[static-moonlight]

Note that this is not quite the same as what I wrote; I wrapped the entire expression in braces. That way it uses expr entirely. Whereas your variant uses both expr and legacy govaluate.

The way I wrote it was:

when: "{{=workflow.labels['something/status'] == warning}}"

Oh, right. I kind of missed that. I will try that.

Metrics do have an exitCode variable. You didn't post your full Workflow though so I'm not sure if there's some other missing context

That's true, for template-level metrics.

Technically I could implement that on template-level. However, the more global logic would be to either report the workflow as warning, or use the standard argo results (like "Succeeded" or "Failure"). That means, to avoid reporting a workflow twice (as warning on template level and error on workflow level) I still need the information about missing data for the workflow level metric (something like: "if task x reported missing data, then increase warning counter, otherwise increase <workflow.status> counter"). And there I can't seem to access the tasks exit code. That's why I'm using Kubernetes labels as a global variable to make that info available where I need it. I admit, it's hacky, and not entirely clean, it's pure desperation.

[agilgur5]

That's why I'm using Kubernetes labels as a global variable to make that info available where I need it. I admit, it's hacky, and not entirely clean, it's pure desperation.

IMO, I would just report both and then do subtraction in your metrics queries if you really need to. That would be substantially less hacky and would make sense for metrics; they're all counters and some overlap

[static-moonlight]

In the end I did exactly that. It comes with a small inconvenience though. Since the missing-data metric is reported early, sometimes I end up with negative (-1) "other" failure causes. Not a big deal, but maybe I find a way to fix that.

However, there is still a lot of inconsistency with expressions. Sometime they work. And when you put them somewhere else is suddenly does not.

Examples:

1. I couldn't get this to work for a task parameter in an exit handler:

{{=jsonpath(workflow.failures, '$.message')}}

... it just wasn't resolved at all. I just wanted to know if there a failures which indicate missing data. In the end I did this, because I could find a cleaner way to do it. Handling workflow.failures as json was just not possible.

        tasks:
          - name: flag-missing-data
            templateRef:
              name: workflow-templates
              template: flag-missing-data
            arguments:
              parameters:
            when: "{{workflow.status}} in (Failed, Error) && {{=sprig.contains('exit code 123',workflow.failures)}} == true"

(even finding out how to use sprig.contains correctly was challenging)

2. Expressions in sensors behave completely different. In a sensor I can do this:

            - dest: metadata.generateName
              src:
                dependencyName: my-dependency
                dataTemplate: 'create-{{ .Input.body.data.key | replace "." "-" }}-'

I don't understand why I don't need to use {{= here and apparently piping is working here as well. If I try to do that in Argo workflows I get some error.

I had a lot of wtf-moments with Argo expressions lately. Plus the documentation is often not helping. Enough to really get frustrated and trying to avoid expressions as best as possible. In some cases I even used workflow tasks to do little string manipulations and export the results as output parameters, because the expressions let me down.

Generally speaking: I really wish the expressions would be more consistent across the board.

[agilgur5]

However, there is still a lot of inconsistency with expressions. Sometime they work. And when you put them somewhere else is suddenly does not.

I did mention this in my very first comment above and linked out to a few long-standing issues. See also #9529 (comment)

I could definitely use some help with those. Unfortunately, I can't clone myself 10x (to the disappointment of many of my former colleagues) and am an unpaid volunteer, so I physically don't have time to do everything (nor get paid to) 🙃

In the interim, learning the currently 3 existing templating engines is helpful. Like that when is the only place that uses govaluate by default and the difference between {{ (fasttemplate/simple replace) and {{= (full expr expressions).
And, in particular, as I mentioned in my above comment, I would just suggest using expr style expressions with {{= everywhere, as they are more future-proof and not deprecated/legacy. That will give you consistency as well.

(even finding out how to use sprig.contains correctly was challenging)

It's just a function in expr. Workflows has no Go template syntax like Helm has

2. Expressions in sensors behave completely different. In a sensor I can do this:

Events has different maintainers (all 4 core Argo projects have different active maintainers) and is unfortunately not very well maintained these days. I don't contribute to it (and didn't use it much as a user), so I can't speak on their behalf. But it seems like they allow Go template syntax and vaguely follow Helm conventions(? such as capital .Inputs akin to .Values).

Also CD has a more than one templating system as well, including Go template and expr in different places 🙃

Same as I wrote above, knowing what project and what templating engine you're using helps quite a bit. Also yes it'd be great if all Argo projects used the same templating engines everywhere for consistency 😐 Feel free to open issues in the other projects if they don't have them already.

Plus the documentation is often not helping.

They're pretty bare on expressions. There are examples/ though and expr has a playground. Following my suggestions above, it is pure expr syntax with all the variables available and some custom functions added (like jsonpath)

In some cases I even used workflow tasks to do little string manipulations and export the results as output parameters, because the expressions let me down.

To be fair, sometimes you literally can't do something in an expression and so need to do that anyway. The beauty of Workflows is that you can easily do that and get what you need.

Generally speaking: I really wish the expressions would be more consistent across the board.

You're far from the only one wanting that -- #10017 was my first ever issue and the previous top contributor to Argo (Alex Collins) wrote up many of the templating engine consistency issues I linked above. I have also organized all of those issues and plans around that. They just need implementation work at this point. Contributions more than welcome; OSS is all a community effort often run by people with enough passion, time, responsibility, need, or desire.