Identifying strings are expressions in place where the string might also be a constant

[fjtirado]

Currently we can write

set: 
  name: Javi

or

set: 
   name: ${.input.name}

However $ is a reserved keyword in JQ and {} are used in json, think of this expression written in json

"set": {"name": "${$loopVariable}"} 

(I wrote it wrong three times :))

I think we should consider following options

1. Everything is an expression

This implies user has to escape literals.

set: 
  name: '"Javi"'

set: 
   name: .input.name

2. Change the expression prefix.

Use something different than ${} for example, use ``

set: 
  name: Javi

set: 
   name: `.input.name`

We can use a different character, for example, #, but with this one we need to add white spaces around the expression

set: 
   name: # .input.name #

3. Use a new keyword.

So after the property name, you expect either constant or expr and use as default expr (that default might be overriden by user in the header of the workflow definition)

• Default to expr:

set: 
  name:
    constant: Javi

set: 
   name: .input.name

• Default to constant:

set: 
  name:  Javi

set: 
   name: 
     expr:  .input.name

[gibson042]

Relevant prior discussion: #285 (#285 (comment) even includes several of these options)

[cdavernas]

@fjtirado as discussed in PM, I totally understand your point of view, which is documented insightfully by others in the issue shared by @gibson042 (thank you!).

As I have never been bothered by the current approach, which I actually like, I'm probably not the most objective person when it comes to that subject.

I did, however, my due diligence, and have checked what major clouds are proposing, and it seems they are all using a similar mechanism. Hell, Google Workflows, which is my personnal favorite, uses the exact same than us.

Google Workflows, however, recommends single quoting expressions in YAML (which is/should be the authoring language) to avoid many of the problems you guys mentionned.

My proposal is:

1. Document single quoting recommendations
2. Optionally provide a top level property that allows configuring the workflow's expressions. Something like:

document:
  dsl: '1.0.0-alpha1'
  namespace: example
  name: example-workflow
  version: '0.1.0'
evaluate:
  language: jq
  expressions: explicitly | implicitly

Where explicitly defines that expressions must be clearly identified by '${ }', save when they are not explicitly required (as opposed to a when/exceptWhen condition, for example).
implicitly defines that everything is to be considered and run as an expressions. Constants must be clearly identified.

Also, I'm not against replacing the actual '${ }' with something else, as suggested in 2.

Finally, while I believe 1 is a great way to go, too, I think it is best "enabled" using my proposal, rather than by default. The reason is that, in the end, this issue is really a matter of tastes, perceptions and habits. Enforcing one or the other (such as it has been done so far) is IMO not the way to go.

What I would avoid is:

3. Use a new keyword.

It's more complicated, less readable, less "user-friendly", and a IMO a PITA to implement (where before I could simply regex my way out of it, I must know define special constructs to "extend" normal properties).

My opinion is that option costs much more than it gives.

---

As for suggestions to replace the current approach with an expr prefix, I believe it bears far too much "technical" meaning, which might put off "business people", who have a lot to gain with workflows, too.

[fjtirado]

@cdavernas
Google Expressions does not use JQ, thats why starting with $ is not an issue.
The model is not a json structure, thats why using {} is not an issue either.
Maybe it was lost in my long description but this expression in json looks really confusing when defining the workflow in json.
"set": {"name": "${$loopVariable}"}
We at least need a different enclosing set of characters.
For example
"set": {"name": "#$loopVariable#"}

[cdavernas]

when defining the workflow in json.

Well, why would you go through the sheer pain of defining anything in JSON when you can do it in YAML, I wonder?

[fjtirado]

@cdavernas
Believe or not, for me is easier to use {} (json) than tabs (yaml) (I am suspect that for other developers that consider that python practise a bad joke)
Regardless my preferences, since the definition file can still be written in json and even in yaml you might want to write expressions that return a json, for example
set: '${{loopVariable:$loopVariable}}'
I am really surprised thay you consider ${} a good separator for a specification using JQ and Json data everywhere just because it was done that way before (specially when we are defining a completely new DSL)

[cdavernas]

Believe or not, for me is easier to use {} (json) than tabs (yaml)

I believe you, and that's your right. But I am convinced that it will probably not be the case of the majority of our (especially non-technical) audience, which I'd rather focus on (as per design).

I am really surprised thay you consider ${} a good separator for a specification using JQ and Json data everywhere just because it was done that way before (specially when we are defining a completely new DSL)

Well, first, it's a matter of taste and colors, really. Second, I was looking to what was done where and why. The reasons identified and addressed by our "competitors" is of high interest to me, especially Google Workflows, which I admitted to be really fond of, on a purely syntaxic perspective.

The fact is that while having to build a JSON string should be doable, it's rather something you'd like to avoid, as you'd loose control over the data, which becomes opaque to any component of you workflow.

Please rest assured I read through and through the discussion we had on slack, the issue #285 and the new example supplied above. My opinion/taste remains the same nonetheless.

I therefore propose we all avoid trying to convince each other, and rather provide constructive solutions that could address the issue at hand!

I still believe we are speaking of edge cases twhich can be addressed in a way that becomes painfull for standard cases.

Let's remain concise in this issue, and move discussions to... discussions. I propose we only make and discuss proposals in here!

On a final note, it might be interesting to document those kind of fundamentals issues in ADRs, what do you @ricardozanini
@JBBianchi @fjtirado @gibson042 think?

[fjtirado]

Given your personal preferences, my proposal is to change ${} by ## $() (which is more neutral in jq and json)
Updated
See #848 (comment)

[cdavernas]

@ricardozanini @JBBianchi @gibson042 @jorgenj WDYT?

[fjtirado]

In #285 I think we were both discussing (makes sense at that moment) expression interpolation (which we are not supporting anymore, since we are deferring to the expression language) with expression identification (and the fact that were forcing the usage of ${} in places were it was not really needed because it was obvious the string was expression, problem that is not there anymore)
The only problem that remains is to identify expression in places where it is not evident that the string is an expression or a literal. So far, in the new DSL, I believe that will only occur in few places (set: and body:). We still have the same options that were collected in this comment, but we can ignore pros/cons related with interpolation and focus on identification.

[fjtirado]

and now my two cents, The markup proposal (either ${} or ##), since it can be used for interpolation, does not clearly send the message that we are NOT supporting it.
Since we just want to identify expressions, and, in my opinion, most of the time the user will be using expression on set: and body: rather than literals, I think I prefer option 3 (this is just personal preference, since I do not foresee implementation problems, but you never know till you actually implement it).
However, I agree that this creates another problem, which string to use as keyword so it does not collided with a "real" property? (but "$expr" and "$const" looks like a safe bet in that regard )
Since there is not unanimity in adopting 3., at the end, rather than pushing for it, I prefer to just change the markup we chose, clearly document that we are not supporting expression interpolation and allow user to indicate (as @cdavernas proposed) that all expressions are implicit
Therefore, we can adopt #848 (comment) , also explicitly documenting that interpolation is delegated to the expression language

[fjtirado]

I realize # is a poor choice for yaml because is the comment character (json does not have comment character and it was a good choice for it)
I would say we should consider using different expression markups for json and yaml if this were not complicating automatic conversion between the one and the other. So thats not a good option either
Given those constraints I would use $(), since neither $, ( and ) are special charaters in yaml/json (it still contains the $ that is used in jq, but the example I mention looks better in my opinion, because the braces to enclose the expression are different from the braces that enclose the json object)
"set": {"name": "$($loopVariable)"}
Therefore I propose to change the markup from ${} to $()
This allows you to write expression in yaml without actually quoting (or adding white spaces) them (as long as you are not building a json object or the expression does not contain a math operator)
Therefore, in yaml you will write

set: 
  name: $(input.name)

when currently you have to write

set: 
  name: ${ input.name }

or

set: 
  name: '${input.name}'

[ricardozanini]

1. +1 for the proposal to set implicit | explicit at the beginning of the workflow definition. We might explore the usage of expressions: strict or strict: true since it's widespread among many languages. Again, a matter of choice since we all agree that we need this feature.
2. +1 to change from ${} to $() to avoid misinterpretation with JSON expressions. Although one can argue that () is super common in programming languages that might also bring misinformation to the expression. From jq manual: Parenthesis work as a grouping operator just as in any typical programming language. Again, a matter of choice.
3. Welcome back @gibson042, please stay around