Painless as a JSON-friendly templating language

[clintongormley]

The problem

Today our only choice for a templating language is Mustache which, while fine for very simple templates, does not work well in the JSON context where we mostly use it (eg search templates, watches etc).

For instance, if you want to return an array of terms in a query, you can't do:

    "terms": "{{ #toJson }} my_terms {{ /toJson }}"

because that would render as:

    "terms": "[\"term_one\",\"term_two\"]"

Instead, the whole template has to be passed as one big string in order to remove the double quotes:

    "terms":  {{ #toJson }} my_terms {{ /toJson }}

Dealing with big queries as a string instead of as a JSON structure is complicated, finicky, and error prone.

On top of that, Mustache's limited support for defaults, conditionals, etc make anything but variable substitution difficult and hacky. Just see https://www.elastic.co/guide/en/elasticsearch/reference/5.4/search-template.html#_conditional_clauses as an example of how ugly it can get.

Making it JSON friendly

Instead, we need a templating solution where JSON is parsed as JSON, and any string element is treated as a potential template, the result of which can be any JSON-compatible data structure.

For instance:

"<< my_undefined_val >>"  →   null
"<< my_bool >>"           →   true
"<< my_num >>"            →   5.4
"<< my_string >>"         →   "some string"
"<< my_array >>"          →   [ "foo", "bar" ]
"<< my_map >>"            →   { "foo": { "bar": "baz" }}

Additionally, templates can be embedded in strings in combination with non-templated strings:

"query": "status:<< status >> title:( << title >>)"
→
"query": "status:active title:( some search keywords)"

Any JSON-compatible data structure could be stringified to its JSON equivalent when used as an embedded template, eg:

"msg": "Here are the broken nodes: << array_of_nodes >>"
→
"msg": "Here are the broken nodes: [\"first\", \"second\"]"

Templates in JSON keys would always be stringified.

Using Painless

Painless is a fast, safe, and powerful scripting language with strong typing; in other words, parameters retain their types instead of just being considered as strings, in the way Mustache does.

Painless can be used as is with a wrapper implementing the stringification mentioned above. By far the majority of templates use simple value substitution:

{
  "template": {
    "lang": "painless",
    "params": {
      "statuses": [
        "active",
        "pending"
      ]
    },
    "inline": {
      "query": {
        "terms": "<< params.statuses >>"
      }
    }
  }
}

Default values:

    "range": {
        "age": {
          "gte": "<< params.min_age ? params.min_age : 21 >>",
          "lte": "<< params.max_age ? params.max_age : 95 >>",
        }
    }

Conditionals:

    "must": [
        "<< if (params.only_active) { return [ 'term': [ 'active' : true ]] }; return; >>",
        ...
    ]

String manipulation:

    "emails": "<< params.emails.join(', ') >>"

It could also be extended to make common templating use cases more succinct, eg

• easier default values for undefined variables (eg params.num || 10
• easier map and filter functions for list manipulation
• addition of functions to escape URI or HTML etc
• Support for JSONpath or similar to do, eg:

def new_nodes = jpath(params.nodes_info, '$.nodes[?(@.version > \"5.2.0\")].name' );

Of course, these additions would be backwards compatible, so old templates will keep working while new templates can take advantage of any new syntax.

[clintongormley]

CC @jdconrad, @nik9000, @s1monw, @spinscale, @colings86, @talevy

[rjernst]

I haven't thought through the syntax all the way (although << and >> bug me a little because {{ and }} or ${ and } is common in other languages), but as far as the "wrapper" part, I think this needs to actually be a separate scripting language (eg painless_template), since it will have a different grammar.

[clintongormley]

when playing around with a new templating language, we started using << and >> because it stood out more than {{ and }} inside JSON, but I'm easy either way.

I think this needs to actually be a separate scripting language (eg painless_template), since it will have a different grammar.

why does it need a different grammar? users already use painless, why do they need to learn a new language for the templating? remember that 90% of use cases will be simple value substitution

[tlrx]

+1, this is something I have in mind since Painless went out and I think @jdconrad thought about it too (it reminds me the JSP with <%...%>, <%=...%> etc).

We added some functions to Mustache like toJson but honestly it was a pain and the library we use does not help when it comes to add new functions for specific needs. I hope Painless might help on this too.

[tvernum]

My concern with the proposal as it stands, is that it's a bit "magical" when it comes to dealing with a template-expression inside of quotes.
Assuming num is the integer 4, then I assume the following are true:

• "<< num >>" = 4
• " << num >>" = " 4"
• "<< num >> " = "4 "
• "<< num.toString() >>" = "4"

The spec would be something like:

If a string consists of a template-expression and no other text, then the string itself is replaced with the JSON representation of the expression value. Otherwise the template-expression is replaced within the string, by substituting in its String representation (*)

(*) Or should be that be the Stringification of its JSON representation?

It's user friendly 99% of the time, but it feels like a little bit too much magic to describe and explain well.

That said, I don't have an alternative that I like.
An option (that is more precise, but confusing to users) would be to have 2 different syntaxes, one for "this replaces the containing string with JSON representation" and another for "this does in place substitution"

• "<< num >>" = 4
• " << num >>" = Error
• "<< num >> " = Error
• "<< num.toString() >>" = "4"
• "$< num >" = "4"
• " $< num >" = " 4"
• "$< num > " = "4 "
• "$< num.toString() >" = "4" or "\"4\"" depending on whether we call toString() or toJson().toString(), I assume the former.

[clintongormley]

• " << num >>" = Error
• "<< num >> " = Error

I thought this could be a solution that would force the user to do something like this instead:

"<< ' ' + num >>" = " 4"

But that makes it harder to write things like:

"The << job >> has failed << num >> times"

[s1monw]

I haven't thought through the syntax all the way (although << and >> bug me a little because {{ and }} or ${ and } is common in other languages), but as far as the "wrapper" part, I think this needs to actually be a separate scripting language (eg painless_template), since it will have a different grammar.

@rjernst I think what @clintongormley is saying is that we would simply parse JSON and if we see a string sequence << (.*) >> we would interpret the group as a painless script. So each template might have N scripts attached to it. I am not 100% sure how that would scale but it sounds reasonable to me from a 10000ft perspective. I don't think it needs a new syntax?

[rjernst]

what @clintongormley is saying is that we would simply parse JSON and if we see a string sequence << (.*) >> we would interpret the group as a painless script.

I realized that much later after my initial comment. But I agree with @tvernum's sentiment that this magic value handling will be error prone. I think instead we should do 2 things:

• Use a single identifier at the beginning of a string which means "this string will be substituted by the resulting evaluation of the string as painless code". I suggest # as it is not a valid operator or identifier in painless.
• Extend painless to allow variable substitution in strings as done in other languages (eg ${ }). This is to avoid confusion with an operator like << from painless. By using a different syntax for variable substitution, we avoid the possibility of "accidentally" making the entire script a substitution in a string.

So, with my proposal, the example between @clintongormley and @tvernum above could be done a few ways:

• "#num" = 4
• "#num.toString()" = "4"
• "# num" = 4
• "# \" ${num}\"" = " 4"
• "# \" ${num + 1}\"" = " 5"
• "# num + 1" = 5

Alternatively, if we don't want to add variable substitution in painless strings, we could just use two different characters to mean "replace the node" vs "replace the contents of the string". But for both, I would just use a single character at the beginning of the string (and in both cases, the rest of the string is all painless code).

I am not 100% sure how that would scale

I think we can make this work, by painless knowing handling the entire template. @jdconrad and I thought through this when working on scripting contexts, and I think this can still be achieved by painless taking in the entire json object, and compiling that into a script which makes statements for each substituted element, and then joins them together into a serialized json string at the end.

[clintongormley]

Use a single identifier at the beginning of a string which means "this string will be substituted by the resulting evaluation of the string as painless code". I suggest # as it is not a valid operator or identifier in painless.

But it is a common character in strings, eg ref numbers. Btw, I'm not tied to << >>, I'd be happy to use {{ }} as well, or even %% %%. I just preferred the angle brackets because they stand out from json more.

Extend painless to allow variable substitution in strings as done in other languages (eg ${ }).

I think this makes the simple case really ugly, eg compare:

"The << job >> has failed << num >> times"

with

"# \"The ${job} has failed ${num} times\""