VRL templates

[fuchsnj]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Proposal

Today templates are very limited, they can only contain the following:

• A literal string
• A literal string with a timestamp
• A reference to a single event field

It would be nice to allow the full power of VRL to calculate template values. For example, a field reference in a template today will generally return an error if the field is missing. In VRL, it could choose an arbitrary default.

VRL supports read-only mode (currently used in VRL conditions), which can prevent mutations. This was one of the issues in the previous RFC.

It can also (with a minor change) type check the result of the VRL template to ensure it always returns a string.

Some sinks (such as Loki) allow deleting fields after they are referenced in a template. I suggest that VRL templates are immutable and never modify the event, they only can calculate and return a new value. If there is data that should be used in a template, but not sent to the sink, it should be moved into event metadata first (in a remap transform).

Since the templating logic is very limited today, this can probably be made fully backwards compatible, so either a "legacy" template, or a VRL script can be used in a template. Internally all templates will be converted to a VRL script. A warning could even be generated if the "legacy" templates are being used, with the VRL code that it can be converted to.

Previous RFC: #4905

[jszwedko]

Related: #1692 . We had discussed adding Liquid-style filters before: https://shopify.github.io/liquid/filters/abs/

Are there any performance concerns here to running VRL for templates? Consider a config like #17487 (comment) which has a good number of template strings in it.

[fuchsnj]

We had discussed adding Liquid-style filters before

I would really push for VRL in templates over designing what will essentially be yet another syntax / language that needs to be learned to use Vector unless there is a very strong reason for it.

Are there any performance concerns here to running VRL for templates?

The obvious answer is we just need to measure it, but I don't think that's been done yet. For configs similar to #17487 (comment) I personally don't think there will be any significant performance concerns for similar VRL templates.

A good first step here is to get a quick proof of concept to see exactly what it looks like, and enable us to do some perf tests.