Defining a set of related log formats sharing common subset of fields

[piotr-dobrogost]

I have a couple log types (each log is in JSON Lines format) where there are fields common to all types and where each type adds additional fields specific to this type.
I would like to create log formats for them but in a way where I define common subset of fields only once and reference this subset in each format.
In other words I would like to create a base log format and create additional formats by extending this base format.

Is this possible?

[piotr-dobrogost]

@tstack Any thoughts?

[tstack]

In other words I would like to create a base log format and create additional formats by extending this base format.

I'm not really a fan of adding any sort of hierarchy. It tends to make things really complicated and hard to reason about. I would just copy and paste between formats. If you really wanted to go further, you could write a python script or something to generate the files since it's easy to spit out a bunch of JSON.

Really, though, some concrete examples of what you're after would help me understand what you're after.

[piotr-dobrogost]

Meanwhile, we have moved to ecs format for logs with all "non-standard" fields being in our own namespace (key).
However, the problem described in this issue remains the same; I would like to extend the ECS format with our custom fields without resorting to making a copy of the ECS format, which is prone to the format becoming out of sync in case it is changed in the future.

I would like to create a new format, stating that it's based on the ECS format and then only specify changes and enhancements to the base format.
In terms of implementation, all that is needed is a new field "based_on": "<name of another (base) format" and deep merging of my specification with the original one to get the final one.