<pattern>: do not include property when value is empty/null

[brenuart]

Suppose the following pattern:

<pattern>
{
   "trace": "%mdc{traceId}"
}
</pattern>

Is there a way to omit the trace property when there is no such traceId MDC key?
Something like what #nullNA{...} does for Access events...

[osigida]

I'm not sure how to do that, but are you aware that you can filter out empty fields in kibana?

[philsttr]

There is currently not a way to do that when using the pattern JSON provider.

As a workaround, you could do this by implementing your own JSON provider.

PRs welcome.

[brenuart]

@philsttr Is there any (technical) reason why #nullNA{...} was implemented only for Access events?
I already tried to configure the Jackson serializer to filter out empty or null values but it doesn't work here as the various JsonProviders are working directly with the JsonGenerator - hence bypassing that logic.
As you mentionned, a custom JsonProvider does the job indeed.

As of a PR, do you think this is a valuable feature that many people may request?

[philsttr]

Unfortunately, #nullNA{...} does not do exactly what you want. It turns something that evaluates to the string value - into a null value. It does not remove the field like you want.

Is there any (technical) reason why #nullNA{...} was implemented only for Access events?

I think only AccessEvents have values that evaluate to the string -. Nothing in a "regular" LoggingEvent will evaluate to -, so #nullNA{...} wouldn't do anything for LoggingEvents. IIRC, most logging event patterns evaluate to an empty string, not -.

From looking at the current code, it would be a little complicated (but not impossible) to implement a directive that will omit a field when its value is null.

Alternatively, I'm wondering if this would be simpler/better if the pattern provider could just be configured to omit fields containing null values, versus having to do so piecemeal with a special directive in the pattern itself. Just thinking aloud. (Might also want to consider omitting empty string values?)

For example

    <pattern>
      <omitFieldsWithNullValue>true</omitFieldsWithNullValue>
      <pattern>
        { "level": "%level" }
      </pattern>
    </pattern>

do you think this is a valuable feature that many people may request?

Yeah, seems worthwhile to me.

[kharnt0x]

we would definitely welcome this feature

[saikirankochi]

How can we achieve the above scenario

[abhishek-jain11]

Any plans in near future for this?

[philsttr]

@abhishek-jain11 Not at the moment. PRs welcome.

[wsargent]

You could do this by writing a class extending LogstashBasicMarker and writing out to the generator only if that MDC value existed.

[philsttr]

@wsargent If my understanding is correct, this request is more about excluding a specific field within a pattern based on the the field's evaluated value, rather than excluding the whole pattern based on a value.

i.e. if the pattern is something like this:

{
    "trace": "%mdc{traceId}",
    "foo": "bar"
}

Then only the trace field would be omitted if the field value evaluated to null. The foo field would still be included regardless.

You could work around this by separating out every possible nullable thing into a different pattern, but that would be cumbersome when working with a large pattern.