Description
Describe the feature:
With an increasing focus on Observability and Security, most of the processing of data is being moved to Ingest pipelines through beats.
Since ingestion rates are always a top priority on these cases, there is one argument in the set processor that always has to be set currently with a painless condition that would be great having as a boolean argument in Elasticsearch instead, and that is checking for Null values in the "value" field.
In most other processors we are have "ignore_missing" argument, in which it will ignore fields that are either missing or is null, but currently if we are trying to use the set processor on a value that is null, it will add it as a empty string, this is something we can currently prevent with something like this:
{
"set" : {
"field": "source.packets",
"value": "{{ctx.checkpoint?.client_outbound_packets}}",
"if": "ctx.checkpoint?.client_outbound_packets != null"
}
},
Due to logs more often than not is coming in with many different structures, it is impossible to utilise the set processor without either painless or having issues with plenty of fields that are empty strings (instead of null).
Would it be possible to add a "ignore_missing" or similar argument to this field as well? So that it is possible to set a field if the value is not null?