Create a split event processor

[dlvenable]

Is your feature request related to a problem? Please describe.

Some users want to split an input event into multiple events by splitting a field from the input event.

Say I have the following two events:

{"query" : "open source", "some_other_field" : "abc" }
{"query" : "data prepper documentation", "some_other_field" : "xyz" }

I'd like to get the following events:

{"query" : "open", "some_other_field" : "abc" }
{"query" : "source", "some_other_field" : "abc" }
{"query" : "data, "some_other_field" : "xyz" }
{"query" : "prepper", "some_other_field" : "xyz" }
{"query" : "documentation", "some_other_field" : "xyz" }

Describe the solution you'd like

Create a split event processor.

It will require a field which is the field we are splitting on. The value of that field could be either a string or an array. When it is a string, the user must provide a delimiter. This could be expressed as concrete value or a regex.

Example 1: Split events from query based on a regex:

processor:
- split_event:
    field: query
    regex_delimiter: '\\s+'

Example 2: Split events from query based on a delimiter:

processor:
- split_event:
    field: query
    delimiter: ' '

Example 3: Split events from an array query. In this example, we are using split_string first, which would be unnecessary. But, it conveys how the processor works in the case of arrays.

processor:
- split_string:
        entries:
          - source: "query"
            delimiter_regex: "\\s+"
- split_event:
    field: query

[dlvenable]

Some things to consider with this:

• Creating new Events will affect how the acknowledgements work. So we need to be sure we can support this and have tests for it.
• This will cause more events downstream than were received by the buffer. What consequences are there of this?
• One thing to consider with having more events is that the overall memory downstream of the buffer will expand. This may cause more out-of-memory errors.

[kkondaka]

Some things to consider with this:

• Creating new Events will affect how the acknowledgements work. So we need to be sure we can support this and have tests for it.
• This will cause more events downstream than were received by the buffer. What consequences are there of this?
• One thing to consider with having more events is that the overall memory downstream of the buffer will expand. This may cause more out-of-memory errors.

All three are valid points but all these points are also applicable to sub-pipelines where we copy the event, we add the new copy of the event to the existing acknowledgementset. Memory related issues are also applicable to sub-pipelines but here it could be lot more severe because many many copies may be made.
But as a processor, I am not sure what can be done during design/implementation of the processor to address increased memory usage That is something that needs to be evaluated at the pipeline level. The system circuit breaker is more likely to be triggered more frequently. And this will result in back-pressure or throttling of the source.

[dlvenable]

@kkondaka , I agree. These are not points to consider in the design of this process. I'm more calling them out as possible side-effects of adding this processor. Overall, I'm in favor of having it. We still should resolve the other core issues at the pipeline-level. And not do it within a processor.

[sharraj]

processor:

• split_string:
  entries:
  - source: "query"
  delimiter_regex: "\s+"
• split_event:
  field: query

Please make we can specify multiple entries to split with in one processor (instead of specifying multiple processors back to back)