[Journald input] Update `include_matches` to reach feature parity with what is exposed by `journalctl`

[belimawr]

#40061 is changing how we read journals in the journald input, as part of this change include_matches.and and include_matches.or have been removed and a simpler matchers implementation has been added.

This issue is about achieving feature parity with the functionalities provided by journalctl.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[belimawr]

@nimarezainia, @jlind23 I could use some help here to better understand if we're missing any specific filtering option/feature in the Journald input.

I created this issue when we migrated from go-systemd to using journalctl directly. The old implementation allowed for very complex filtering using include_matches.and and include_matches.or that are just not supported by journalctl. Honestly I'm not completely sure that everything that was a valid configuration for Filebeat would work as expected.

Reading journalctl docs:

If one or more match arguments are passed, the output is filtered
accordingly. A match is in the format "FIELD=VALUE", e.g.
"_SYSTEMD_UNIT=httpd.service", referring to the components of a
structured journal entry. See systemd.journal-fields(7) for a
list of well-known fields. If multiple matches are specified
matching different fields, the log entries are filtered by both,
i.e. the resulting output will show only entries matching all the
specified matches of this kind. If two matches apply to the same
field, then they are automatically matched as alternatives, i.e.
the resulting output will show entries matching any of the
specified matches for the same field. Finally, the character "+"
may appear as a separate word between other terms on the command
line. This causes all matches before and after to be combined in
a disjunction (i.e. logical OR).

I can see we support pretty much everything. The only thing missing is the logical OR (+) operand.

I'm thinking about keeping it simple, and just accept + as a valid entry, effectively allowing the users to build their filtering expression like they were using journalctl.

Given an expression that is accepted by journalctl:

A=a + B=b C=c + D=d B=b

The documentation is not very clear how multiple + work in terms of precedence, however based on some tests, it is equivalent to the following boolean expression:

A=a OR (B=b AND C=c) OR (D=d AND B=1)

I propose we accept it as:

    include_matches:
      match:
        - A=a
        - +
        - B=b
        - C=c
        - +
        - B=1

If we use the old "syntax" (<= v8.15.0), then we'd have:

    include_matches:
      or:
        - match:
          - A=a
        - and:
          - match:
            - B=b
            - c=c
        - and:
          - match:
            - D=d
            - B=1

Which, at least for me, looks more complex. The implementation will also be more complex. journalctl documentation does not mention anything about grouping, which this old syntax allows for.

What do you folks think? Shall I proceed with the first option I presented?

If you folks want to do some testing yourselves you can use journal.tar.gz. It contains a journal file ready to use, matches.export is the "export" format that can be used to generate a journal file following those instructions. If you get any error from your journalctl, just re-generate it, the newer versions generate a journal file that is not compatible with the older ones. matches.txt is a text file with the message and fields from the entries in the journal. The message is just the extra fields I added, so we can easily test and assert the behaviour. Here they are:

MESSAGE=A=a
A=a

MESSAGE=B=b
B=b

MESSAGE=C=c
C=c

MESSAGE=D=d
D=d

MESSAGE=A=a B=1
A=a
B=1

MESSAGE=B=1 C=3
B=1
C=3

[jlind23]

I'm thinking about keeping it simple, and just accept + as a valid entry, effectively allowing the users to build their filtering expression like they were using journalctl.

That suits me well, let's start simple and not keep any old behaviour that we will have a hard time supporting.

TL;DR: first solution is better IMO

[cmacknz]

Yes, just allow passing through directly to journalctl with the syntax journalctl already supports. We do not need to rebuild journalctl features in the beats pipeline.

[cmacknz]

One thing to keep in mind, we are building the input to a shell command here make sure that things are properly escaped so that it isn't possible to remotely execute arbitrary shell scripts by passing in malicious expressions here.