Allow more flexible journal log filters #7096
Conversation
Previously it was only possible to filter on the _SYSTEMD_UNIT property of journal log entries. Users may want to filter their logs on other fields, so this change allows users to use any journal KEY=VALUE filter to include/exclude journal logs from collection.
|
|
|
Yes, very much like I imagined this feature to be 👍 Thanks for the PR! I hope it will be reviewed soon. |
djmitche
added
the
[deprecated] team/agent-core
There was a problem hiding this comment.
Well, this has been open for a while -- sorry about that!
I think this implementation looks pretty good! The issues I see are minor:
- We don't typically deprecate things unless it's impossible to continue to support them. So, I think we'd consider the
.._unitsconfig still supported. - This needs a rebase, but it shouldn't be too bad (the files have moved but not changed too much)
| submatches := matchRe.FindStringSubmatch(match) | ||
| if len(submatches) < 1 { | ||
| return fmt.Errorf("incorrectly formatted ExcludeMatch (must be `[field]=[value]`: %s", match) | ||
| } |
There was a problem hiding this comment.
For consistency, this check should be done on IncludeMatches, as well.
| @@ -31,7 +32,7 @@ type Tailer struct { | |||
| source *config.LogSource | |||
| outputChan chan *message.Message | |||
| journal *sdjournal.Journal | |||
| blacklist map[string]bool | |||
| blacklist map[string]map[string]bool // a map of string to a "set" of strings | |||
There was a problem hiding this comment.
| blacklist map[string]map[string]bool // a map of string to a "set" of strings | |
| blacklist map[string]map[string]bool // blacklist[key][value] -> true, perhaps? |
Also, and I realize you didn't introduce this issue, but since the value of the map is never false, this could be map[string]map[string]struct{}. The struct{} idiom better suggests "set" instead of "map".
What does this PR do?
Previously it was only possible to filter on the _SYSTEMD_UNIT property of journal log entries.
Users may want to filter their logs on other fields, so this change allows users to use any journal KEY=VALUE filter to include/exclude journal logs from collection.
Motivation
We have logs in our journals that are not associated with specific systemd units, but that we don't want to send to datadog.
Also addresses #5579
Describe your test plan
Unit tests have been provided.
End-to-end testing can be done by modifying the journald logging file of a datadog process to use different filters.