Description
Since #21088 was included, Filebeat will ignore CloudTrail logs when they are not in the root of the S3 bucket path. This situation can occur if the user is using AWS Organizations to manage their CloudTrail logs, or if they have manually set an S3 Prefix in their CloudTrail configuration.
The following configuration included with the module is intended to be used to filter out different log types based on the user configuration. If a filter is set to allow a log type (e.g. CloudTrail-Digest logs), then the regex is parsed. If the input matches the regex, the logfiles are further expanded and ingested. If a logfile does not match the regex, it is discarded/ignored.
This regex accounts for the default AWS CloudTrail log destination, in which a log object will have the key
AWSLogs/1234567890/CloudTrail/....
however does not account for users specifying a prefix manually, or using AWS Organizations
custom-prefix/AWSLogs/1234567890/CloudTrail/.... or
o-xxxxxxx/AWSLogs/1234567890/CloudTrail/....
I have resolved this in testing by modifying the beginning of the regex to match * up until the AWSLogs string - for example
^*./AWSLogs/\d+/CloudTrail/
I searched around and also found this reference to the issue https://discuss.elastic.co/t/request-for-filebeat-aws-cloudtrail-documentation-configuration-options-for-s3prefix-etc/256801
This failure mode will not appear in logs, as the filter simply ignores any logfiles that do not match the specified path, and handles them accordingly - it's not an error, just a use-case that isn't handled as expected