New aws_cloudtrail_logs source

[binarylogic]

This source will collect events from AWS CloudTrail and ingest them into Vector as log events. We need to investigate what this integration will look like, and therefore a spec/RFC is in order.

[jszwedko]

I discovered you can forward CloudTrail logs to CloudWatch Logs so #3566 could be tractable for getting CloudTrail logs out too; however it has a pretty big limitation:

Because CloudWatch Logs has an event size limitation of 256 KB, CloudTrail does not send events larger than 256 KB to CloudWatch Logs. For example, a call to the EC2 RunInstances API to launch 500 instances will exceed the 256 KB limit. CloudTrail does not send the event to CloudWatch Logs. To ensure that CloudTrail sends events to CloudWatch Logs, break large requests into smaller batches.

https://docs.aws.amazon.com/awscloudtrail/latest/userguide/send-cloudtrail-events-to-cloudwatch-logs.html

I'm imagining we'll prefer the traditional route of CloudTrail -> S3 with a new Vector S3 source.