[Filebeat] http_endpoint - make input compatible with AWS Kinesis http delivery

[andrewkroh]

Describe the enhancement:

The http_endpoint input should be able to accept data in the format given by AWS Kinesis which is an object containing an array of records. For example:

{
  "requestId": "ed4acda5-034f-9f42-bba1-f29aea6d7d8f",
  "timestamp": 1578090901599
  "records": [
    {
      "data": "aGVsbG8="
    },
    {
      "data": "aGVsbG8gd29ybGQ="
    }
  ]
} 

Today the input only accepts objects or arrays.

Describe a specific use case for the enhancement or feature:

To receive pushed events from Kinesis.

References

• https://aws.amazon.com/blogs/big-data/stream-data-to-an-http-endpoint-with-amazon-kinesis-data-firehose/
• https://docs.aws.amazon.com/firehose/latest/dev/httpdeliveryrequestresponse.html

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)