(Enhancement)Add redis stream output

[maguec]

• Feature additions

What does this PR do?

Added Redis streams output to filebeat

Why is it important?

Redis outputs are already supported and streams is a new feature enabled in Redis 5+
Streams allow for an ordered persistent queue in Redis and is often used in data processing pipelines in more modern applications

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works

Author's Checklist

Check out the redis streams docs https://redis.io/topics/streams-intro

How to test this PR locally

1. create an example filebeat config

filebeat.modules:
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/*.log
output.redis:
  enabled: true
  hosts: ["localhost:6379"]
  datatype: stream     

2. Startup a redis container with a version greater than 5.0 (when Streams was introduced)

docker run -p 6379:6379 redis:5.0.7

3. Add some logfile entries

echo "TEST1" >> /tmp/example.log
echo "TEST2" >> /tmp/example.log

4. Ensure the entries are added to the redis stream

$ redis-cli XRANGE filebeat - +
1) 1) "1583164104376-0"
   2) 1) "payload"
      2) "{\"@timestamp\":\"2020-03-02T15:48:23.372Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.0.0\"},\"log\":{\"offset\":0,\"file\":{\"path\":\"/tmp/example.log\"}},\"message\":\"TEST1\",\"input\":{\"type\":\"log\"},\"host\":{\"name\":\"tamarin\"},\"agent\":{\"type\":\"filebeat\",\"ephemeral_id\":\"94339c8e-4da8-4c16-b9f1-bf9f0c9a6592\",\"hostname\":\"tamarin\",\"id\":\"309ecbd4-2704-4843-ba2c-dc1ffe230566\",\"version\":\"8.0.0\"},\"ecs\":{\"version\":\"1.4.0\"}}"
2) 1) "1583164104376-1"
   2) 1) "payload"
      2) "{\"@timestamp\":\"2020-03-02T15:48:23.372Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.0.0\"},\"host\":{\"name\":\"tamarin\"},\"agent\":{\"id\":\"309ecbd4-2704-4843-ba2c-dc1ffe230566\",\"version\":\"8.0.0\",\"type\":\"filebeat\",\"ephemeral_id\":\"94339c8e-4da8-4c16-b9f1-bf9f0c9a6592\",\"hostname\":\"tamarin\"},\"log\":{\"offset\":6,\"file\":{\"path\":\"/tmp/example.log\"}},\"message\":\"TEST2\",\"input\":{\"type\":\"log\"},\"ecs\":{\"version\":\"1.4.0\"}}"

Related issues

• Supersedes (feature) add redis streams as an output #10703

Use cases

• Use a persistent ordered stream to store logs
• Ingest logs for processing with Redis Gears (http://redisgears.io)

Logs

You can see the addition using redis-cli MONITOR command

1583020615.597469 [0 172.17.42.1:41816] "XADD" "filebeat" "*" "payload" "{\"@timestamp\":\"2020-02-29T23:56:54.596Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.0.0\"},\"log\":{\"offset\":46,\"file\":{\"path\":\"/tmp/fml.log\"}},\"message\":\"test4\",\"input\":{\"type\":\"log\"},\"ecs\":{\"version\":\"1.4.0\"},\"host\":{\"name\":\"tamarin\"},\"agent\":{\"type\":\"filebeat\",\"ephemeral_id\":\"e247baf1-fed2-47e9-a6ce-73fbd4a4ea3e\",\"hostname\":\"tamarin\",\"id\":\"309ecbd4-2704-4843-ba2c-dc1ffe230566\",\"version\":\"8.0.0\"}}"

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[cla-checker-service]

💚 CLA has been signed

[maguec]

Signed the contributor agree ment

[sayden]

Hi @maguec 🙂 Thank you for contributing to this. I guess this is a WIP PR.

You have fill all the fields in the title, including What does this PR do?, Why is it important?, you need to add tests too and the section of how to test locally because someone (even me) might test this manually before the next release and a detailed set of instructions must be provided. Look at an example here: #16001

[maguec]

@sayden Local test case and better documentation of the change has been updated

[sayden]

You need to add proper integration testing, like the ones here https://github.com/elastic/beats/blob/master/libbeat/outputs/redis/redis_integration_test.go

Also, this change is fundamentally bigger than it seems because it requires v5 and right now we only support v3.2.4.

CI complains about few things that has not been done in the repo, run make check to see what

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b add_redis_stream_output upstream/add_redis_stream_output
git merge upstream/master
git push upstream add_redis_stream_output

[mergify]

This pull request does not have a backport label. Could you fix it @maguec? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[jlind23]

@maguec - Closing this one as there were no activity for a while