feat(inputs.firehose): Add new plugin

[syedmhashim]

Summary

The firehose input plugin would be used to receive data from AWS Data Firehose.

Checklist

• No AI generated code was used in this PR

Related issues

resolves #15870

[srebhan]

@syedmhashim I do have some comments, mostly on style, so overall this looks quite promising. I urge you to keep the code as simple and less nested as possible to ease review and debugging. And of course some unit-tests would be great!

[syedmhashim]

@syedmhashim I do have some comments, mostly on style, so overall this looks quite promising. I urge you to keep the code as simple and less nested as possible to ease review and debugging. And of course some unit-tests would be great!

Thanks for the review. Will definitely go over the comments and update accordingly. Just FYI, I have been using http_listener_v2 input plugin as a reference to write the code. The unit tests will come soon as well

[srebhan]

Yeah we do have some older code we didn't adapt yet but things changed both on the golang side as well as on us being more strict with the way things are done. ;-) No worries, we will figure it out together. :-)

[srebhan]

@syedmhashim I'm loosing a bit the track in the discussion above, could you please push an update with the stuff that is clear and then we discuss the unclear parts?

[syedmhashim]

@srebhan Hey! Just pushed some changes and commented "done" on the threads that I resolved. Sorry for the delay. I got occupied with my job

[srebhan]

That looks much better @syedmhashim! Some more cleanups and comments from my side...

[syedmhashim]

@srebhan Hi! Pushed some changes and added comments where relevant

[srebhan]

@syedmhashim thanks for the update. I tried to put more concrete suggestions into my review. Overall, I think we are almost there, the only thing missing are unit-tests with a mocked sender...

[srebhan]

Really nice @syedmhashim! Some very small things with the biggest being that log messages should start with a captial letter... The only thing missing are the unit-tests...

[syedmhashim]

Hi @srebhan. In addition to the changes that you suggested, I also made some changes to the code(see last commit) that I think would make it more clean and readable. P.S I think it also helps to achieve separation of concern since now all processing of the request is handled by firehose_request.go while firehose.go mostly focus on adding metrics:

Changes to :

• firehose_request.go
  • Added newRequest method to handle the creation of request object and (gzip +) json decoding of the request body
  • Added processRequest method to handle the authentication, validation, base64 decoding and parameter extraction
  • I’ve tried to set the response status code mostly in firehose_request.go
• firehose.go
  • Added handleRequest method. This helps to get rid of the redundant sendResponse code block.

I'd like to hear your thoughts on this

[srebhan]

@syedmhashim thanks for the update. I do have some smaller comments in the code but nothing big.

Regarding the tests, I suggest that you don't test the request separately but simply use a http client and send data to the plugin. You might also want to implement a general test-case setup similar to what we do in socket listener tests...

If you need help with the tests, please provide some payload examples and invite me to your branch and I'll help you to drive this PR over the finish line...

[syedmhashim]

@srebhan Hey. Just sent you an invite! The payload needs to be in the following format with data having the actual b64 encoded text that would be parsed (based on the parser set) after decoding:

{
    "requestId": “test-id",
    "timestamp": 123456789,
    "records": [
        {
          "data": "aGVsbG8gd29ybGQK" // "hello world"
        }
    ]
}

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

🥳 This pull request decreases the Telegraf binary size by -4.52 % for linux amd64 (new size: 261.2 MB, nightly size 273.6 MB)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz