chore(output.kinesis): Log kinesis consumer events

[mskonovalov]

Summary

Currently we have 3 levels of abstraction:
telegraf output kinesis plugin -> kinesis-consumer lib -> aws sdk

Logging is happening only on Telegraf's level. So if there are some warnings or some debug logging on other levels, it cannot be enabled ATM.

In this PR I'm wrapping telegraf logger to comply with interfaces of kinesis-consumer and aws-sdk loggers and pass it when initializing the kinesis client.

For kinesis-consumer there is only a single Log function, so I map it to the DEBUG level.

Also enabling logging of retries in aws sdk.

Checklist

• No AI generated code was used in this PR

Related issues

resolves #15854

[DStrand1]

Thanks for the PR! I just have one comment for now on the code, but could you also open an issue for this PR to resolve? Thanks!

[srebhan]

Thanks @mskonovalov for your contribution! I would mark the inner logs of the library with log-level "trace" to be able to selectively enable those as I don't think this is something users need to see all the time!?

[srebhan]

I would use log-level Trace for all those and add the classification as prefix to the message. This way we can control if we want to see the inner logs of the library or not.

[mskonovalov]

TBH I believe it is better to map Warn inside the library to Warn here as well, though Debug and Info I guess is OK to map to Trace.
But it is up to you as a maintainer. So let me know wdyt and I can fix accordingly

[mskonovalov]

OK, I think I didn't get what you mean here :(
If I change it to Trace, then the prefix will be TRACE, so if I add classification as a prefix it will look like TRACE DEBUG <message>. I guess this will be confusing. Or maybe I'm missing something

[srebhan]

@mskonovalov in normal operations it should not be necessary to see the library-internal log messages! If you map those to e.g. WARN everybody will now see library-internal warnings without a sensible way to ignore those. What I want is that the user CAN enable those log-messages setting the trace level for the plugin in Telegraf if he whishes to see them. You can keep the classification information by prepending the library message like

	switch classification {
	case logging.Debug:
		format  = "DEBUG " + format
	case logging.Warn:
		format  = "WARN" + format
	case logging.Info:
		format  = "INFO " + format
	}
	t.Logger.Tracef(format, v...)

which will produce

<timestamp> T! WARN this is an internal warn message

in case of a warning...

[mskonovalov]

ok, I can make this change.
Although, I'd just mention that I don't fully agree.
If we are using the underlying library, there are not a lot of options how the library can notify us that something is wrong:

1. of course, if the issue is critical - it will return the error
2. if it is just some informational messages, then it is fine to put it under Trace level
3. But what if it is in between: not critical to return error but still something that need to be displayed. E.g. library that writes into some storage can only do it after 3rd retry for every write. The write is successful so it is not returning error, but if we map WARN messages to TRACE level we will never see that something is not working properly.

So I still believe for WARN case it is better to allow to write it to WARN Telegraf log.
But it is up to you to decide.
Though I agree we need to hope that the lib author put only meaningful WARN messages and didn't make it too noisy

[srebhan]

Well TBH if a library requires logging to propagate important information there is something wrong with the library. A library is an abstraction that should allow the user to use the functionality without knowledge of the internal implementation. On error the library should return an error. If something goes wrong the library should either return some information and let the caller decide on how to continue or return an error and provide means to retry accepting the issue.

So at best the logging serves the purpose of debugging some connection issues or other trouble but this should not be necessary in normal working conditions.

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

[srebhan]

Thanks @mskonovalov!

[srebhan]

@DStrand1 assigning it back to you if you want to have another short look as some of the code changed. Feel free to directly merge the PR!