Fix metrics reported as written but not actually written

[MyaLongmire]

Required for all PRs:

• Updated associated README.md.
• Wrote appropriate unit tests.

resolves #9514

Reworked code to handle more than database not found errors so it will not say metrics are getting pushed when the database goes down. Also updated test

[srebhan]

Hey @MyaLongmire, thanks for the PR! There are some debug messages left in the code and I'd like to ask you to add a comment on the "strange" error handling. I guess it has to be like this for telegraf to trigger a retry, however it might be nicer to retry directly after successfully creating the database.
Assume your metric buffer is full and that's why a flush/write was triggered in the first place. In this first try you create the database and return an error (as you currently do). Now your metric buffer is still full. If new metrics arrive between this and the next retry you will loose metrics due to buffer fullness...

[telegraf-tiger]

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

• windows_amd64.zip

• windows_i386.zip

• darwin_amd64.tar.gz

• linux_mips.tar.gz

• mips.deb

• ppc64le.rpm

• linux_ppc64le.tar.gz

• ppc64el.deb

• aarch64.rpm

• linux_arm64.tar.gz

• arm64.deb

• static_linux_amd64.tar.gz

• linux_mipsel.tar.gz

• mipsel.deb

• armv6hl.rpm

• freebsd_armv7.tar.gz

• linux_armhf.tar.gz

• armhf.deb

• x86_64.rpm

• freebsd_amd64.tar.gz

• linux_amd64.tar.gz

• amd64.deb

• i386.rpm

• freebsd_i386.tar.gz

• linux_i386.tar.gz

• i386.deb

• s390x.rpm

• linux_s390x.tar.gz

• s390x.deb

• armel.rpm

• linux_armel.tar.gz

• armel.deb

[srebhan]

@MyaLongmire can you comment on the potential buffer fullness issue I mentioned. It's ok for me if we do not tackle it in this PR, but if this is not just hypothetical, we need to take care of it.

[MyaLongmire]

@MyaLongmire can you comment on the potential buffer fullness issue I mentioned. It's ok for me if we do not tackle it in this PR, but if this is not just hypothetical, we need to take care of it.

@srebhan in my understanding if the buffer is full when new metrics come in the oldest are dropped, there nothing much to be done about that.

[srebhan]

@MyaLongmire imagine the case where the database does not yet exist on the InfluxDB side. Further assume the buffer runs full (e.g. due to a delayed start of the database service or whatever). In the ideal case, the output plugin would create the database and flush the metric to this newly created database. That's what I would expect as user.
However, from my understanding what the plugin does is, it creates the database in a first flush cycle without flushing the data to the newly created database and waits for a second flush cycle to write the data, The "data" in the second cycle comprises the "old" one from the first cycle and the new data arrived in the meantime. If now the buffer is already full (or almost full) in the first cycle, the data arriving between the first and second cycle might lead to data-dropping.

My suggestion is to change the current state

plugin.Write() // --> will create the database and return error without flushing the buffer
... gathering of data...
plugin.Write() // --> will flush the buffer containing data of the first "Write()" call and new data

to

plugin.Write() // --> will create the database and try flushing the buffer
... gathering of data...
plugin.Write() // --> will flush the buffer containing ONLY new data

[ssoroka]

I get what you're saying @srebhan , but I don't think it's very likely that the buffer is full and the database has not yet been created but will be created successfully on this call. If that is the case, it's likely the buffer was already full and dropping messages.
I'm a bit hesitant to repeat the write immediately as the code is not very straight-forward. the choice to return an error to trigger retry is one chosen to keep the code simple in an area we've already had a lot of issues with code complexity.

[srebhan]

@ssoroka understood. It's fine with me.

[srebhan]

Looks good to me.