Fluent-bit 3.0.4 throwing repeated warnings ( [ warn] [record accessor] translation failed, root key=hec_token)

[Nithinrao9696]

Bug Report

Describe the bug
After upgrading fluent-bit from 3.0.3 to 3.0.4, we have noticed fluent-bit pods started throwing repeated warnings in the logs

To Reproduce
upgrade to 3.0.4 version of fluent-bit

• Rubular link if applicable:
• Example log message if applicable:

[2024/05/22 13:12:09] [ warn] [record accessor] translation failed, root key=hec_token
[2024/05/22 13:12:09] [ warn] [record accessor] translation failed, root key=hec_token
[2024/05/22 13:12:09] [ warn] [record accessor] translation failed, root key=hec_token
[2024/05/22 13:12:09] [ warn] [record accessor] translation failed, root key=hec_token
[2024/05/22 13:12:09] [ warn] [record accessor] translation failed, root key=hec_token

[digarok]

I think there's a problem with this PR.
https://github.com/fluent/fluent-bit/pull/8793/files

We're getting spammed with the same message as the user above, millions of times per minute:
[2024/05/22 16:08:04] [ warn] [record accessor] translation failed, root key=hec_token

[olfway]

We have same error

[yuliyan-valchev-ft]

Same error
[2024/05/22 15:06:30] [ warn] [record accessor] translation failed, root key=hec_token
[2024/05/22 15:06:30] [debug] [output:splunk:splunk.0] Could not find hec_token in metadata

It started appearing right after we started using the latest helm chart 0.46.7 with app version 3.0.4

[edsiper]

we are looking into this.

[edsiper]

Fix will be there shortly: #8864

[nehjain17]

We are facing security issue and need to upgrade to 3.0.4 asap. Getting the same error.

[nehjain17]

With 3.0.5 getting below error.
[2024/05/24 05:47:13] [ warn] [output:splunk:splunk.16] http_status=401:
{"text":"Token is required","code":2}

[edsiper]

With 3.0.5 getting below error. [2024/05/24 05:47:13] [ warn] [output:splunk:splunk.16] http_status=401: {"text":"Token is required","code":2}

@nehjain17 What is your config?

[nehjain17]

Output Config:

[OUTPUT]
Name splunk
Match index1*
event_index index1
host splunk_host_name
Port 443
splunk_token ${TOKEN}
tls On
tls.verify Off
Retry_Limit 5
net.connect_timeout 30
net.keepalive on

[nehjain17]

output plugin [OUTPUT] Name splunk Match index1 event_index index1 host splunk_host_name Port 443 splunk_token ${TOKEN} tls On tls.verify Off Retry_Limit 5 net.connect_timeout 30 net.keepalive on

…

On Fri, May 24, 2024 at 5:07 AM Eduardo Silva ***@***.***> wrote: With 3.0.5 getting below error. [2024/05/24 05:47:13] [ warn] [output:splunk:splunk.16] http_status=401: {"text":"Token is required","code":2} @nehjain17 <https://github.com/nehjain17> What is your config? — Reply to this email directly, view it on GitHub <#8859 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AETMRSSU7SK63B6MLRWJ3CTZD4URTAVCNFSM6AAAAABIDULTUGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRZGM3TCMBXGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>