TCP Input, Invalid JSON

[albertoCrego]

Bug Report

Describe the bug
Many errors are related to the invalid JSON message, skipping in the TCP Input but no errors when flowing to fluentd.

To Reproduce
I have a producer of TCP Messages from an external place (APIGEE). Every request is forwarded to Fluentbit INPUT TCP, parsed as JSON and sent directly to OpenSearch.

1. I have one INPUT and one OUTPUT to route the data.
2. I tested to enable Tap functionality over Inputs
3. I tested to enable LogLevel Trace trying to get more insights.
4. I tested disabling json parser and enabling none but the logs disappeared, I suppose because the message contains the JSON contentType and is JSON formated.
5. Another interesting test that I did was send from the producer a dummy JSON without any dynamic data or long string, only one key-value dummy, but the problem persisted.
6. My last attempt was to deploy a fluentd with the same configuration and none/json parser and duplicate the data to try to get the error but exactly in the same time range, I received about twice documents in fluentd and no warning about json parsing.

Evidence, around 50% of the documents are dropped.

[2024/09/03 12:55:55] [ warn] [input:tcp:tcp-input-nonprod] invalid JSON message, skipping
[2024/09/03 12:55:55] [ warn] [input:tcp:tcp-input-nonprod] invalid JSON message, skipping
[2024/09/03 12:55:56] [ warn] [input:tcp:tcp-input-nonprod] invalid JSON message, skipping
[0] trace: [[1725368155.236477610, {}], {"type"=>1, "trace_id"=>"67", "plugin_instance"=>"tcp.5", "plugin_alias"=>"tcp-input-nonprod", "records"=>"start_time", 1725368155=>"end_time", ---> log OK 
[2] trace: [[1725368155.481906301, {}], {"type"=>1, "trace_id"=>"68", "plugin_instance"=>"tcp.5", "plugin_alias"=>"tcp-input-nonprod", "records"=>"start_time", 1725368155=>"end_time", ---> log OK 

Expected behavior
All logs are parsed properly.

Your Environment

• Version used: Tested with 3.0.4 and 3.1.7
• Configuration:

    [INPUT]
        Name        tcp
        Alias       tcp-input-nonprod
        Listen      0.0.0.0
        Port        2021
        Format     json
        Tag         apigee-nonprod.*
        Source_Address_Key Off
        Buffer_Size 4096k
        Chunk_Size  4096k
        Threaded    On

    [OUTPUT]
        Name          opensearch
        Alias           opensearch_apigee_nonprod
        Match         apigee-nonprod.*
        Host            ...
        Port            443
        Type            doc
        Logstash_Format On
        Logstash_Prefix logs-apigee-nonprod
        Include_Tag_Key Off
        Replace_Dots    On
        AWS_Auth        On
        AWS_Region      eu-central-1
        tls             On
        Suppress_Type_Name    On
        Buffer_Size     4MB

• Environment name and version Kubernetes v1.30
• Server type and version: Opensearch 2.11
• Operating System and version: NA
• Filters and plugins: TCP and Opensearch

Additional context

• Any hint to solve the issue?
• In case that is not possible, can get the parser an output about what is the issue with the JSON, Fluentd do this:

logging-playground-65bfdb67c9-gbxgc fluentd 2024-09-03 13:18:58 +0000 [warn]: #0 pattern not matched message="{\"name\": \"wrong-json\",}\u0000"