parse_json processor doesn't work with TCP receiver

[ViliusS]

Describe the bug
Apparently parse_json processor doesn't work when used with tcp receiver. I need to parse timestamp for a log. If I use the same parse_json configuration with files receiver with exactly the same log source everything works as expected.

To Reproduce
Steps to reproduce the behavior:

1. Start a GCE VM with image 'rocky-linux-9-v20240515'
2. Install Ops Agent version 'google-cloud-ops-agent-2.55.0-1.el9.x86_64.rpm'
3. Customize the agent with the following configuration' and restart the agent

logging:
  receivers:
    test_log:
      type: tcp
      format: json
  processors:
    parse_json:
      type: parse_json
      time_key: timestamp
      time_format: "%Y-%m-%dT%H:%M:%S.%LZ"
  service:
    pipelines:
      default_pipeline:
        processors: [parse_json]
        receivers: [test_log]

4. Configure any of the Log4J sources with JSON logging. You can use GcpLayout template provided by the Log4J project.

rootLogger.level = INFO
rootLogger.appenderRef.json_log.ref = json_log
appender.json_log.type = Socket
appender.json_log.name = json_log
appender.json_log.host = localhost
appender.json_log.port = 5170
appender.json_log.layout.type = JsonTemplateLayout
appender.json_log.layout.eventTemplateUri = classpath:GcpLayout.json

5. Observe that timestamp field is not parsed in Google Logs:
   

6. Now change Google Ops Agent configuration to use files receiver:

logging:
  receivers:
    test_log:
      type: files
      include_paths:
      - /var/log/json/test.log
  processors:
    parse_json:
      type: parse_json
      time_key: timestamp
      time_format: "%Y-%m-%dT%H:%M:%S.%LZ"
  service:
    pipelines:
      default_pipeline:
        processors: [parse_json]
        receivers: [test_log]

7. Set the source to log to the file, for example:

rootLogger.level = INFO
rootLogger.appenderRef.json_log.ref = json_log
appender.json_log.type = RollingFile
appender.json_log.name = json_log
appender.json_log.fileName = /var/log/json/test.log
appender.json_log.filePattern = /var/log/json/test.log.%i
appender.json_log.strategy.type = DefaultRolloverStrategy
appender.json_log.strategy.max = 1
appender.json_log.layout.type = JsonTemplateLayout
appender.json_log.layout.eventTemplateUri = classpath:GcpLayout.json
appender.json_log.policies.type = Policies
appender.json_log.policies.cron.type = CronTriggeringPolicy
appender.json_log.policies.cron.schedule = 0 0 0 * * ?
appender.json_log.policies.cron.evaluateOnStartup = true

8. You will now see that timestamp is parsed correctly
   

Expected behavior
parse_json processor should be able to parse timestamps when logs is received via tcp receiver.

Environment (please complete the following information):

• VM distro / OS: Rocky Linux 9.5
• Ops Agent version 2.55

[igorpeshansky]

parse_json looks at the message field of the incoming log and interprets it as a serialized JSON string. Because the tcp receiver uses format: json, the incoming log is already structured, so parse_json gets to look at the message that isn't in JSON format, and that doesn't work. So this is basically a feature request to either allow the tcp receiver to keep JSON strings unparsed (to allow parse_json to do that), or to find some other way to extract the timestamp from JSON (e.g., by adding JSON parsing parameters to the tcp receiver, or by adding a timestamp parsing operation to the modify_fields processor).

[ViliusS]

@igorpeshansky I then don't get the point of tcp receiver. Initially, I thought that it would parse received JSON and map all fields automatically to the Google Cloud Logging format. After playing for a while I found that I still need to move severity field and timestamp manually. That's why I tried to use parse_json.

Does tcp receiver do any auto magic at all, or does it only ensure that received log is valid JSON and that's it? Or maybe there is a way to format source log JSON format that it would match what is required by Google Logging in the first place?

[braydonk]

I thought that it would parse received JSON and map all fields automatically to the Google Cloud Logging format

It does do some of that on output, but it does that via recognizing special fields in the structured log. You can find a full list of those along with explanations here: https://github.com/fluent/fluent-bit-docs/blob/master/pipeline/outputs/stackdriver_special_fields.md#log-entry-fields

Your structured log would need to have these fields to be recognized though. Nothing special is done on ingestion with regards to putting things in Cloud Logging formats.

Does tcp receiver do any auto magic at all, or does it only ensure that received log is valid JSON and that's it?

The tcp receiver parses the JSON it receives.

Say we start with the following log:
{"severity":"INFO","timestamp":"<time>",...}

Coming out of the files receiver, the log is shaped like this:

message: '{"severity":"INFO", "timestamp":"<time>", ...}'

So parse_json will read message and parse the JSON into:

severity: 'INFO'
timestamp: '<time>'

If you send it into tcp, it will actually come out like the parse_json result, i.e. directly to:

severity: 'INFO'
timestamp: '<time>'

Or maybe there is a way to format source log JSON format that it would match what is required by Google Logging in the first place?

Most of this will work fine, but this does expose the limitation around timestamp parsing. With the tcp receiver as it is today, there is no way to specify a timestamp format to parse timestamp from the log. I think we should come up with a solution for that. The easiest way would be to support the none format for this receiver, which just means logs are not parsed and are received over tcp delimited by LF characters. I will have to check internally if we had a reason we didn't support that in the first place.

[ViliusS]

I thought that it would parse received JSON and map all fields automatically to the Google Cloud Logging format

It does do some of that on output, but it does that via recognizing special fields in the structured log. You can find a full list of those along with explanations here: https://github.com/fluent/fluent-bit-docs/blob/master/pipeline/outputs/stackdriver_special_fields.md#log-entry-fields

Your structured log would need to have these fields to be recognized though. Nothing special is done on ingestion with regards to putting things in Cloud Logging formats.

Maybe I'm missing something but my tested structured log looks like this before TCP input:

{"timestamp":"2025-04-01T07:23:41.724Z","severity":"ERROR","message":"AMQ224096: Error setting up connection from /178.128.147.230:26881 to /10.90.160.4:5671; protocol CORE not found in map: [AMQP]","logging.googleapis.com/sourceLocation":{"function":"org.apache.activemq.artemis.core.server.ActiveMQServerLogger_impl.failedToFindProtocolManager"},"logging.googleapis.com/insertId":"59","logging.googleapis.com/trace_sampled":true,"_exception":{"stackTrace":""},"_thread":"Thread-1 (activemq-netty-threads)","_logger":"org.apache.activemq.artemis.core.server"}

tcp receiver should parse timestamp and severity fields correctly in such case, shouldn't it?

Does tcp receiver do any auto magic at all, or does it only ensure that received log is valid JSON and that's it?

The tcp receiver parses the JSON it receives.

Say we start with the following log: {"severity":"INFO","timestamp":"<time>",...}

Coming out of the files receiver, the log is shaped like this:

message: '{"severity":"INFO", "timestamp":"<time>", ...}'

So parse_json will read message and parse the JSON into:

severity: 'INFO'
timestamp: '<time>'

If you send it into tcp, it will actually come out like the parse_json result, i.e. directly to:

severity: 'INFO'
timestamp: '<time>'

I guess this makes sense. In theory I should be able to feed a log like this then?

{"timestamp_json":{"timestamp":"2025-04-01T07:23:41.724Z"},"severity":"ERROR","message":"my message"}

And then just do:

logging:
  receivers:
    test_log:
      type: tcp
      format: json
  processors:
    parse_json:
      type: parse_json
      field: timestamp_json
      time_key: timestamp
      time_format: "%Y-%m-%dT%H:%M:%S.%LZ"

Or maybe there is a way to format source log JSON format that it would match what is required by Google Logging in the first place?

Most of this will work fine, but this does expose the limitation around timestamp parsing. With the tcp receiver as it is today, there is no way to specify a timestamp format to parse timestamp from the log. I think we should come up with a solution for that. The easiest way would be to support the none format for this receiver, which just means logs are not parsed and are received over tcp delimited by LF characters. I will have to check internally if we had a reason we didn't support that in the first place.

I think the better approach would be to actually do more automatic mapping if JSON format matches special fields. It's still unclear to me why this is not working currently, but that would be more understandable behaviour. Having format none implies that one needs to know Ops Agent internals to be able to correctly choose the format (or very good documentation, which currently Ops Agent lacks in advanced areas :)).

[braydonk]

tcp receiver should parse timestamp and severity fields correctly in such case, shouldn't it?

If the severity field was named logging.googleapis.com/severity then the severity would be detected and handled correctly on output. The timestamp wouldn't be parsed, it would end up in the structured log as timestamp: "2025-04-01T07:23:41.724Z" and would not affect the timestamp of the actual log.

I guess this makes sense. In theory I should be able to feed a log like this then?

This would not work unfortunately. Fluent Bit's JSON parsing is recursive, meaning that the timestamp_json field would not end up being a JSON string that could be parsed at the next step, rather an object in Fluent Bit's internal representation.

I think the better approach would be to actually do more automatic mapping if JSON format matches special fields.

The only one we'd want to recognize would be timestamp, as any other field should be handled in output. We don't have any control over what parsing the tcp plugin actually does. I'm not sure if Fluent Bit has a way to actually do this, but if they had a way to just parse the timestamp out of an already structured log then we could add time_key and time_format configuration to the Ops Agent tcp receiver config which if present would generate the additional config to parse the timestamp. We'd need to investigate if this is possible though.

Having format none implies that one needs to know Ops Agent internals to be able to correctly choose the format (or very good documentation, which currently Ops Agent lacks in advanced areas :)).

Yes our documentation is lacking here. I had to look at our source code to figure this stuff out. I've opened an internal tracker for someone on our team to look into how we can resolve this. Part of that work will be improving documentation for tcp receiver to explain the expectations for input, regardless of which solution we end up on. Thanks for opening the issue with reproduction details!

[ViliusS]

tcp receiver should parse timestamp and severity fields correctly in such case, shouldn't it?

If the severity field was named logging.googleapis.com/severity then the severity would be detected and handled correctly on output. The timestamp wouldn't be parsed, it would end up in the structured log as timestamp: "2025-04-01T07:23:41.724Z" and would not affect the timestamp of the actual log.

Ah, understood regarding severity field. It's strange a bit though that label prefix is needed, because it works pretty well if such JSON log is fed into Google Cloud Logging directly via Java SDK. I guess that's another thing to tune. I will try to reach Log4J project to find out if we could have FluentBitLayout.json template in addition to the current https://github.com/apache/logging-log4j2/blob/3da98f7c9de9bf4abb17e10dad678e05ab658a3a/log4j-layout-template-json/src/main/resources/GcpLayout.json

I guess this makes sense. In theory I should be able to feed a log like this then?

This would not work unfortunately. Fluent Bit's JSON parsing is recursive, meaning that the timestamp_json field would not end up being a JSON string that could be parsed at the next step, rather an object in Fluent Bit's internal representation.

I think the better approach would be to actually do more automatic mapping if JSON format matches special fields.

The only one we'd want to recognize would be timestamp, as any other field should be handled in output. We don't have any control over what parsing the tcp plugin actually does. I'm not sure if Fluent Bit has a way to actually do this, but if they had a way to just parse the timestamp out of an already structured log then we could add time_key and time_format configuration to the Ops Agent tcp receiver config which if present would generate the additional config to parse the timestamp. We'd need to investigate if this is possible though.

One more idea, according to these docs https://cloud.google.com/logging/docs/agent/logging/configuration#timestamp-processing I should be able to use {"time":"epochtime"} in the structured logs. Would it work with tcp receiver, or again these docs are for file based logs only?

Having format none implies that one needs to know Ops Agent internals to be able to correctly choose the format (or very good documentation, which currently Ops Agent lacks in advanced areas :)).

Yes our documentation is lacking here. I had to look at our source code to figure this stuff out. I've opened an internal tracker for someone on our team to look into how we can resolve this. Part of that work will be improving documentation for tcp receiver to explain the expectations for input, regardless of which solution we end up on. Thanks for opening the issue with reproduction details!

At very least special fields here https://cloud.google.com/logging/docs/agent/logging/configuration#special-fields and here https://github.com/fluent/fluent-bit-docs/blob/master/pipeline/outputs/stackdriver_special_fields.md#log-entry-fields do not match. For example severity, logName JSON description, possibly others.

Thank you too for digging into this. Usually I don't expect much when filling GitHub issues regarding GCP but Ops Agent support exceeded my expectations!

[igorpeshansky]

I will try to reach Log4J project to find out if we could have FluentBitLayout.json template…

Note that this is not fluent-bit specific, but Google Cloud specific. The special fields are recognized by the Ops agent and the configuration it generates for fluent-bit's out_stackdriver plugin.

At very least special fields here and here do not match.

The first link is for the legacy fluentd-based logging agent. The second is for the default configuration of out_stackdriver. The Ops agent has some differences with the latter, for various historical reasons, which are handled by the generated fluent-bit config. We've made a conscious decision to introduce the discrepancies with the former, for a cleaner interface. The Ops agent special fields are documented here (note the ops-agent URL).

[ViliusS]

OK, now it makes sense. According to https://cloud.google.com/logging/docs/agent/ops-agent/configuration#special-fields timestamp requires to be in seconds + nanoseconds format. First option is out of the window as FluentBit JSON parser is recursive. However second option gave me an idea.

I have modified GCP log format provided by Log4J with the following template (let's call it OpsAgentLayout.json):

{
  "timestampSeconds": {
    "$resolver": "timestamp",
    "epoch": {
      "unit": "secs",
      "rounded": true
    }
  },
  "timestampNanos": {
    "$resolver": "timestamp",
    "epoch": {
      "unit": "secs.nanos"
    }
  },
  "severity": {
    "$resolver": "pattern",
    "pattern": "%level{WARN=WARNING, TRACE=DEBUG, FATAL=EMERGENCY}",
    "stackTraceEnabled": false
  },
  "message": {
    "$resolver": "pattern",
    "pattern": "%m",
    "stackTraceEnabled": true
  },
  "logging.googleapis.com/labels": {
    "$resolver": "mdc",
    "stringified": true
  },
  "logging.googleapis.com/sourceLocation": {
    "file": {
      "$resolver": "source",
      "field": "fileName"
    },
    "line": {
      "$resolver": "source",
      "field": "lineNumber"
    },
    "function": {
      "$resolver": "pattern",
      "pattern": "%replace{%C.%M}{^\\?\\.$}{}",
      "stackTraceEnabled": false
    }
  },
  "logging.googleapis.com/trace": {
    "$resolver": "mdc",
    "key": "trace_id"
  },
  "logging.googleapis.com/spanId": {
    "$resolver": "mdc",
    "key": "span_id"
  },
  "logging.googleapis.com/trace_sampled": true,
  "exception": {
    "$resolver": "exception",
    "field": "stackTrace",
    "stackTrace": {
      "stringified": true
    }
  },
  "thread": {
    "$resolver": "thread",
    "field": "name"
  },
  "logger": {
    "$resolver": "logger",
    "field": "name"
  }
}

Loaded that template into my Java app source with:

rootLogger.level = INFO
rootLogger.appenderRef.json_log.ref = json_log
appender.json_log.type = Socket
appender.json_log.name = json_log
appender.json_log.host = localhost
appender.json_log.port = 5170
appender.json_log.layout.type = JsonTemplateLayout
appender.json_log.layout.eventTemplateUri = file:///path/to/template/OpsAgentLayout.json

And it works with a very simple Ops Agent configuration now:

logging:
  receivers:
    test_log:
      type: tcp
      format: json
  service:
    pipelines:
      default_pipeline:
        receivers: [test_log]

Can be used as a workaround instead of files receiver until tcp receiver will have more features.

P.S. Do know if such JSON format would be compatible with Google Cloud Logging when used via Java Client libraries directly? I definitely can create a patch to Log4J just don't know if that won't break other Stackdriver logging implementations. The biggest question is regarding severity field which at least on Spring GCP libraries still uses older field name but works without issues in Spring projects deployed on Google Cloud.

[braydonk]

I don't know Log4j so I can't say specifically, but I'm assuming secs.nanos is not just the number of seconds in nanoseconds format, and the remainder of nanoseconds? Because the way it works is the time is calculated as timestampSeconds + timestampNanos. I'm assuming if you ended up seeing logs in Cloud Logging that secs.nanos works as I'd expect.

Thanks for posting the layout, I bet this could be useful to other users as well. If there was somewhere to contribute the OpsAgentLayout.json that would be really awesome, cause I'm sure other people could make use of it!

Do know if such JSON format would be compatible with Google Cloud Logging when used via Java Client libraries directly?

I'm not a Java guy and our team isn't associated with the Client Libraries, but I would wager that this format would not work if used with the Client Library. The logging.googleapis.com/* fields are completely tailored to the out_stackdriver plugin in Fluent Bit; if you sent a log with a field like logging.googleapis.com/severity to the Client Library it's incredibly unlikely that it would know and do anything special with that.

The Client Libraries are setup in kind of a similar way to what the Fluent Bit output plugin is. Given a raw structured log, there is a set of field names that are declared as recognized fields that will be mapped to the LogEntry proto fields before being sent as a Cloud Logging request. But for the Client Libraries, they map severity -> severity. Originally for Fluentd, and ported forward for Fluent Bit, since these applications can be used to send to lots of sources and severity is a very common field name that could clash with other plugins' expectations, we instead declare a logging.googleapis.com prefix for the fields we want to recognize and parse into LogEntry fields. Thus instead of severity -> severity we do logging.googleapis.com/severity -> severity.

Long winded explanation, but long story short that is the reason there is a difference between what Client Libraries do and what the Ops Agent expects. So I don't think your layout will work directly with Client Libraries. But, and I don't really know where this goes exactly, but if that layout could be provided somewhere upstream then other people may be able to make use of it when sending Log4j logs to be read by the Ops Agent (either through TCP or files I imagine would be fine).

[ViliusS]

Thank you. I will try to manage

I don't know Log4j so I can't say specifically, but I'm assuming secs.nanos is not just the number of seconds in nanoseconds format, and the remainder of nanoseconds? Because the way it works is the time is calculated as timestampSeconds + timestampNanos. I'm assuming if you ended up seeing logs in Cloud Logging that secs.nanos works as I'd expect.

Thanks for posting the layout, I bet this could be useful to other users as well. If there was somewhere to contribute the OpsAgentLayout.json that would be really awesome, cause I'm sure other people could make use of it!

secs.nanos are just nanoseconds so the math should work fine.

Do know if such JSON format would be compatible with Google Cloud Logging when used via Java Client libraries directly?

I'm not a Java guy and our team isn't associated with the Client Libraries, but I would wager that this format would not work if used with the Client Library. The logging.googleapis.com/* fields are completely tailored to the out_stackdriver plugin in Fluent Bit; if you sent a log with a field like logging.googleapis.com/severity to the Client Library it's incredibly unlikely that it would know and do anything special with that.

The Client Libraries are setup in kind of a similar way to what the Fluent Bit output plugin is. Given a raw structured log, there is a set of field names that are declared as recognized fields that will be mapped to the LogEntry proto fields before being sent as a Cloud Logging request. But for the Client Libraries, they map severity -> severity. Originally for Fluentd, and ported forward for Fluent Bit, since these applications can be used to send to lots of sources and severity is a very common field name that could clash with other plugins' expectations, we instead declare a logging.googleapis.com prefix for the fields we want to recognize and parse into LogEntry fields. Thus instead of severity -> severity we do logging.googleapis.com/severity -> severity.

Long winded explanation, but long story short that is the reason there is a difference between what Client Libraries do and what the Ops Agent expects. So I don't think your layout will work directly with Client Libraries. But, and I don't really know where this goes exactly, but if that layout could be provided somewhere upstream then other people may be able to make use of it when sending Log4j logs to be read by the Ops Agent (either through TCP or files I imagine would be fine).

Thank you for explanation. As far as I understood I will revert to use severity (because Ops Agent has a workaround with move_field) and will try to work with Log4J to have this template implemented in official GcpLayout JSON template.

[ppkarwasz]

Thanks for posting the layout, I bet this could be useful to other users as well. If there was somewhere to contribute the OpsAgentLayout.json that would be really awesome, cause I'm sure other people could make use of it!

@braydonk, I think that the best location for the template file are the Debian packages produced by this repo, for several reasons:

• the log format used by Ops Agent is not a standard and is only useful to Ops Agent users.
• if we keep the format in Log4j, every change to the format needs first to propagate to Java applications, before it can be used by Ops Agent users.
• if you keep the format here, you can easily tweak it to provide the best performance in a Google Cloud environment.

Therefore I would propose to:

1. Move Adjust GcpLayout JSON to latest format apache/logging-log4j2#3586 here. From Log4j perspective the format proposed by @ViliusS is already optimized for performance.
2. Modify the Monitoring third-party applications documentation to recommend the usage of JSON Template Layout and the template provided by Ops Agent for all Java applications that use Log4j 2 Core (Apache ActiveMQ, Kafka 4.0, Solr, ElasticSearch and probably some others).
   The current documentation suggests to use the default log format for those applications (which produces human-readable "garbage") and then configure Ops Agent to try parse that "garbage". Such a configuration is susceptible to all sort of log injection attacks.
3. Deprecate the Ops Agent receivers for those Java applications. If those apps output logs in standardized JSON format, a single receiver can handle all of them.
4. In Apache Log4j, we can deprecate the GcpLayout.json template and point GCP users to the Ops Agent documentation.

What do you think?

PS: If OpsAgent were to also distribute (or generate) a log4j2.xml configuration, sending all logs to Google Cloud Logging might be as simple as setting the correct value to the LOG4J_CONFIGURATION_FILE environment variable.

[aabmass]

4. In Apache Log4j, we can deprecate the GcpLayout.json template and point GCP users to the Ops Agent documentation.

I'm a bit out of the loop here but why would we want to do this?

I would just call attention that Ops Agent is not the only logging agent in GCP. GKE, AppEngine, Cloud Run, etc. also support parsing JSON logs from stdout/stderr. I want to make sure any changes to GcpLayout.json like apache/logging-log4j2#3586 also continue to work on these environments.

Unfortunately the docs are a bit spread out and unspecific, so I'm happy to help validate any changes.

[ViliusS]

@aabmass: It's a little bit late as Log4J PR was already merged, but it would be great if you can validate how it works on AppEngine, Cloud Run and other services. I tried them only with Ops Agent + Compute Engine and with direct stdout in GKE.

What do you need specifically to validate those changes on your end? Instruction how to configure Log4J for JSON parsing with latest GCP template are in #1920 (comment) . They are for TCP receiver, but you can easily switch the log to stdout/stderr. I can also provide a generated random short log in JSON format, just not sure if it will be useful for you.