ResourceLogs emitted by experimental-otlp/stdout lack line breaks being incompatible with the OTelCol otlpjson connector

[cyrille-leclerc]

Describe the bug

ResourceLogs emitted by OTEL_LOGS_EXPORTER="experimental-otlp/stdout" lack line breaks being incompatible with the OTel Collector filelog receiver and otlpjson connector.

Example of 2 consecutive log lines that are outputted in stdout without any character between them.

{"resourceLogs":[{"resource":{"attributes":[{"key":"deployment.environment","value":{"stringValue":"staging"}},{"key":"host.arch","value":{"stringValue":"aarch64"}},{"key":"host.name","value":{"stringValue":"cyrille-le-clerc-macbook.local"}},{"key":"os.description","value":{"stringValue":"Mac OS X 15.1.24.0.1"}},{"key":"os.type","value":{"stringValue":"darwin"}},{"key":"process.command_args","value":{"arrayValue":{"values":[{"stringValue":"/opt/homebrew/Cellar/openjdk@21/21.0.5/libexec/openjdk.jdk/Contents/Home/bin/java"},{"stringValue":"-javaagent:./../.otel/opentelemetry-javaagent-2.9.0.jar"},{"stringValue":"-Dotel.instrumentation.logback-appender.experimental-log-attributes=true"},{"stringValue":"-Dotel.instrumentation.logback-mdc.add-baggage=true"},{"stringValue":"-Dotel.instrumentation.micrometer.base-time-unit=s"},{"stringValue":"-Dotel.semconv-stability.opt-in=http/dup"},{"stringValue":"-DshippingServiceUrl=http://shipping.local:8088/"},{"stringValue":"-jar"},{"stringValue":"target/checkout-1.1-SNAPSHOT.jar"}]}}},{"key":"process.executable.path","value":{"stringValue":"/opt/homebrew/Cellar/openjdk@21/21.0.5/libexec/openjdk.jdk/Contents/Home/bin/java"}},{"key":"process.pid","value":{"intValue":"92078"}},{"key":"process.runtime.description","value":{"stringValue":"Homebrew OpenJDK 64-Bit Server VM 21.0.5"}},{"key":"process.runtime.name","value":{"stringValue":"OpenJDK Runtime Environment"}},{"key":"process.runtime.version","value":{"stringValue":"21.0.5"}},{"key":"service.instance.id","value":{"stringValue":"01c0a650-fea5-48a8-b7a3-ffe6c256f18e"}},{"key":"service.name","value":{"stringValue":"checkout"}},{"key":"service.namespace","value":{"stringValue":"shop"}},{"key":"service.version","value":{"stringValue":"1.1"}},{"key":"telemetry.distro.name","value":{"stringValue":"opentelemetry-java-instrumentation"}},{"key":"telemetry.distro.version","value":{"stringValue":"2.9.0"}},{"key":"telemetry.sdk.language","value":{"stringValue":"java"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"1.43.0"}}]},"scopeLogs":[{"scope":{"name":"com.mycompany.checkout.CheckoutServiceServer","attributes":[]},"logRecords":[{"timeUnixNano":"1730209613443367000","observedTimeUnixNano":"1730209613446828000","severityNumber":9,"severityText":"INFO","body":{"stringValue":"GRPC server started, listening on 50051"},"attributes":[{"key":"thread.id","value":{"intValue":"1"}},{"key":"thread.name","value":{"stringValue":"main"}}]}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.24.0"}]}{"resourceLogs":[{"resource":{"attributes":[{"key":"deployment.environment","value":{"stringValue":"staging"}},{"key":"host.arch","value":{"stringValue":"aarch64"}},{"key":"host.name","value":{"stringValue":"cyrille-le-clerc-macbook.local"}},{"key":"os.description","value":{"stringValue":"Mac OS X 15.0.1"}},{"key":"os.type","value":{"stringValue":"darwin"}},{"key":"process.command_args","value":{"arrayValue":{"values":[{"stringValue":"/opt/homebrew/Cellar/openjdk@21/21.0.5/libexec/openjdk.jdk/Contents/Home/bin/java"},{"stringValue":"-javaagent:./../.otel/opentelemetry-javaagent-2.9.0.jar"},{"stringValue":"-Dotel.instrumentation.logback-appender.experimental-log-attributes=true"},{"stringValue":"-Dotel.instrumentation.logback-mdc.add-baggage=true"},{"stringValue":"-Dotel.instrumentation.micrometer.base-time-unit=s"},{"stringValue":"-Dotel.semconv-stability.opt-in=http/dup"},{"stringValue":"-DshippingServiceUrl=http://shipping.local:8088/"},{"stringValue":"-jar"},{"stringValue":"target/checkout-1.1-SNAPSHOT.jar"}]}}},{"key":"process.executable.path","value":{"stringValue":"/opt/homebrew/Cellar/openjdk@21/21.0.5/libexec/openjdk.jdk/Contents/Home/bin/java"}},{"key":"process.pid","value":{"intValue":"92078"}},{"key":"process.runtime.description","value":{"stringValue":"Homebrew OpenJDK 64-Bit Server VM 21.0.5"}},{"key":"process.runtime.name","value":{"stringValue":"OpenJDK Runtime Environment"}},{"key":"process.runtime.version","value":{"stringValue":"21.0.5"}},{"key":"service.instance.id","value":{"stringValue":"01c0a650-fea5-48a8-b7a3-ffe6c256f18e"}},{"key":"service.name","value":{"stringValue":"checkout"}},{"key":"service.namespace","value":{"stringValue":"shop"}},{"key":"service.version","value":{"stringValue":"1.1"}},{"key":"telemetry.distro.name","value":{"stringValue":"opentelemetry-java-instrumentation"}},{"key":"telemetry.distro.version","value":{"stringValue":"2.9.0"}},{"key":"telemetry.sdk.language","value":{"stringValue":"java"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"1.43.0"}}]},"scopeLogs":[{"scope":{"name":"com.mycompany.checkout.CheckoutServiceServer$CheckoutServiceImpl","attributes":[]},"logRecords":[{"timeUnixNano":"1730209630540189000","observedTimeUnixNano":"1730209630540384000","severityNumber":9,"severityText":"INFO","body":{"stringValue":"Order successfully placed customerId=customer-94 shipping=ok durationInMillis=31"},"attributes":[{"key":"thread.id","value":{"intValue":"39"}},{"key":"thread.name","value":{"stringValue":"grpc-default-executor-0"}}],"flags":1,"traceId":"468a15931e800d1644dee4e0e62b1fa2","spanId":"8465d2938245281d"}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.24.0"}]}

Example failing OTel Collector pipeline, it expects line breaks between log records

receivers:
  filelog/otlp-json-logs:
    # debug: restart at the beginning of the file
    start_at: beginning
    include: [/path/to/otlp-json-logs.log]

processors:

connectors:
  otlpjson:

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    logs/raw:
      receivers: [filelog/otlp-json-logs]
      processors: []
      exporters: [otlpjson]
    logs/normalized:
      receivers: [otlpjson]
      processors: []
      exporters: [debug]
  telemetry:
    metrics:
      level: none

Steps to reproduce

• Use a Java app that produces log lines through Logback
• Instrument it with otel java instrumentation agent v2.9.0
• Export logs through the new stdout/otlp-json instead of OTLP setting OTEL_LOGS_EXPORTER="experimental-otlp/stdout"
• Launch the app and produce a few log lines,
• Verify that they are outputted in stdout as JSON documents {"resourceLogs":[{"resource":...
• Verify that the OTLP JSON log records ({"resourceLogs":[{"resource":...) lack line break delimiters

What did you expect to see?

OTLP JSON log records ({"resourceLogs":[{"resource":...) delimited by line breaks so it can be parsed by the OTel Collector filelog receiver and otlpjson connector

What did you see instead?

OTLP JSON log records ({"resourceLogs":[{"resource":...) lacking delilmiters

What version and what artifacts are you using?
Artifacts: OpenTelemetry Java Instrumentation Agent with the experimental-otlp/stdout exporter
Version: v2.9.0
How did you reference these artifacts? `java -javaagent:...

Environment
OS: MacOS 15.0.#1
Runtime (if different from JDK above): openjdk version "21.0.5" 2024-10-15 / OpenJDK 64-Bit Server VM Homebrew (build 21.0.5, mixed mode, sharing)

Additional context

cc @zeitlinger

[zeitlinger]

As per open-telemetry/opentelemetry-specification#4183 it it allowed to have multiple log lines in one resource logs. If the connector doesn't support this it should be fixed - not the Java part.
I didn't test that myself BTW.

[zeitlinger]

Oh, I see the problem now in the Java part.