slf4j (logback) logs not getting forwarded to open-telemetry-collector by OpenTelemetryAppender

[ff-sdesai]

I have created a spring-boot micro-service with spring-cloud-sleuth. I have also configured a open-telemetry-collector to forward the logs to datadog. The otel-collector is running in a docker container along with the micro-service. I have configured io.opentelemetry.instrumentation.logback.v1_0.OpenTelemetryAppender in logback.xml file to forward the logs also to otel-collector along with traces. However, in datadog, I can only see traces and not logs. The logs printed are using the same traceId and spanId as that of trace. But due to some unknown reason, otel-collector is not exporting logs like it is exporting traces.

This issue can be reproduced using a POC with very few files and is uploaded at https://github.com/ff-sdesai/distributed-tracing-spring. The readme.md has the steps to run the application. You will need to enter a valid datadog API key in the .env (hidden) file.I will not be able to share the key since the key is specific to my organization.
Please let me know if there is something else that I should specify along with these details.

I am using docker image otel/opentelemetry-collector-contrib:0.70.0 and Ubuntu 22.04.1 LTS.

[trask]

hi @ff-sdesai!

can you narrow down the issue to one of these components so we can get this issue to the appropriate folks/repository?

• OpenTelemetry Java Instrumentation
• OpenTelemetry Collector
• OpenTelemetry Collector DataDog Exporter
• DataDog Ingestion

[ff-sdesai]

Opentelemetry collector is not receiving the logs sent by the application. I confirmed this by sending logs to otel-collector using curl and it works fine.
So, it looks to be an issue with OpenTelemetry Java Instrumentation(io.opentelemetry.instrumentation.logback.v1_0.OpenTelemetryAppender)

[laurit]

Firstly in https://github.com/ff-sdesai/distributed-tracing-spring/blob/main/spring-cloud-sleuth-otel-slf4j/src/main/resources/logback.xml#L13 you are using io.opentelemetry.instrumentation.logback.v1_0.OpenTelemetryAppender this is the appender for mdc instrumentation. For exporting logs you'll need io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender.
Secondly in https://github.com/ff-sdesai/distributed-tracing-spring/blob/ee36dcd78ce9c0b517355642af3c5e0b8b6d67d0/spring-cloud-sleuth-otel-slf4j/src/main/resources/logback.xml#L17 you should use the appender for it to have any effect.
Thirdly in https://github.com/ff-sdesai/distributed-tracing-spring/blob/main/spring-cloud-sleuth-otel-slf4j/pom.xml you have

		<dependency>
			<groupId>io.opentelemetry</groupId>
			<artifactId>opentelemetry-exporter-otlp-trace</artifactId>
			<version>1.14.0</version>
		</dependency>

you don't need this, you already have dependency to a newer version of opentelemetry-exporter-otlp. Also

		<dependency>
			<groupId>io.opentelemetry.instrumentation</groupId>
			<artifactId>opentelemetry-logback-1.0</artifactId>
			<version>1.9.2-alpha</version>
			<scope>runtime</scope>
		</dependency>

is for mdc instrumentation. If you need it then please try to align the version with the rest of the otel artifacts. You'll need to add

		<dependency>
			<groupId>io.opentelemetry</groupId>
			<artifactId>opentelemetry-exporter-otlp-logs</artifactId>
			<version>1.22.0-alpha</version>
		</dependency>

Finally see https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/logback/logback-appender-1.0/library#usage You'll have to configure the GlobalLoggerProvider. Try adding

        LogRecordExporter otlpGrpcLogRecordExporter =
                OtlpGrpcLogRecordExporter.builder()
                        .setEndpoint("http://localhost:4317")
                        .build();
        SdkLoggerProvider sdkLoggerProvider =
                SdkLoggerProvider.builder()
                        .addLogRecordProcessor(SimpleLogRecordProcessor.create(otlpGrpcLogRecordExporter))
                .build();
        GlobalLoggerProvider.set(sdkLoggerProvider);

to the start of the main method. There probably is a way to get SdkLoggerProvider from spring configured otel sdk. If you don't have collector running you should now see lines like 13:26:29.851 [OkHttp http://localhost:4317/...] ERROR i.o.e.i.grpc.OkHttpGrpcExporter serviceName: requestId: trace_id: span_id: - Failed to export logs. The request could not be executed. Full error message: Failed to connect to localhost/0:0:0:0:0:0:0:1:4317

[ff-sdesai]

@laurit Please ignore my previous comment (which I have deleted). I am checking if the configuration suggested by you is working for me. Will update here soon

[ff-sdesai]

@laurit I did not understand the second point you said. After doing remaining changes, I am now getting following error:-
ERROR in ch.qos.logback.core.joran.action.AppenderRefAction - Could not find an AppenderAttachable at the top of execution stack. Near [appender-ref] line 19
Looks like it is not able to find a property appender-ref in io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender. Any suggestions what can be done here?

[laurit]

remove <appender-ref ref="STDOUT" /> from otel appender and add <appender-ref ref="OTEL" /> under <root level="INFO">

[ff-sdesai]

Thanks @laurit for your help . I can see the logs in datadog now