Can't found the trace chain between Armeria service when use opentelemetry as java agent

[cxjava]

Hi Armeria team,

Recently I try to use the opentelemetry in our armeria service, but found can't find the trace chain in the jaeger.

for example: spring-boot-tomcat -> spring-boot-jetty -> demo, it will found the trace chain in the jaeger normally,

the result in our service is that: service A call service B , service A and service B is use the build in armeria server( not tomcat and jetty). Only found the service A in the Jaeger, don't found the Service B in the Jaeger.

versions

• opentelemetry java agent: 1.31.0
• Armeria: 1.25.2
• Jaeger v1.48.0

code

to make it sample, I try to reproduce it in the armeria example project.

call chain is : spring-boot-jetty -> spring-boot-tomcat -> spring-boot-minimal

spring-boot-minimal(no changes, port is 8080):

@Component
@Validated
@ExceptionHandler(ValidationExceptionHandler.class)
public class HelloAnnotatedService {

    @Get("/")
    public String defaultHello() {
        return "Hello, world! Try sending a GET request to /hello/armeria";
    }

    @Get("/hello/{name}")
    public String hello(
            @Size(min = 3, max = 10, message = "name should have between 3 and 10 characters")
            @Param String name) {
        return String.format("Hello, %s! This message is from Armeria annotated service!", name);
    }
}

how to run spring-boot-minimal

./gradlew clean bootJar
java -javaagent:/Users/user/Downloads/opentelemetry-javaagent.jar \
     -Dotel.resource.attributes=service.name=spring-boot-minimal \
     -Dotel.traces.exporter=jaeger \
     -Dotel.metrics.exporter=none \
     -Dotel.exporter.otlp.endpoint=http://localhost:4317 \
     -jar ./build/libs/spring-boot-minimal.jar

spring-boot-tomcat(change port to 8180) :

@RestController
public class HelloController {
    @RequestMapping(method = RequestMethod.GET, path = "/hello")
    String hello() {
        return "Hello, from tomcat &  "
                + WebClient.of("http://localhost:8080")
                        .get("/hello/minimal")
                        .aggregate()
                        .join()
                        .contentUtf8();
    }
}

how to run spring-boot-tomcat

./gradlew clean bootJar
java -javaagent:/Users/user/Downloads/opentelemetry-javaagent.jar \
     -Dotel.resource.attributes=service.name=spring-boot-tomcat \
     -Dotel.traces.exporter=jaeger \
     -Dotel.metrics.exporter=none \
     -Dotel.exporter.otlp.endpoint=http://localhost:4317 \
     -jar ./build/libs/spring-boot-tomcat.jar

spring-boot-jetty(change port to 8280):

@RestController
public class HelloController {
  
    @RequestMapping(method = RequestMethod.GET, path = "/hello")
    String hello() {
        return "Hello, from jetty & "
                + WebClient.of("http://localhost:8180")
                        .get("/hello")
                        .aggregate()
                        .join()
                        .contentUtf8();
    }
}

how to run spring-boot-jetty

./gradlew clean bootJar
java -javaagent:/Users/user/Downloads/opentelemetry-javaagent.jar \
     -Dotel.resource.attributes=service.name=spring-boot-jetty \
     -Dotel.traces.exporter=jaeger \
     -Dotel.metrics.exporter=none \
     -Dotel.exporter.otlp.endpoint=http://localhost:4317 \
     -jar ./build/libs/spring-boot-jetty.jar

When I run curl http://localhost:8280/hello , the response is Hello, from jetty & Hello, from tomcat & Hello, minimal! This message is from Armeria annotated service!

But the call chain is like below, trace id is end in the spring boot minimal.

don't find the spring-boot-minimal:

If I call the spring-boot-minimal curl http://localhost:8080/hello/test, it will work fine and can find the trace in the Jeager.

end

Seems opentelemetry works fine with tomcat and jetty.

I am not sure about that is it the correct way armeria integrate with opentelemetry.

I also read some issue such as #4570 , don't find any solution. And I also don't know it is the issue of Armeria, or an issue of opentelemetry java agent.

Waiting for your response~

[jrhee17]

Hi, I think this is probably better to ask to the opentelemetry team.

When I tried setting the exporter to logging (i.e. -Dotel.traces.exporter=logging) I found that actually armeria server isn't exporting any traces. (the netty extension seems to be exporting spans, and the armeria client seems to be exporting spans)

Checking the code at otel side for the java agent, I couldn't find any points which actually exports a span.

https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/armeria-1.3/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/armeria/v1_3/ServerDecorator.java

Do you mind opening a thread at the otel instrumentation side and mentioning me?

[cxjava]

Do you mind opening a thread at the otel instrumentation side and mentioning me?

Sure, I will create it later

[cxjava]

@jrhee17 Already created the issue in the otel instrumentation side, -> open-telemetry/opentelemetry-java-instrumentation#9695 .

[jrhee17]

Thanks for the ping, left a comment

[cxjava]

Hi @jrhee17 , I think OTEL team already fixed it in this PR, I test it again with the snapshots version, it works.

Thank you for the support!

[cxjava]

It seems still have issue when web client call Service A's GRPC method, and service A's GRPC method will call service B's GRPC method. Call chain is web client -> Service A's GRPC method -> Service B's GRPC method, let me test it again.

[cxjava]

seems the snapshot version also miss the client span if the call chain is web client -> Service A's GRPC method -> Service B's GRPC method

server a span -> client span(missed) -> server b span

[jrhee17]

I see, my guess is that grpc doesn't use WebClient which our gRPC clients are using

https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/93e06678168c5bce2c2a892685b06cceb8c1519d/instrumentation/armeria-1.3/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/armeria/v1_3/ArmeriaWebClientBuilderInstrumentation.java#L23

My guess is that the following might be a better injection point, but I'll have to do some testing

armeria/core/src/main/java/com/linecorp/armeria/client/AbstractClientOptionsBuilder.java

Line 503 in bb79158

protected final ClientOptions buildOptions(@Nullable ClientOptions baseOptions) {

[cxjava]

I create other issue in OTEL side, just for tracking
open-telemetry/opentelemetry-java-instrumentation#9726

[cxjava]

@jrhee17 Can you help to have a look in this issue? Any actions need to do in the Armeria side? Thank you!

[jrhee17]

Left a comment 😄

[cxjava]

The related issue in opentelemetry-java-instrumentation open-telemetry/opentelemetry-java-instrumentation#9726 has already been closed with this PR, so I think we can close this one too.

Now if we use the opentelemetry-javaagent-2.4.0-SNAPSHOT.jar, it can show the trace chain between Armeria services.

trace screenshot

@jrhee17 Feel free to close this issue.

[minwoox]

Thanks a lot for reporting, @cxjava! 😉