Using Application Insights java Agent is impossible to suppress some handled exceptions

[apescione]

Context

I'm using a Spring Boot Application (Java) and WebFlux as the web layer (Reactor), and Tomcat is used as an Embedded web server.
In my application, I've Server-Sent-Event API like this:

  @RequestMapping(value = "/messages",
        produces = {"text/event-stream", "application/stream+json"},
        method = RequestMethod.GET)
    public Mono<ResponseEntity<Flux<ServerSentEvent<String>>>> getSseMessages() {

This API sends events to clients using a HOT Stream.
I'm using the application insight java agent to instrument my application deployed into a docker container.
When a client disconnects himself from my application, the Spring application, in the next update through the client raises a
ClientAbortException and inner cause java.io.IOException: Broken pipe.

Expected behavior

Having a way to choose what exception I want to track. So, in my scenario, I don't want to consider this kind of exception "ClientAbortException and inner cause java.io.IOException: Broken pipe." on a particular API, because I know that sooner or later a client will disconnect itself (Close front end application, change page, etc...)

Actual behavior

Reading Microsoft documentation, it seems there is a way to choose the "Sampling" for each metric, so that, if you set percentage to 0, you can suppress the metric and avoid that agent sents it to cloud.
Here's the documentation: https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-sampling-overrides

Despite it is a preview, it works fine for AgentLogExporter, in fact, if the "exception" is coming from a log, it can be managed, but if the metric is from TelemetryItemExporter, it seems there is no way to suppress it.
Here's the metric that I'd like to suppress:

2023-02-27 16:25:38.808Z DEBUG c.a.m.o.e.i.p.TelemetryItemExporter - sending telemetry to ingestion service:
....
{"ver":1,"name":"Exception","time":"2023-02-27T16:25:38.31Z","iKey":"9805179f-a509-4c79-ac20-bf6753b671f9","tags":{"ai.internal.sdkVersion":"java:3.4.9","ai.operation.id":"434698b7492fa6cbc81e233da0f92af4","ai.cloud.roleInstance":"9caa85af7fa8","ai.operation.name":"GET /api/messages","ai.cloud.role":"ANTONIO_TEST_APP_52","ai.user.userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36","ai.operation.parentId":"eb54096db74ef832"},"data":{"baseType":"ExceptionData","baseData":{"ver":2,"exceptions":[{"typeName":"java.io.IOException","message":"Broken pipe","stack":"java.io.IOException: Broken pipe\n\tat sun.nio.ch.FileDispatcherImpl.write0(Native Method)\n\tat sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)\n\tat sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)\n\tat sun.nio.ch.IOUtil.write(IOUtil.java:65)\n\tat sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470)\n\tat org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:135)\n\tat org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1417)\n\tat org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:773)\n\tat org.apache.tomcat.util.net.SocketWrapperBase.flushNonBlocking(SocketWrapperBase.java:744)\n\tat org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:718)\n\tat org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.flush(Http11OutputBuffer.java:573)\n\tat org.apache.coyote.http11.filters.ChunkedOutputFilter.flush(ChunkedOutputFilter.java:157)\n\tat org.apache.coyote.http11.Http11OutputBuffer.flush(Http11OutputBuffer.java:221)\n\tat org.apache.coyote.http11.Http11Processor.flush(Http11Processor.java:1255)\n\tat org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:402)\n\tat org.apache.coyote.Response.action(Response.java:209)\n\tat org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:306)\n\tat org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:273)\n\tat org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118)\n\tat org.springframework.http.server.reactive.ServletServerHttpResponse.flush(ServletServerHttpResponse.java:211)\n\tat org.springframework.http.server.reactive.ServletServerHttpResponse.access$600(ServletServerHttpResponse.java:50)\n\tat org.springframework.http.server.reactive.ServletServerHttpResponse$ResponseBodyFlushProcessor.flush(ServletServerHttpResponse.java:322)\n\tat org.springframework.http.server.reactive.AbstractListenerWriteFlushProcessor$State$3.writeComplete(AbstractListenerWriteFlushProcessor.java:312)\n\tat org.springframework.http.server.reactive.AbstractListenerWriteFlushProcessor$State$WriteResultSubscriber.onComplete(AbstractListenerWriteFlushProcessor.java:465)\n\tat org.springframework.http.server.reactive.WriteResultPublisher$State.publishComplete(WriteResultPublisher.java:266)\n\tat org.springframework.http.server.reactive.WriteResultPublisher$State$1.subscribe(WriteResultPublisher.java:171)\n\tat org.springframework.http.server.reactive.WriteResultPublisher.subscribe(WriteResultPublisher.java:77)\n\tat org.springframework.http.server.reactive.AbstractListenerWriteProcessor.subscribe(AbstractListenerWriteProcessor.java:205)\n\tat org.springframework.http.server.reactive.AbstractListenerWriteFlushProcessor$State$2.onNext(AbstractListenerWriteFlushProcessor.java:294)\n\tat org.springframework.http.server.reactive.AbstractListenerWriteFlushProcessor.onNext(AbstractListenerWriteFlushProcessor.java:120)\n\tat org.springframework.http.server.reactive.AbstractListenerWriteFlushProcessor.onNext(AbstractListenerWriteFlushProcessor.java:43)\n\tat org.springframework.http.server.reactive.ChannelSendOperator$WriteBarrier.onNext(ChannelSendOperator.java:173)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onNext$1(TracingSubscriber.java:62)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:83)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onNext(TracingSubscriber.java:62)\n\tat reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onNext$1(TracingSubscriber.java:62)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:83)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onNext(TracingSubscriber.java:62)\n\tat reactor.core.publisher.FluxFlatMap$FlatMapMain.tryEmit(FluxFlatMap.java:543)\n\tat reactor.core.publisher.FluxFlatMap$FlatMapInner.onNext(FluxFlatMap.java:984)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onNext$1(TracingSubscriber.java:62)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:83)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onNext(TracingSubscriber.java:62)\n\tat reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.lambda$onNext$1(TracingSubscriber.java:62)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.withActiveSpan(TracingSubscriber.java:83)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.reactor.TracingSubscriber.onNext(TracingSubscriber.java:62)\n\tat reactor.core.publisher.FluxInterval$IntervalRunnable.run(FluxInterval.java:125)\n\tat reactor.core.scheduler.PeriodicWorkerTask.call(PeriodicWorkerTask.java:59)\n\tat reactor.core.scheduler.PeriodicWorkerTask.run(PeriodicWorkerTask.java:73)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:750)\n"}]}}}
...

To Reproduce

It's very simple, create with web flux a server sent event api that produce an event every few seconds, connect the client (browser is enough) and close the browser. The next event that cannot be write on stream because the client is disconnected you are going to have a IOException: Broken pipe.

Sample Application

@RestController
@RequestMapping("/api")
public class ServerSentEventApi {
...


    @RequestMapping(value = "/messages",
        produces = {"text/event-stream", "application/stream+json"},
        method = RequestMethod.GET)
    public Mono<ResponseEntity<Flux<ServerSentEvent<String>>>> getSseMessages() {
        Flux<ServerSentEvent<String>> sseFlux = Flux.interval(Duration.ofSeconds(5))
            .map(i -> ServerSentEvent.<String>builder().event("ping")
                .data(i.toString()).id(RandomStringUtils.randomAlphanumeric(8)).build());
        return Mono.just(ResponseEntity.ok().body(sseFlux));
    }
...

}

System information

Please provide the following information:

• SDK Version: Java 8 Zulu, Spring Boot 2.6.8, Webflux 5.3.20, tomcat 9.0.63
• OS type and version: Container Linux alphine

Consideration

Avoiding the sending of these metrics, I want to also reduce the cloud cost, because in my scenario I have always a huge number of this unuseful exceptions

Thank you in advance.

[trask]

hi @apescione! what version of Application Insights Java are you using?

see https://github.com/microsoft/ApplicationInsights-Java/releases/tag/3.4.0:

• Exceptions are no longer captured directly on dependency records for these reasons:
  • in order to reduce ingestion cost
  • dependency exceptions which are uncaught, bubble up to the request-level where they are already captured
  • dependency exceptions which are caught, tend to be logged if they are important, where they are also already captured
    (#2423).

[trask]

oh, nevermind, I see you are using 3.4.9 and these are exceptions on requests (not dependencies)

[heyams]

@apescione can you push a sample app to https://github.com/microsoft/ApplicationInsights-Java-Repros for a repro? thanks.

[apescione]

Hi @heyams, I have no permission on that repo. Could you create one branch to push the example on it?

[trask]

hey @apescione, can you fork the repo and send PR from your fork?

[apescione]

I've push in a my repo. You can find an example here.
https://github.com/apescione/applicationinsightTestApps
Let me know if you need more information

[heyams]

@apescione

quote:
It's very simple, create with web flux a server sent event api that produce an event every few seconds, connect the client (browser is enough) and close the browser. The next event that cannot be write on stream because the client is disconnected you are going to have a IOException: Broken pipe.

can you help me with generating an IOException? I sent api request using your app and then close the browser. Nothing happens. What do you mean connect the client (browser is enough)? Can you push some repro instructions to your repo? I attached it using `java -javaagent:xxxx.jar -jar ./target/message-0.0.1-SNAPSHOT.jar' and didn't use your docker-compose.

never mind, i was able to repro using http://127.0.0.1:8080/api/messages