Memory Leak - Object throwableToSpan keeps growing over time

[klem231188]

Integration

sentry-spring

Java Version

11

Version

5.7.4

Steps to Reproduce

I use Sentry in a springboot project.
The project offers some REST and SOAP endpoints used by third parties.
I use Sentry mainly to :

• easily find bottlenecks (using sentry-jdbc)
• collect errors and group them in "Issues" (using sentry-logback)

Here are the artifacts I use:

		<dependency>
			<groupId>io.sentry</groupId>
			<artifactId>sentry-spring-boot-starter</artifactId>
		</dependency>

		<dependency>
			<groupId>io.sentry</groupId>
			<artifactId>sentry-logback</artifactId>
		</dependency>

		<dependency>
			<groupId>io.sentry</groupId>
			<artifactId>sentry-jdbc</artifactId>
		</dependency>

		<dependency>
			<groupId>com.github.gavlyukovskiy</groupId>
			<artifactId>p6spy-spring-boot-starter</artifactId>
			<version>${p6spy-spring-boot-starter.version}</version>
		</dependency>

In production, there are around 50K calls/week. A few weeks ago I got my first OOM which surprised me as the app generally works like a charm.

It happened few times after I added the Sentry features to the project. Unfortunately, an API caller made a lot (lot, lot) of unsuccessful calls successively.
So I tried to reproduce it using a LoadTest: 5 threads doing 100_000 iteration loop, wrongly calling the api which generates an error.
The idea is to look at the Heap space over time and perform a heap dump at the end.

Here is my analysis :
In my controller, I do the try/catch/finally block manually to handle the errors.
In the catch block, I do the following thing:

            // Handle Sentry transaction
            ISpan transaction = Sentry.getSpan();
            if (transaction != null) {
                transaction.setThrowable(e);
                transaction.setStatus(SpanStatus.INTERNAL_ERROR);
            }

Note that I do NOT propagate the exception at the end of the catch block.
Before returning the response, it enters into the SentryTracingFilter, in the finally block it calls transaction.finish() (line 87)
In this piece of code, it stores the throwable in the throwbableToSpan map of the Hub .
However, I can't see any throwbableToSpan.remove() in the code. I noticed that you use a WeakHashMap, but I didn't find neither where the key reference is set to null. --> So the map keeps growing over time.

Either in the catch block of my controller, I should not store the throwable like that... or there is an error in the Sentry lib.

Expected Result

The field throwableToSpan from class Hub should not grow indefinitely.

Actual Result

I get an OOM
Below some screenshots of visualvm + IntelliJ debug breakpoint.
The heapdump is more than 500MB, so I can't upload it here.

[adinauer]

@klem231188 thank you for the detailed report. Can you please give this a try with the latest version? I am unable to reproduce it locally. The throwableToSpan map is cleaned up after the request is finished in my tests. In case this also happens with the latest version, can you please share your options that you use for the Sentry SDK?

[klem231188]

I tested it in 6.4.0. I still have the issue.
See image below. In the throwableToSpan object, I can see the previous entries in the map not being discarded.

Here is my application-sentry.yml file :

sentry:
  dsn: https://some-secret-hash.ingest.sentry.io/some-number
  environment: ${environment.type}
  logging:
    enabled: true
    minimumBreadcrumbLevel: INFO
    minimumEventLevel: ERROR
  release: @project.version@
  sample-rate: 0.1
  send-default-pii: true
  server-name: ${environment.name}
  traces-sample-rate: 0.1
  use-git-commit-id-as-release: false

Note that, for my test case, I also reduced the number of tomcat thread to 1, so all the HTTP requests are always processed by the same thread.

server:
  tomcat:
    max-threads: 1

[adinauer]

Thanks @klem231188 I'm now able to reproduce it using server.tomcat.threads.max=1 in application.properties. I'll investigate and get back to you once I know more.

[adinauer]

The problem seems to be caused by having the root cause of the throwable as the key of the WeakHashMap while at the same time the Span, which is the value of the WeakHashMap is holding on to the throwable. This means the value of the WeakHashMap is holding a reference to its own key thus keeping it from being garbage collected. I'll talk to the team to figure out if we can remove some code or need to keep and fix it.

[klem231188]

Thanks @adinauer for your fast response.

As a quick fix, I decided to comment/remove the following line :

try {
   // Process the request
} catch (Exception e) {
    // Exception handling

    // Handle Sentry transaction
    ISpan transaction = Sentry.getSpan();
    if (transaction != null) {
        // Do I really need this statement below ? 
        // I can't even find where it is displayed in the Sentry web portal
        // It causes the OOM on long term
        // transaction.setThrowable(e); 
        transaction.setStatus(SpanStatus.INTERNAL_ERROR);
    }
} finally {
    // ...
}