Remove Observation.Context from WebClient request

[erimolNordnet]

Have a running app since 3 years in production without any issue. Experienced a memory leak when bumped Spring dependencies and have now traced it down to a single library: spring-webflux 6.1.1 -> 6.1.2 (no other code change)

It seems like their is a circular dependency between ClientRequestObservationContext and ClientRequest.Builder that cannot be garbage collected. The behaviour is deterministic in the sense that I managed to recreate the issue multiple times, it takes a couple of hours before running out of heap space. One instance of the application is quite heavy used with 50-100 requests / s and lots of downstream calls to other MicroServices using WebClient where the issue seems to be.

Changes that I think could be related:

• b878526#diff-3eb216adea96b2aa6651e8b1522b4c6b18a5271698d4478ac7b920eb90bbdfbe
• Improve "active" metrics handling in WebClient observations #31702

Please let me know if I can provide more information, first time posting a bug report here.

// Erik

Have done an heap dump multiple times and retained memory increasing for the specified types. Attaching a print screen of the heap dump:

Heap before update

Heap after update

[bclozel]

Thanks for the report.
Unfortunately, I can't reproduce this issue locally. I'm using Spring Boot 3.2.2 and I've locked the Spring Framework version to ext['spring-framework.version'] = "6.1.2".

I'm using the following controller to run many requests with the WebClient:

package com.example.webclientleak;

import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.reactive.function.client.WebClient;

@RestController
public class TestController {

    private final WebClient webClient;

    public TestController(WebClient.Builder builder) {
        this.webClient = builder.baseUrl("http://localhost:8080").build();
    }

    @GetMapping("/test")
    public String test() {
        return "TEST";
    }

    @GetMapping("/")
    public Mono<String> index(@RequestParam int count) {
        return Flux.range(1, count)
                .flatMap(i -> this.webClient.get().uri("/test").retrieve().bodyToMono(String.class))
                .last();
    }
}

I ran many of queries against the main endpoint (so millions of webclient requests) and I don't see any memory leak in my tools. The Java heap is not growing and I can consistently get it down when triggering manual GCs.

I don't understand why #31702 would change anything, as this commit doesn't change the nature of this "cycle" since the observation context was already pointing to the request builder and the request (which itself contains the context as an attribute). This change merely changes the ordering of things. I don't think cycle like this cause memory leaks in the first place, as they're not reachably from GC roots once the request is sent. Do you have a different perspective on this?

It would be interesting to get why those context instances are still reachable in your case and what's holding a reference to them. In my case, I only had a couple of those in memory as they were held by the observation registry, waiting to be sent/processed.

Additionnally, any indication on how to reproduce this locally would be very much welcome.

Thanks!

[erimolNordnet]

Not really sure either, but as the heapdump indicates some memory are retained and the change happened to the classes what I can see. That was my only lead:
From 6.1.2:

ClientRequestObservationContext observationContext = new ClientRequestObservationContext(requestBuilder);

and before in 6.1.1 it was created without the request:

ClientRequestObservationContext observationContext = new ClientRequestObservationContext();

Also this in 6.1.2

	ClientRequest request = requestBuilder
						.attribute(ClientRequestObservationContext.CURRENT_OBSERVATION_CONTEXT_ATTRIBUTE, observationContext)
						.build();

At least it feels related from the bump in version and what I can see from the dump.

WIll give it a try locally in my integration test if I can hammer on one endpoint, haven't been able to reproduce locally so far either

[erimolNordnet]

Will give you the trace from heapdump as well:

[bclozel]

Did the reactor-netty dependency version change at all between both?
Can you replicate that on a smaller sample?

[erimolNordnet]

Only change was webflux, I done a lot of tests to be able to isolate which change caused the issue:
I explicit added this dependency and the only change was from 6.1.1 to 6.1.2

<dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-webflux</artifactId>
            <version>6.1.2</version>
        </dependency>