Flaky test: R2DBCTestKitImplTest.batch intermittently fails with "Connection pool shut down"

[polyglotAI-bot]

Description

com.clickhouse.r2dbc.spi.test.R2DBCTestKitImplTest.batch (module clickhouse-r2dbc) fails intermittently in CI. The other 31 tests of the same class pass in the same run, and the sibling matrix legs (other r2dbc-spi versions / server versions) stay green, so an unrelated PR looks like it introduced a regression.

The failure is not a query failure: the Apache HttpClient connection manager is already shut down before the request leases a connection, and the test fails in ~0.04 s.

Steps to reproduce

1. Run the R2DBC CI job: mvn --batch-mode --projects clickhouse-r2dbc -DclickhouseVersion=<ver> -D'r2dbc-spi.version=<ver>' -Dprotocol=http verify.
2. Repeat. The failure is intermittent — a re-run of the identical commit passes.

Error Log or Exception StackTrace

java.lang.AssertionError: expectation "expectComplete" failed
  (expected: onComplete(); actual: onError(java.lang.IllegalStateException: Connection pool shut down))
  Suppressed: java.lang.IllegalStateException: Connection pool shut down
    at org.apache.hc.core5.pool.LaxConnPool.lease(LaxConnPool.java:163)
    at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.lease(...)
    ...
    at com.clickhouse.client.http.ApacheHttpConnectionImpl.post(ApacheHttpConnectionImpl.java:301)
    at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:196)

Evidence of non-determinism

Same commit, same check, fail then pass — run 32022532612 on PR #3056:

• attempt 1, leg R2DBC 1.0.0.RELEASE + CH 26.3 (http) → failure, Tests run: 32, Failures: 1, batch 0.041 s: https://github.com/ClickHouse/clickhouse-java/actions/runs/32022532612/job/95368314131
• attempt 2, same leg, same head 7c4b7a66 → success (all six R2DBC legs green).

An independent second observation, two weeks earlier and on a different PR and matrix leg — PR #3010, head d341a338, which touches only jdbc-v2 (SqlParserFacade + its tests) and cannot reach the R2DBC/HTTP path:

• leg R2DBC 0.9.1.RELEASE + CH latest (http) → failure, identical signature, batch 0.035 s: https://github.com/ClickHouse/clickhouse-java/actions/runs/30856790714/job/91832373067
• the other five R2DBC legs of that same run → success.

Local reproduction

Does not reproduce on an idle or loaded devbox (external ClickHouse 26.3, JDK 17, -Dprotocol=http, r2dbc-spi 1.0.0.RELEASE):

• 25 × full R2DBCTestKitImplTest (32 tests) with 6 busy CPU loops → 0 failures.
• 12 × R2DBCTestKitImplTest#batch pinned to a single CPU (taskset -c 0) with 8 busy CPU loops → 0 failures.

Total 37 local runs, 0 failures. CI runners are slower and more contended, which is consistent with the CI-only appearance. The job configuration itself notes thread pressure in this environment (build.yml: "http_client and apache_http_client do not work in CI environment (due to limited threads?)").

Expected Behaviour

batch completes deterministically. No request should be able to start after the connection that owns its HTTP connection manager has been closed.

Candidate mechanism (hypothesis — not proven)

batch is the only test in the class that runs two statements through one Batch, and the batch path executes them concurrently:

• ClickHouseBatch.execute() (clickhouse-r2dbc/src/main/java/com/clickhouse/r2dbc/ClickHouseBatch.java:32-38) maps the SQL list to Mono.fromFuture(request::execute) over a single shared, mutable ClickHouseRequest, then flatMaps them — flatMap subscribes inner sources concurrently (default concurrency 256), and the requests carry ClickHouseClientOption.ASYNC = true (set in ClickHouseConnection.createBatch()), so the actual POST runs on a client executor thread rather than the subscribing thread.
• ClickHouseConnection.close() (clickhouse-r2dbc/src/main/java/com/clickhouse/r2dbc/connection/ClickHouseConnection.java:70-78) calls client.close(), which closes that connection's PoolingHttpClientConnectionManager. Flux.usingWhen(..., Connection::close) in TestKit.batch triggers it as soon as the outer Flux terminates.

If the outer Flux can terminate while one of the two batch requests has been submitted to the executor but has not yet leased a connection, the close wins the race and that POST fails at LaxConnPool.lease — exactly the observed signature and the sub-100 ms failure time. This would make it a client-side race in product code, not test debt: an application closing an R2DBC connection right after a batch completes could see the same IllegalStateException instead of a clean completion.

We could not confirm this locally, so it is offered as a starting point only; no root cause is claimed. Note also #2976, a separate confirmed R2DBC connection-lifecycle defect in the same area.

Suggested direction

Make the batch's completion synchronize with all of its in-flight requests, so the connection cannot be closed while one is still starting (e.g. execute the batch statements sequentially with concatMap, or gate the connection close on the completion of every submitted request), instead of retrying or ignoring the failure in the test.

Configuration

Environment

• Cloud
• Client version: main (0.10.0-rc1-SNAPSHOT)
• Language version: JDK 17 (temurin)
• OS: ubuntu-latest (CI); Debian container (local attempt)

ClickHouse Server

• ClickHouse Server version: observed on 26.3 and on latest; not reproducible locally on 26.3
• Non-default settings: CI test fixtures (users.d/users.xml from the repo), custom_http_params=async_insert=0
• Tables involved: created by the r2dbc-spi TestKit (CREATE TABLE test ( test_value INTEGER ) ENGINE = Memory)

---

Found by automated CI monitoring of our own PRs across unrelated runs, then verified by re-checking the run history and attempting a local reproduction; filed as CI-observed.