[FLINK-34644][runtime] Fix race condition in RestServerEndpoint shutdown

[MartijnVisser]

What is the purpose of the change

Fix the flaky test RestServerEndpointITCase.testShouldWaitForHandlersWhenClosing by addressing a race condition in AbstractRestHandler.respondToRequest() where the HTTP response write future was silently discarded, allowing the in-flight request tracker to deregister requests before the response was fully flushed to the network.

Brief change log

• Changed thenAccept to thenCompose in AbstractRestHandler.respondToRequest() so that the returned future only completes after HandlerUtils.sendResponse() has fully flushed the HTTP response to the network

Verifying this change

This change is already covered by existing tests, such as RestServerEndpointITCase.testShouldWaitForHandlersWhenClosing which verifies that in-flight requests complete successfully during server shutdown. The test was previously flaky due to this race condition and now passes reliably.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[flinkbot]

CI report:

• 3aef03c Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[rionmonster]

@MartijnVisser

I did a quick cursory search for other use of the thenAccept() calls to identify any other potential related problems and I think we may want to also update the AbstractSqlGatewayRestHandler as it seems like it may be subject to the same problem:

// AbstractSqlGatewayRestHandler:91-99
return response.thenAccept(
    resp ->
        HandlerUtils.sendResponse(
            ctx,
            httpRequest,
            resp,
            messageHeaders.getResponseStatusCode(),
            responseHeaders));

Thoughts? Otherwise the proposed fix seems reasonable! 👍

[MartijnVisser]

I did a quick cursory search for other use of the thenAccept() calls
Thoughts? Otherwise the proposed fix seems reasonable! 👍

Should have done that myself, good call-out. Let me fix that, thanks!

[rionmonster]

@MartijnVisser

All good! Approved! 👍