HealthStatusManager.setStatus can throw a runtime exception when notifying (already cancelled) watchers #11853
Description
What version of gRPC-Java are you using?
1.65.1
What is your environment?
Jammy Ubuntu running on ARM64, JDK 21.0.4
What did you expect to see?
No exception. Healthcheck watchers with connections cancelled can/should be ignored.
What did you see instead?
The follow exception occurs:
io.grpc.StatusRuntimeException: CANCELLED: call already cancelled. Use ServerCallStreamObserver.setOnCancelHandler() to disable this exception
at io.grpc.Status.asRuntimeException(Status.java:525)
at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onNext(ServerCalls.java:366)
at io.grpc.protobuf.services.HealthServiceImpl.notifyWatchers(HealthServiceImpl.java:179)
at io.grpc.protobuf.services.HealthServiceImpl.setStatusInternal(HealthServiceImpl.java:130)
at io.grpc.protobuf.services.HealthServiceImpl.setStatus(HealthServiceImpl.java:122)
at io.grpc.protobuf.services.HealthStatusManager.setStatus(HealthStatusManager.java:67)
...
Due to this exception not all Healthcheck watching clients are notified.
Steps to reproduce the bug
Use HealthStatusManager.setStatus to set the health check status on a running server with a large number of clients, including some recently closed clients. Setting the status too recently after a client has closed its stream will trigger the exception.