Graceful shutdown depends on order of services

[butonic]

Even with #1478 the order in which services are shut down is random.
It is still possible for the storage-users service to stop accepting new requests that the ocdav service is trying to make while draining his request queue.

GRPC and HTTP servers will immediately close their socket, no longer accepting new connections.

So we no longer randomly kill the service with an exit(0) but requests might still fail...

In kubernetes we can work around this by configuring a preStop that waits longer than the graceful shutdown timeout as described in the example:

      terminationGracePeriodSeconds: 120 # extra long grace period
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        lifecycle:
          preStop:
            exec:
              # Real life termination may take any time up to terminationGracePeriodSeconds.
              # In this example - just hang around for at least the duration of terminationGracePeriodSeconds,
              # at 120 seconds container will be forcibly terminated.
              # Note, all this time nginx will keep processing requests.
              command: [
                "/bin/sh", "-c", "sleep 180"
              ]

To make the opencloud process itself shut down gracefully, we would have to know which service still has to accept requests, keep its listeners open until no other service depends on it. Unfortunately we cannot stop the proxy, because the ocdav might make an inernal request to it ... #1341 (comment) describes what is happening. Tracked in #705

For a single process, single host deployment we could try stopping only the proxy first and then after a configurable grace time the rest of the services.