feat: Add graceful shutdown timer to GRPC frontend

[mattwittwer]

What does the PR do?

This PR adds a shutdown timer to the gRPC endpoint for both infer and streaming infer requests. Inflight requests will be allowed to complete before the timer expires and new requests made after shutdown has started will be rejected.

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Related PRs:

N/A

Where should the reviewer start?

Start with the new functions in src/grpc/grpc_server.cc and the flow in src/main.cc
GracefulStop();
unload the models in core
Stop();

Test plan:

Review L0_lifecycle and grpc related tests.
Updated shutdown behavior changes the tested responses for some lifecycle tests

• CI Pipeline ID: 27245783

Caveats:

The updated shutdown process includes three responses. The CANCELLED state cannot be removed as it is part of the GRPC endpoint shutdown behavior:

• "GRPC server is shutting down and has stopped accepting new requests."
• "CANCELLED"
• "failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8001: Failed to connect to remote host: connect: Connection refused (111)"

Background

Shutdown behavior for the gRPC endpoint could lead to unexpected errors when unloading the models from the core

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

N/A

[indrajit96]

1. Let's create a TEP for this as discussed.
2. Test a simple no-model perf with and without changes to see if any perf regression was introduced

[indrajit96]

LGTM, make sure you attach perf numbers to the ticket and the update the latest CI pipeline on the reviews thread.
Nice work!