Update GRPC client calls and service handlers to support cancellations and deadlines

[tegefaulkes]

Specification

GRPC calls need to support cancellation and deadlines. This means that the GRPC client call needs to respond to a abortion and stop the current call. The GRPC service handler will also need to respond to an abortion, either triggered by a deadline connection failure or the client abortion, and finish processing the request early.

Additional context

• Related Apply timedCancellable across the board to control how long side-effects are allowed to complete #450
• Related Integrate TaskManager into NodeGraph and Discovery #445

Tasks

1. Determine how to interface with the GRPC API to implement timeouts and cancellablility.
2. Convert the Client side methods to use timedCancellable.
3. Covert the service handlers to use timeouts and cancellability
4. Update and use the test client/ handlers to do the same and make tests to check the behaviour.

[tegefaulkes]

This one is pretty complex. We need to apply cancellation to the GRPC client calls, this includes the timeouts using timedCancellable. We also need to implement this for the service handlers where we set up a timer based on the metadata and abortion if the connection ends.

The first stage will deal with researching the GRPC API to work out how to hook up cancel ability with the API. This includes aborting the client connection, setting a timeout for the client calls, getting the timeouts on the service side and handling connection drops. The next stage is updating the client side to make use of these and then converting all of the service handlers.

Needs some tests for good measure, but we don't want to test EVERY client call or service handler. I'll have to look into it a little but I think we can make use of the test client/server for this? Might be a good place to start.

[CMCDragonkai]

Yea I think it's enough to test that the mechanism works for unary, client streaming, server streaming and bidi streaming and that's it.

[CMCDragonkai]

The most important RPC calls will be the ones that have "long running things" that depend on other agents like node cross claim.

[CMCDragonkai]

Hey @tegefaulkes instead of just setting it to 8, anything larger than a 5 should be broken down into smaller issues so that way it's easier to track separately. I think this issue in particular may require an epic to split out in to all the domains.

[CMCDragonkai]

Also once it works on one domain, I'm sure it can be copied out to all the other ones.

[CMCDragonkai]

In terms of reserving the meta property of the request params:

{
  params: {
    meta: {
      token: "abc"
      deadline: 342341,
    }
  }
}

This should allow the client to communicate to the agent how much time it wants to give the agent to complete the work.

Even if the client doesn't send a deadline or timeout, the agent still needs to have a default timeout for each handler.

Furthermore there should be a limit to the deadline - how large it can be.

There are some situations where a deadline doesn't apply. It is possible in some cases for the deadline to just be null. Notice that null is separate from undefined. Where null is an explicit configuration of having no deadline at all.

This means we can have push-flows that last forever. But still rely on the underlying transport idleness timer that can close connections if nothing progresses there.

[CMCDragonkai]

Ideas regarding request metadata:

1. token
2. deadline
3. version

Ideas regarding response metadata:

1. token
2. version

[tegefaulkes]

Yep, we can add these fields the the request and response metadata wrapper types. We can use the middleware to enforce the version requirements. The deadlines needs a little bit of thought since we need to pass it into the handler using the provided context object. So either deadline support needs to be hard coded into the RPC OR the middleware needs access to an abort controller so it can abort the current handler on a condition such as a timer.