chainhead backend: notify subscribers when the backend is closed

[niklasad1]

Close #1808

This PR fixes that once Shared::done then we also wake up the "waker" such that the FollowStreamSubscription poll impl gets notified about.

This issue was really that the FollowStreamSubscription wasn't woken up after that Shared.done == true was set.

Currently the backend only returns error once before it's shutdown and then return None but this forces users of the driver to run it completion and one doesn't run it to completion i.e, such as bailing on error then Shared::done == true is never set.

Thus, I also had to change that we run the driver until completion.

Since we already now that backend is shutdown when it returns error we could already set done == true here but it's doesn't feel correct to me but would probably help users of the low-level API not run into such tricky issues ^^, thoughts?

[jsdw]

Is the idea here that an RpcClient implementation would need to emit FollowEvent::BackendClosed (of the JSON of that at least) in order to tell subscriptions that they can actually completely stop? Whereas if the subscriptions just get a FollowEvent::Stop event then things might continue? (I guess the actual RpcClient impls would need tweaking too to make this PR useful, or maybe I'm missing something?)

[jsdw]

Looking at the PR, I wonder about a couple of things:

1. Would it make more sense to indeed forward the Error, since you can call is_disconnected_will_reconnect() on it to see if you need to handle that. In this PR we create a new state from the error that's a bit redundant perhaps because we already have the error? it feels most future proof to forward the error too, since then any other specific errors can be handled easily in subscriptions or whatever in the future.
2. If that doesn't work out, I also wonder whether to avoid adding FollowEvent::BackendClosed and instead have a wrapper enum like enum FollowEventOrClosed { Event(FollowEvent), Closed }. This means we don't have to handle for instance the possibility of the RpcClient returning an event like "backendClosed" (which should never happen), and then perhaps the InnerStreamState would need only one variant, BackendClosed(Error),

I'd be up for seeing whether (1) was possible anyways or whether it adds a bunch of complexity or something that I'm not thinking about!

[niklasad1]

1. The annoying thing is that error type is not cloneable and we can't really return that error on DriverStream as well as emitting it on the subscription. It shouldn't require much changes if we could just add error: Option<Error> in the shared state

I suppose we could Arc it or something to work around that though... I guess it's not super useful with that error on driver except whether it was disconnecting_will_reconnect

[jsdw]

Yep; I like your idea of the is_closed flag as the quick fix, and then maybe the next step is to Arc the error so we can share it with the subscribers, which is especially useful if the driver task is hidden away behind a helper function :)

[jsdw]

Nice!