webui keeps API server (daemon) from shutting down

[torarnv]

The API server is served by braintree/manners, which keeps serving until all connections are closed.

Parts of the webui, such as polling for peers in the swarm, will open a connection and keep it open, but there doesn't seem to be any check for whether or not the server is in "shut down mode", which means the webui will keep the server alive forever.

On the ipfs daemon side there's no logging of why the server is kept alive, such as e.g. "Waiting for API connections to finish", and closing the tab with the webui in Chrome doesn't seem to immediately close the connection either, so the reason for the stuck server is quite opaque.

The only option is to forcefully close the daemon with another Ctrl+C, resulting in "Received another interrupt before graceful shutdown, terminating...", which can be a bit scary, as the user doesn't know what bad effects that might result in.

[STRML]

If the server publishes something to indicate it is closing connections, the polling could be stopped in the ui.

However this seems like a rather small problem - if the user is intentionally terminating the daemon, they likely have no need for the webui either and will close it.

This could alternatively be fixed on the API side by refusing new connections on shutdown but still completing established ones.

It appears there are other problems related to this in manners - braintree/manners#19

[torarnv]

However this seems like a rather small problem - if the user is intentionally terminating the daemon, they likely have no need for the webui either and will close it.

This assumes the user knows that the webui is what's keeping the daemon from shutting down. Without --debug there's no indication of neither what's blocking shutdown (the API server), or in turn which connections the API server is waiting on.

It's also complicated by the fact that closing the webui tab (in Google Chrome) does not seem to instantaneously close the connection, so if the user suspects the webui is at fault, the "fix" of closing the tab doesn't have any effect.

This could alternatively be fixed on the API side by refusing new connections on shutdown but still completing established ones.

This is what's happening today. New connections are refused, but the swarm-polling connection is already set up, and is kept alive by the webui, so that doesn't help.

[whyrusleeping]

Yeah, i've noticed this problem a few times before. I think we should move away from manners, as manners "key feature" is whats causing us issues. Lets just go back to using a normal go http server with a listener we can close ourselves.

[cryptix]

👍 on stdlib http server.

afaict it should be possible to also add graceful shutdown ourselves with listens to the context.

[jbenet]

fine with either 👍

[travisperson]

I think it would be best to keep a system in place that would gracefully shutdown the http interface by not accepting new connections if possible. It's just a bit nicer, but honestly I don't think it's a huge issue.

[jbenet]

I think it would be best to keep a system in place that would gracefully shutdown the http interface by not accepting new connections if possible. It's just a bit nicer, but honestly I don't think it's a huge issue.

It's a good point.

is the problem with manners that polling requests hang it open?

the right way to do this would clearly label (in the go-ipfs commands code) which requests are allowed to complete, and which are interrupted.

Wait-- shouldn't the context cancellation make them all return asap? We may be looking at the wrong end of the problem.

[whyrusleeping]

from the chrome side a polling request will not close immediately, it will stay open for a while. and manners wont exit until all its requests are finished

[jbenet]

from the chrome side a polling request will not close immediately, it will stay open for a while. and manners wont exit until all its requests are finished

we should be able to close it early though. cancelling context should (not sure that it does) cancel everything all the way, and we should stop processing request input

[torarnv]

we should be able to close it early though. cancelling context should (not sure that it does) cancel everything all the way, and we should stop processing request input

As mentioned, we currently stop accepting new connections. ipfs/kubo@6c64360b takes care of disabling keep-alive on existing connection so clients can detect the situation. What's left is dealing with those connections that still hang around, and I think drastically lowering the read timeout might solve that part, still need to play around with the code a bit.

[travisperson]

We can set a timeout on the request object returned by the ipfs-node-api, and then call abort to cancel the request. This would solve this from the webui side. I'm not sure how it would affect the UI from a polling standpoint, some work would need to be done to look at that, it might be a drop in though.

The other side would be to kill connections on the server. Do we know what requests are polling and causing this issue?

[jbenet]

Would be nice to fix this soon.