Graceful server reloading

[shazow]

So many new features, sad to drop all connections to restart the server.

There are lots of posts about how to do this for http servers, but not sure how much more complicated it would be to do for an ssh tcp socket. Any investigating/resources appreciated.

Relevant links:

• https://github.com/rcrowley/goagain/
• https://github.com/facebookgo/grace

These seem to work by keeping two processes open until the old connections finish. This won't work for upgrading existing connections.

I'd also entertain a failover proxy frontend but I suspect that might not be possible without breaking ssh-chat into two big chunks (one of which could not be gracefully reloaded).

Another possibility is if there was some ssh protocol to request the client to reconnect.

[film42]

These might generate some ideas:

1. http://unix.stackexchange.com/questions/27636/how-does-ssh-connection-survive-a-network-restart
2. http://www.cyberciti.biz/faq/if-sshd-is-restart-will-any-connections-be-disconnected/

Edit: They're just TCP sockets at the end of the day, so handling them like an HTTP server with sockets should be the same, right?

[mrfabbri]

DISCLAIMER: very very beginner Go level here, so I may have totally misinterpreted/mystified things, still the project is neat and the problem (graceful restart with persistent connections) is interesting.

Basically, both rcrwoley/goagain and facebookgo/grace provide same way to recover Listener connections from the restarted process (via a clever use of environment variables for passing the number/file descriptors of active listener connections, and then getting back the net connection from the file descriptor).

I guess a similar approach can be replicated on the file descriptors of the active connections but will prove slightly more complicated than with just Listener connection(s) (Accept() makes a very natural point where to break/restart the service and almost no state is carried across restart):

• some state to rebuild clients (from connection file descriptors) should be saved/serialised somewhere;
• client.handleChannels() should be "restart-wise" and handle the restart gracefully (stopping at the "right" point / saving info for restart).

Well, maybe this would just turn out an "ad-hoc, informally specified, implementation of half of a failover proxy frontend".

[persontwo]

Maybe a small proxy could be written in Erlang? I dunno, I've just heard people talk about hot code reloading in Erlang.

[persontwo]

There's a thread on HN today about graceful server restart with Go: https://news.ycombinator.com/item?id=8772636. Might be of interest to you.