Merge the per-peer daemons into a single peerd

[cdecker]

As the protocol becomes ever more complex the common parts that need
to be handled by the various per-peer daemons (openingd, channeld,
closingd, potentially even onchaind) forces us to duplicate
functionality quiet a bit. In addition, as the phases become less
well-separated, for dual-funding, RBF funding support, splicing, etc,
we need to move existing functionality between daemons (see #4293 and
#4271 for examples), leading to rather large overheads when adding
support for these.

One solution is to merge the per-peer daemons into a single peerd
that gets started when first connecting, and stopped when we lose the
connection (onchaind being the one exception that could be kept
separate).

Advantages

• We maintain the separation between lightningd and anything that
  touches a user-connection (unless we merge connectd back into
  lightningd as well, in which case we'd be execveing peerd
  with the raw connection before exchanging any data).

• peerd has full control of the crypto_state and the connection,
  which means lightningd never has to touch it, and we don't need
  to fd-pass the peer connection.

  • This in turn makes Windows support much easier, since the only
    FDs we now pass are from hsmd and gossipd which can be
    inherited from lightningd. New connections can be created in
    lightningd and muxed onto the single connection it now keeps
    with hsmd, removing the need for fd-passing altogether.

• Simplifies debugging, since the daemons are longer lived and
  debuggers don't have to re-attach to new processes as we progress
  through the states.

Disadvantages

• Daemons become more complex, and all have to switch to async
  communication

• Breaks all pending changes during the refactoring period.

• Weakens the hsmd capabilities system since all peer daemons share
  the same connection with a superset of all currently needed
  capabilities.

Bottom-up approach

We can extract common functionality into common/ and slowly start
converging to a common internal representation, then start migrating
individual daemons to peerd. This is the least disruptive, but also
the long approach.

Top-down approach

A quick and dirty approach to the problem: create peerd, rename
main from the various daemons to <daemon>_main, create links to
peerd from the old daemon name, then call <daemon>_main based on
the executable name. Then start merging everything.

Over time we start differentiating daemons via the first message they
receive (usually an init message of some kind), then add daemon
transitions without stopping and starting the daemon, and finally
integrate all io_loops into a single loop.

[rustyrussell]

We still need connectd, which passes fds back once we've completed the handshake.

Another possibility would be to have connectd maintain the connection, and do all the crypto, forwarding to/from peerd. This opens the door to connectd demuxing messages, and sending appropriate things to/from gossipd. Then it's fairly easy to support multiple channels per peer?

[cdecker]

Interesting idea, however I prefer having the peer-connection isolated into its own process, rather than encapsulating the logic and keep the connections in a single daemon. What was the reason to split out connectd again? I was implicitly assuming we'd merge connectd into lightningd, which makes the fd-inheritance a reasonable option. I'd rather proxy the hsmd and gossipd connections through lightningd, since they tend to be less chatty and being able to intercept and serialize them in lightningd may give us additional insights while we're at it :-)

[ZmnSCPxj]

Passing FDs around are certainly a pain point when considering Windows, and in that regard anything that removes FD-passing completely would be a plus. On the other hand we could just use sockets and special Windows-specific APIs for passing HANDLEs around (I would like to specifically deny that I know anything about the Windows API, and no, DWORD does not mean 32-bit unsigned integer).

I proposed, long ago, a similar idea, however:

• My proposal would retain the existing channeld/openingd/closingd daemons; the peerd would simply multiplex messges, between gossipd, and specific channel-level daemon.
  • This has the advantage that the existing synchronously-coded daemons could be retained synchronously-coded; only the peerd itself, as a mutliplexsing switch, absolutely needed to be asynchronously coded, but since it was only a switch, it would not be very complex.
  • Basically, for example the openingd would have a socket with the lightningd, and a socket with the peerd, and the code for openingd would act as if the socket with peerd was the actual peer socket, except (a) the messages are sent in the clear and (b) it will never send or receive gossip messges.
  • As noted as well it means the crypto state does not need to be serialized and passed around.
• The lightningd would listen to the bound port, and as soon as it accepted a socket, would launch a peerd that would later launch specific openingd/channeld/closingd as appropriate. Until the handshake completed the peerd could be coded synchronously.

[cdecker]

Passing FDs around are certainly a pain point when considering Windows, and in that regard anything that removes FD-passing completely would be a plus. On the other hand we could just use sockets and special Windows-specific APIs for passing HANDLEs around (I would like to specifically deny that I know anything about the Windows API, and no, DWORD does not mean 32-bit unsigned integer).

Indeed there seem to be posix equivalents in Windows, but we have no experts at hand that could write the shims necessary, so removing the requirement seems to be the easiest way to get Windows support.

• My proposal would retain the existing channeld/openingd/closingd daemons; the peerd would simply multiplex messges, between gossipd, and specific channel-level daemon.

Notice that my primary goal isn't to eliminate FD-passing, but rather remove the duplicated message handling code in each daemon (where each daemon is implemented just slightly differently), and reflect the reduction in specialization (due to dual-funding, splicing, etc), because the phases of a channel are not as clearly separated as we thought initially.

This proposal surfaced again due to the gargantuan #4293 which moved existing functionality between two daemons since the cutoff between the phases shifted slightly, and resulted in a 91 commit PR!