multi-process template

[vilgotf]

Twilight's architecture supports partitioning a bot into multiple processes, but it can be difficult to know how to. We should provide a template for the most common partition: a stateful gateway (server) and a stateless worker (client). To update commands, the worker retains support for the previous commands (it may optionally be restarted without such support once the command update has propagated). IPC is easiest done through HTTP; below is a sample gateway API:

• GET /events/{type}: gateway forwards incoming events
• GET /{type}/{id}: gateway retrieves a cached resource
• POST /shards/{id}: gateway submits a message

The events stream terminates in one of two ways:

1. Cleanly: another worker called GET /events --> shutdown
2. Uncleanly: gateway exited --> reconnect

Events in the InteractionCreate stream should be retained for 3 seconds in case another worker connects in time.

[vilgotf]

Having thought about this issue a bit more (and partly completed an implementation) I'm no longer sure this type of setup is a good idea. There are two related features that I now believe to be totally orthogonal, but which I previously thought of as dependent upon this:

1. Zero downtime restarts
2. Vertical scaling

While updates can be delivered without severing the Discord gateway connection, we still have to sever it locally: the gateway process needs to buffer and queue incoming events while the worker is offline in such a way that it requires something equivalent to Discord's gateway session and resumption. My take is, just use Discord's gateway resumption instead. The only substantial difference I can think of is latency, but I'm curious if there's another.

(Answering point 2) Splitting the bot into multiple processes seems like a required or at least a way towards being able to run the bot on multiple machines (vertical scaling). You could for instance have one machine running all the shards and then fan events out across N machines. This, however, is overcomplicated. Not only are the IPC requirements non-trivial (especially if you need synchronization, say twilight-standby integration), but you could have just partitioned the shard count across machines, with each machine running the complete bot instance. Machine 1 could run all the even shards and machine 2 all the odd ones. This scales perfectly linearly, because guild IDs are semi-random, and thus evenly spread across shards; 1/2 the shards is thus 1/2 the workload.

My current state of mind is to stop working on the multi-process template and to create a PR documenting and implementing the required scaffold for a multi-machine & shard partitioned bot. I remain open to listen to reasons for a multi-process twilight bot (you may respond in our support server as well as here).

---

With the above said, a multi-process architecture makes sense for split code bases, where the gateway process is written in Rust for performance reasons, but the event handlers are in JS/Python for better DX. In that case, gateway-proxy or twilight-dispatch might be what you're looking for. This repo's aim is to provide a 100% twilight-based template, so the previously mentioned use case is out of scope.