RFC: Sofie high availability

[Julusian]

About Me

This RFC is posted on behalf of NRK

Use Case

Given Sofie's role of producing broadcasts, downtime during the show is not acceptable.

When running in docker in a VM, downtime can usually be limited to maintenance windows. The occasional vm live migration can happen (perhaps going unnoticed?) and occasional restart/crash of docker containers is usually perceived as system slowness, taking usually a couple of seconds to recover.
The current abilities of Sofie are typically happy here, this is what the current system design was targetting.

When running in kubernetes, the system wants to be able to restart running containers freely. This is not good with sofie, as a restart of a container can often take 30s to come back up, which will cause shows to suffer.
Disallowing this is possible but can have side effects such as blocking the cluster from being able to perform maintenance on the kubernetes nodes.
Some system design work is needed to be more kuberenetes friendly

Proposal

This is not a concrete proposal, but intended to be a place to collect ideas. There are no plans on when/if we will tackle anything on this, but we hope that this RFC will allow the community to formulate a plan on how this should be tackled and allow for it to be gradually tackled by anyone who needs it.

The needed solution is to be able to run replicas of every portion of sofie:

• sofie-core
• job-worker
• playout gateway
• ingest gateway (optional?)
• input gateway
• package manager?
• live status gateway

Whether this should be done as primary+spare, load balanced or something else should be figured out

---

Another of the underlying problems is that within sofie-core there are a few debounces/caches kept in memory that would cause inconsistencies if there were multiple instances running. There has been work to minimise the number of these, but some remain.

Process

The Sofie Team will evaluate this RFC and open up a discussion about it, usually within a week.

• RFC created
• Sofie Team has evaluated the RFC
• A workshop has been planned
• RFC has been discussed in a workshop
• A conclusion has been reached, see comments in thread

[Julusian]

This comment is my Superfly/personal thoughts.
I have not done any research into how other systems handle this, this is a dump of my immediate thoughts.

I see a few challenges that need solving:

1) gateway DDP connections

Each gateway opens a DDP connection to sofie-core. This means that when the sofie-core they are connected to restarts, they will be disrupted until they are able to reconnect. This wont be 'free', as the subscriptions the gateway is running will likely need recalculating and will resend all subscribed data.

Should each gateway be connecting to 2 sofie-core instances at the same time (in a primary+spare strategy) so that when the primary goes down it can seamlessly recover?
Or perhaps some inspiration from trpc where it sends a 'reconnect' message when shutting down (the expectation to be that the ingress would route the new connection to a still alive sofie-core)
Or maybe I am worrying too much, and we think that the reconnection will be quick and cheap enough that this isnt worth worrying about.

I don't think that replacing DDP with something else is necessarily the solution here (although it might make the implementation easier to achieve). Other protocols will have similar issues.

2) Limiting one replica to be active

The job-worker was written to be the sole runner for a studio. Playout-gateway will have a similar problem of that only one of the replicas should ever be controlling devices at a time.

So as we will want replicas of these to allow for quick handover/failover, they will want to be aware of if they are 'active' or 'inactive'.
In 'active' mode, they perform as today. In 'inactive' mode, they should subscribe to data and maintain state, ready to be promoted to 'active' at any time.

How do we decide which is which? For the gateways, that could be coordinated by sofie-core that they are connected to over ddp?

This is inspired by how the Stream Deck Studio and Stream Deck Network Dock behave, their protocol has a similar primary/secondary connection flag to allow for this failover.

The exact interpretation of this active/inactive flag may vary depending on the process.
Some connections in playout-gateway may want to be disabled, as the device may not allow any/enough concurrent connections (eg old hyperdecks)
Input gateway talking to tcp stream decks could run normally, as each stream deck connection reports a 'primary'/'secondary' connection flag, so this could be utilised to determine which replica is the active one.

3. Running job-worker as its own process

From the beginning of the job-worker, it was written to be fed jobs via some work queue. As it was not needed to be deployed as its own container, this has not been fully realised with the job queue instead being kept in memory of the parent sofie-core.

Some work queue system needs to be chosen, and the job-worker and sofie-core plumbed in to use this.

It might be desirable to retain the current approach to allow for simpler deployments if HA is not required (or for development).

3) replication strategies

I think that some cotnainers want different replication strategies depending on what they are.

If the job-worker is split out, then sofie-core is a fairly normal web app, and can be horizontally scaled easily. Although this does have side effects (the gateway ddp connections).
job-worker, playout-gateway and a few others will want an approach where only one isntance can be active at a time, with a running spare.
some ingest gateways may not want to have a running spare, they may simply not operate correctly with a NRCS and are perhaps less critical to have perfect uptime.
live status gateway is likely fine to replicate today, other than the general issues elsewhere. If put behind some loadbalancer then it will have failover. It may want to gain a similar 'reconnect' message to help its clients recover seamlessly. Perhaps it should be thought about whether LSG should be retrieving all its data over ddp, or if it should be connecting directly to mongodb.

If running any replicas of a gateway, some work will likely be needed so that sofie-core doesn't get confused over having multiple gateways connected with the same id.

4) monitoring process health

With this talk of primary+spare setups, there will need to be some thought put into how to ensure they stay in sync on which is which.

Additionally, kubernetes relies on ready and health probes. These will need to be implemented widely, and some thought will need to go into what the correct implementations are so that kubernetes understands what is active or ready to receive ddp connections.

[jstarpl]

Input gateway talking to tcp stream decks could run normally, as each stream deck connection reports a 'primary'/'secondary' connection flag, so this could be utilised to determine which replica is the active one.

Could you elaborate some more on how this works in Stream Deck TCP protocol? Is this only a flag that is set only on connection, or is this a flag + some sort of control message to change that flag on an already opened connection?

I feel that Stream Decks will be somewhat of a rarity here, I think we need to assume that we need to do the hand-over ourselves, since this doesn't seem to me to be a particularly common protocol feature?

It might be desirable to retain the current approach to allow for simpler deployments if HA is not required (or for development).

I would definitely second an option for a simple, built-in work queue option. Sofie is already difficult to set up for development as it is.

[Julusian]

Could you elaborate some more on how this works in Stream Deck TCP protocol? Is this only a flag that is set only on connection, or is this a flag + some sort of control message to change that flag on an already opened connection?

I can't point to anything right now as I didn't implement it in the library yet. But yes, it reports the number upon connection and will send an update to existing connections. So a connection can start as secondary, and become primary later on and be told about it.
Of course we don't have to use this, and whichever instance we declare as the primary/active one could do all the talking instead.

I feel that Stream Decks will be somewhat of a rarity here, I think we need to assume that we need to do the hand-over ourselves, since this doesn't seem to me to be a particularly common protocol feature?

Yes it might be a rarity, but is also maybe something to handle in a special way.
I suppose that incoming http webhooks could (if being routed via kubernetes ingress) also dont need to hit the primary.
But I haven't put much thought into specifics here, just general bigger ideas.

[PeterC89]

Assuming we required Sofie to be deployed in Kubernetes to achieve HA we could look at something like leases to establish a locking mechanism for components.

I have more thoughts but will take some time to collate them properly, it would be nice to think that we could run any number of replicas of components, although there's nothing wrong with a main / reserve pattern.