ensure system tablets are alive when an entire dc is in a crash-loop

[vporyadke]

System tablets (specifically, coordinators and mediators) are kept in one datacenter when possible. Assume we are deploying a new problematic version, and at first rolled it out on one datacenter, causing all nodes there to become stuck in a crash-loop. Then, system tablets remain in this DC, are not working, and the entire database is effectively down, despite only one location having problems. This issue is for improving the behavior in this case.

• Run simulations to see whether the "stick system tablets together" functionality is the cause of the problems, or if it would be the same with it turned off.
• Make a fix based on the simulations

[vporyadke]

Simulations:

As expected, most other tablets leave the bad DC, while system tablets remain there.

If we simply turn off the "stick together" option, they quickly move to other DCs too.

[vporyadke]

This means that to solve the issue, "stick together" functionality needs to be adjusted.

How it currently works

1. First, dc preference/stick together is applied - that means only the nodes from this DC are considered, if there are any of them alive and able to run tablets.
2. Then, among them, only nodes with maximum "priority" are chosen. Priority is provided by the node (a feature only used by NBS/NFS), and is decreased if the tablet just died on this same node.
3. Then, among max priority nodes a choice algo is run.

How to solve this?

Option 1: keep the steps the same

This can be done if we consider nodes that "restart often" to be not able to schedule tablets. Cons:

• Depends on threshold for "often" - in the incident that inspired this issue, the alert for "often restarting" triggered 3 minutes after the SLO alert - but still earlier than the problem was manually solved
• If every single node restarts often, under this approach we will stop launching tablets entirely

Option 2: merge steps 1 and 2

Use DC preference not as a hard constraint, but as an increase in priority AND decrease priority for restarting nodes

• Lots of room for tuning! How much do we increase for DC, and how much do we decrease per restart?
• Interacts with dc preference and priorities - could this have unwanted consequences there? (Need to check if anyone even uses dc preference at this point)

I prefer option 2, and would try a simple formula like +1 priority for DC, -(RestartsPerPeriod) / 3 for restarts penalty.