Synapse gets stuck after seeing psycopg2.errors.ReadOnlySqlTransaction: cannot execute nextval() in a read-only transaction

[matrixbot]

This issue has been migrated from #16490.

---

https://github.com/matrix-org/synapse/blob/166ffc0f23419bc99d9597fe95deaae3bbee7caf/synapse/rest/health.py#L29-L31

Hi :)

It seems like the health endpoint is a little too simple. For example, I am getting with patroni (pg cluster) sometimes synapse in a state where it gets stuck in a read-only transaction state psycopg2.errors.ReadOnlySqlTransaction: cannot execute nextval() in a read-only transaction. Synapse never recovers until a restart from this, even when the database is already healthy for hours again. Meanwhile, however, my Kubernetes which could have easily restarted synapse by now thinks the server is healthy while all api endpoints fail.

Would it be possible to include fatal errors from the database in the evaluation of the health endpoint? Or are there other suggestions on how to solve this for someone hosting a synapse server?