Script to run database migrations

[richvdh]

At the moment, if you attempt a rolling upgrade (where you restart one worker process at a time), then if there have been any database migrations, the worker processes refuse to start with:

2021-09-10 09:50:20,663 - synapse.storage.prepare_database - 116 - INFO - main - ['main', 'state']: Checking existing schema version
2021-09-10 09:50:20,666 - synapse.storage.prepare_database - 124 - INFO - main - ['main', 'state']: Existing schema is 63 (+3 deltas)
2021-09-10 09:50:20,669 - synapse.storage.databases.main - 348 - INFO - main - Checking database for consistency with configuration...
2021-09-10 09:50:20,670 - synapse.storage.prepare_database - 415 - INFO - main - Checking schema deltas for v63
2021-09-10 09:50:20,670 - synapse.app._base - 163 - ERROR - main - Exception during startup
Capture point (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/synapse/src/synapse/app/generic_worker.py", line 499, in <module>
    start(sys.argv[1:])
Traceback (most recent call last):
  File "/home/synapse/src/synapse/app/generic_worker.py", line 480, in start
    hs.setup()
  File "/home/synapse/src/synapse/server.py", line 310, in setup
    self.datastores = Databases(self.DATASTORE_CLASS, self)
  File "/home/synapse/src/synapse/storage/databases/__init__.py", line 61, in __init__
    databases=database_config.databases,
  File "/home/synapse/src/synapse/storage/prepare_database.py", line 150, in prepare_database
    databases=databases,
  File "/home/synapse/src/synapse/storage/prepare_database.py", line 506, in _upgrade_existing_database
    UNAPPLIED_DELTA_ON_WORKER_ERROR % relative_path
synapse.storage.prepare_database.PrepareDatabaseException: Database schema delta 63/02populate-rooms-creator.sql has not been applied: run the main synapse process to upgrade the database schema before starting worker processes.

It should nowadays usually be safe to run the schema migrations while the Synapse processes is still running.

We could have workers do it automatically, but I'd be a bit concerned about multiple processes all trying to do the upgrade at once. I'd also like this process to be manual until we're a bit more comfortable with it.

Accordingly, I propose a script we can run which will take a homeserver.yaml file and run the database migrations on it.

Now it turns out we have https://github.com/matrix-org/synapse/blob/v1.41.1/scripts-dev/update_database, which does almost the right thing, except that it also runs the background updates. Accordingly I think that all we need to do is modify that script so that the "run background updates" behaviour is optional.