Migrate automatically on service start

[ConradIrwin]

This avoids a forgettable manual step in deploying collab

Release Notes:

• N/A

[maxdeviant]

Is there another way we can achieve the same thing (avoiding the need to remember to run migrations)?

I've seen issues with running migrations on server startup that led to production outages.

[maxdeviant]

Is there another way we can achieve the same thing (avoiding the need to remember to run migrations)?

I've seen issues with running migrations on server startup that led to production outages.

How I've solved this in the past is having a separate step that runs during the deployment pipeline that runs the migrations, and then only attempts the deploy when they succeed. This does require that migrations are backwards-compatible (which I believe ours are), but it means that a bad migration (e.g., one that times out) will get caught before we deploy the new version and get stuck in a failure loop when trying to run migrations when the app comes up.

I haven't familiarized myself with our deployment pipeline yet, so not sure what is easily doable.

[ConradIrwin]

I like the idea of doing this as part of a build, it would also be nice to get the logs (which this doesn't let you do).

I've also run in production with this setup quite successfully; and I think with the migration linter we added, it should be a reasonable risk trade-off: Typically the worst that happens is the new release gets stuck in a crash loop while the old one continues to serve traffic (though I haven't verified that this happens yet, I was going to do some monkeying around on staging).

I think the ideal next step is that Github Actions deploys collab automatically; so we could defactor this when we do that and do it in separate steps (but I don't like that it makes the happy path of slower).

[maxdeviant]

Typically the worst that happens is the new release gets stuck in a crash loop while the old one continues to serve traffic (though I haven't verified that this happens yet, I was going to do some monkeying around on staging).

True, I guess it does depend on the rest of our architecture, and whether services stuck in a crash loop can receive traffic (which was the case on Heroku).

[ConradIrwin]

Let's give it a go :D