ops: Stagger deployment of migrations #7670
Conversation
If we have migrations, we can run the API deployment first since it will run the migration. If that succeeds we can deploy the workers, and they will be able to be deployed without any issues with mismatching schemas.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 2 Skipped Deployments
|
|
Opened this up for the sake of discussion @frankie567, @psincraian. As I understand it, there's not really a problem in having the workers fail for a few seconds given that they will retry, but it would feel better if we could reduce the noise during a deployment of a migration. I think this coupled with splitting migrations and code changes as much as we can will make deployments a lot less volatile. |
|
@Yopi if we want to reduce the errors and noise during DB migrations, I think the solution is to enforce an expand-contract pattern to be applied in every migration 🤔 If I'm not mistaken, this will make the errors and noise worse, as we are going to deploy changes sequentially, so the worker will try to use a removed column for longer. |
|
Yes, that is the other half of the solution. The code using new columns needs to come after the migration, and vice-versa with removed columns. If I'm not fully mistaken, the reason why we got issues yesterday is that we need to add new columns to the models for the DB migration to run, but the code with the models containing the new columns will be deployed before the migration has run. That causes the workers to fail to query data using the models. |
psincraian
left a comment
psincraian
left a comment
There was a problem hiding this comment.
Ahh, then combining the two makes total sense!
Approved for me 👍
If we have migrations, we can run the API deployment first since it will run the migration. If that succeeds we can deploy the workers, and they will be able to be deployed without any issues with mismatching schemas.