-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prune the migration files for Airflow 3 #41120
Conversation
3c720ce
to
b67b0d4
Compare
6eed845
to
2762f60
Compare
2762f60
to
d5f0a4f
Compare
For a long time now, we have been able to create the metadata database from the ORM without going through the migration files. However, the migration files were not pruned because it would result in users being unable to roll back to a previous version. There's now an opportunity to prune this and start afresh. The migration files are getting more significant, and there are benefits to this pruning. Top in my mind is that the FAB migrations will no longer be in the airflow migration file, which would help in removing FAB from the Airflow core. This change also means we can no longer create a new database from the migration file. We can only do that through the ORM. I considered creating the 'current schema' in the migration file, which means having the database representation in the migration file and helping to create it from scratch. I dropped the idea because there would be elements of FAB in the migration files again. Due to this change, the db command option "--use-migration-files" was removed since this creates the database from the migration file.
d5f0a4f
to
0120154
Compare
Overall LGMT, but we need to figure out what the min version we support upgrading from should be (2.10 vs some earlier 2.x). Ephraim is going to see where the FAB things are so in the migration files so we can figure out the best balance here. |
Good point - my proposal is that migration from 2.x & should be another migration step (possibly combined with the main This way we could allow a direct migration from pre 2.10 versions as well. |
FAB-related migrations stopped at 2.6.0, so we are at liberty to still have the migration cut off way down, but I think we should cut 3.0.0 migration at 2.10 as in this PR and have a migration tool for anyone upgrading, as @potiuk suggested. |
Merging this, we can still extend it to lower versions if need be |
* Prune the migration files for Airflow 3 For a long time now, we have been able to create the metadata database from the ORM without going through the migration files. However, the migration files were not pruned because it would result in users being unable to roll back to a previous version. There's now an opportunity to prune this and start afresh. The migration files are getting more significant, and there are benefits to this pruning. Top in my mind is that the FAB migrations will no longer be in the airflow migration file, which would help in removing FAB from the Airflow core. This change also means we can no longer create a new database from the migration file. We can only do that through the ORM. I considered creating the 'current schema' in the migration file, which means having the database representation in the migration file and helping to create it from scratch. I dropped the idea because there would be elements of FAB in the migration files again. Due to this change, the db command option "--use-migration-files" was removed since this creates the database from the migration file. * fixup! Prune the migration files for Airflow 3
* Prune the migration files for Airflow 3 For a long time now, we have been able to create the metadata database from the ORM without going through the migration files. However, the migration files were not pruned because it would result in users being unable to roll back to a previous version. There's now an opportunity to prune this and start afresh. The migration files are getting more significant, and there are benefits to this pruning. Top in my mind is that the FAB migrations will no longer be in the airflow migration file, which would help in removing FAB from the Airflow core. This change also means we can no longer create a new database from the migration file. We can only do that through the ORM. I considered creating the 'current schema' in the migration file, which means having the database representation in the migration file and helping to create it from scratch. I dropped the idea because there would be elements of FAB in the migration files again. Due to this change, the db command option "--use-migration-files" was removed since this creates the database from the migration file. * fixup! Prune the migration files for Airflow 3
* Prune the migration files for Airflow 3 For a long time now, we have been able to create the metadata database from the ORM without going through the migration files. However, the migration files were not pruned because it would result in users being unable to roll back to a previous version. There's now an opportunity to prune this and start afresh. The migration files are getting more significant, and there are benefits to this pruning. Top in my mind is that the FAB migrations will no longer be in the airflow migration file, which would help in removing FAB from the Airflow core. This change also means we can no longer create a new database from the migration file. We can only do that through the ORM. I considered creating the 'current schema' in the migration file, which means having the database representation in the migration file and helping to create it from scratch. I dropped the idea because there would be elements of FAB in the migration files again. Due to this change, the db command option "--use-migration-files" was removed since this creates the database from the migration file. * fixup! Prune the migration files for Airflow 3
* Prune the migration files for Airflow 3 For a long time now, we have been able to create the metadata database from the ORM without going through the migration files. However, the migration files were not pruned because it would result in users being unable to roll back to a previous version. There's now an opportunity to prune this and start afresh. The migration files are getting more significant, and there are benefits to this pruning. Top in my mind is that the FAB migrations will no longer be in the airflow migration file, which would help in removing FAB from the Airflow core. This change also means we can no longer create a new database from the migration file. We can only do that through the ORM. I considered creating the 'current schema' in the migration file, which means having the database representation in the migration file and helping to create it from scratch. I dropped the idea because there would be elements of FAB in the migration files again. Due to this change, the db command option "--use-migration-files" was removed since this creates the database from the migration file. * fixup! Prune the migration files for Airflow 3
For a long time now, we have been able to create the metadata database from the ORM without going through the migration files. However, the migration files were not pruned because it would result in users being unable to roll back to a previous version. There's now an opportunity to prune this and start afresh. The migration files are getting more significant, and there are benefits to this pruning. Top in my mind is that the FAB migrations will no longer be in the airflow migration file, which would help in removing FAB from the Airflow core.
This change also means we can no longer create a new database from the migration file. We can only do that through the ORM. I considered creating the 'current schema' in the migration file, which means having the database representation in the migration file and helping to create it from scratch. I dropped the idea because there would be elements of FAB in the migration files again.
Due to this change, the db command option "--use-migration-files" was removed since this creates the database from the migration file.