Re-introducing --use-migration-files and fix inconsistences between ORM/migration files#62234
Re-introducing --use-migration-files and fix inconsistences between ORM/migration files#62234ephraimbuddy wants to merge 6 commits intoapache:mainfrom
--use-migration-files and fix inconsistences between ORM/migration files#62234Conversation
ephraimbuddy
requested review from
amoghrajesh,
ashb,
bugraoz93,
dheerajturaga,
gopidesupavan,
jason810496,
jscheffl,
kaxil and
potiuk
as code owners
boring-cyborg
bot
added
area:CLI
area:db-migrations
ephraimbuddy
force-pushed
the
re-introduce-use-migration-files
branch
from
86ed1f2 to
20d3a5f
Compare
ephraimbuddy
force-pushed
the
re-introduce-use-migration-files
branch
2 times, most recently
from
91cf520 to
bedbf53
Compare
ephraimbuddy
requested review from
bbovenzi,
choo121600,
guan404ming,
pierrejeambrun,
rawwar,
ryanahamilton,
shubhamraj-git and
vatsrahul1001
as code owners
| down_revision = "9d34dfc2de06" | ||
| branch_labels = None | ||
| depends_on = None | ||
| edge3_version = "3.0.2" |
There was a problem hiding this comment.
Hi @jscheffl , I found some inconsistencies in edge3's migration on the use of sa.DateTime in migration file vs UtcDateTime used in ORM. For migration file dates, we use TIMESTAMP from from airflow.migrations.db_types if UtcDateTime is used in ORM. Both are equivalent. My changes detected it and I had to add this migration file. Now, the issue I want clarification on is the edge3_version here. Is it supposed to be 3.0.2 or 3.0.0?
There was a problem hiding this comment.
Good question. We are currently on Edge3 v3.0.2 (in Pypi) and @potiuk is currently preparing a providers release cut... so this will get "under the bus" before cut. I assume the next one will be 3.1.0 - as some features piled-up.
So most probably this PR will land in next wave then... hard to foresee which version will this get. Either 3.1.1 then or... if #61646 (multi team) will land then it will for sure be a 3.2.0 (and DB migration will compete in conflict with this PR. And surprises... you know how releases work. Mostply un-predictable.
… ORM/migration files The `--use-migration-files` flag was removed in apache#41120 when we pruned the migration history for Airflow 3. At the time, we couldn't create a database from scratch using only migration files because the migration chain no longer started from an empty database. The oldest migration assumed an existing 2.6.2 schema. Now that we have a squashed migration (0000_2_6_2) with `down_revision = None` that creates the full baseline schema from scratch, we can re-introduce this flag. This enables a critical development workflow, which is: creating the database purely from migration files, then using `alembic revision --autogenerate` to detect schema drift between the ORM models and the migration-produced schema. Without this flag, `autogenerate` compares against an ORM-created database (via `create_all`), which masks any drift since both sides come from the same ORM definitions.
…between ORM/migration files
ephraimbuddy
force-pushed
the
re-introduce-use-migration-files
branch
from
46282e1 to
f2aa0c5
Compare
ephraimbuddy
force-pushed
the
re-introduce-use-migration-files
branch
from
1f0e30a to
4637686
Compare
2 participants
The
--use-migration-filesflag was removed in #41120 when we pruned the migration history for Airflow 3. At the time, we couldn't create a database from scratch using only migration files because the migration chain no longer started from an empty database. The oldest migration assumed an existing 2.6.2 schema.Now that we have a squashed migration (0000_2_6_2) with
down_revision = Nonethat creates the full baseline schema from scratch, we can re-introduce this flag. This enables a critical development workflow, which is: creating the database purely from migration files, then usingalembic revision --autogenerateto detect schema drift between the ORM models and the migration-produced schema. Without this flag,autogeneratecompares against an ORM-created database (viacreate_all), which masks any drift since both sides come from the same ORM definitions.