Skip to content

Conversation

@vincbeck
Copy link
Contributor

Follow-up of #46942


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added the area:API Airflow's REST/HTTP API label Feb 27, 2025
@vincbeck vincbeck added the legacy api Whether legacy API changes should be allowed in PR label Feb 27, 2025
@vincbeck vincbeck closed this Feb 27, 2025
@vincbeck vincbeck reopened this Feb 27, 2025
@vincbeck vincbeck force-pushed the vincbeck/api_connexion branch from 8e058f9 to 141df0b Compare February 27, 2025 17:15
Copy link
Member

@pierrejeambrun pierrejeambrun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we also need to update:

  • FAB provider
  • dev tools (pre-commit, selective checks, breeze release commands)
  • Documentation (release process,etc.)

@potiuk
Copy link
Member

potiuk commented Feb 27, 2025

Yeah. There is a bit more to it - I tried to do it yesterday, but I realized that this needs a bit more careful removals and in some cases some replacements :)

@potiuk
Copy link
Member

potiuk commented Feb 27, 2025

Most notably. Our python client is still built using the old v1.yaml not the new one - so while it is largely compatible, there is this one 422 error I pointed out yesterday even in our test_python_client.py, but also we need to figure out a bit more things:

  • the new openapi yaml is auto-generated so we likely need to copy it over in pre-commit
  • we thought about changing generator to a newer version (which I think it's an ideal opportunity to do so)

@pierrejeambrun -> is this also your thinking here.

@jedcunningham
Copy link
Member

It's also used in the dag commands.

@vincbeck
Copy link
Contributor Author

It's also used in the dag commands.

Yeah for now I kept this file only in api_connexion, we can resolve that later

@vincbeck vincbeck force-pushed the vincbeck/api_connexion branch from ae8f8f8 to 2db2b56 Compare February 27, 2025 19:22
Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, together with the WWW deletion PR this would have made 100k LoC deletion :-D

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we keep this file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jscheffl
Copy link
Contributor

@vincbeck Can you also move this PR to the apache/main repo such that I can push some fixes on it?
Otherwise you need to make it all yourself from MWAA team :-D

@jscheffl
Copy link
Contributor

@vincbeck Can you also move this PR to the apache/main repo such that I can push some fixes on it? Otherwise you need to make it all yourself from MWAA team :-D

As long as I can not push, this diff fixes breeze unit tests:

diff --git a/dev/breeze/tests/test_selective_checks.py b/dev/breeze/tests/test_selective_checks.py
index fe79b1bb1b..45e1d27548 100644
--- a/dev/breeze/tests/test_selective_checks.py
+++ b/dev/breeze/tests/test_selective_checks.py
@@ -149,7 +149,7 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
             pytest.param(
                 ("airflow/api/file.py",),
                 {
-                    "selected-providers-list-as-string": "amazon common.compat fab",
+                    "selected-providers-list-as-string": "amazon common.compat databricks fab",
                     "all-python-versions": "['3.9']",
                     "all-python-versions-list-as-string": "3.9",
                     "python-versions": "['3.9']",
@@ -164,8 +164,8 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
                     "mypy-docs,mypy-providers,mypy-task-sdk,ts-compile-format-lint-ui",
                     "upgrade-to-newer-dependencies": "false",
                     "core-test-types-list-as-string": "API Always",
-                    "providers-test-types-list-as-string": "Providers[amazon] Providers[common.compat,fab]",
-                    "individual-providers-test-types-list-as-string": "Providers[amazon] Providers[common.compat] Providers[fab]",
+                    "providers-test-types-list-as-string": "Providers[amazon] Providers[common.compat,databricks,fab]",
+                    "individual-providers-test-types-list-as-string": "Providers[amazon] Providers[common.compat] Providers[databricks] Providers[fab]",
                     "testable-core-integrations": "['celery', 'kerberos']",
                     "testable-providers-integrations": "['cassandra', 'drill', 'kafka', 'mongo', 'pinot', 'qdrant', 'redis', 'trino', 'ydb']",
                     "needs-mypy": "true",
@@ -314,7 +314,7 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
                     "providers/postgres/tests/unit/postgres/file.py",
                 ),
                 {
-                    "selected-providers-list-as-string": "amazon common.compat common.sql fab google openlineage "
+                    "selected-providers-list-as-string": "amazon common.compat common.sql databricks fab google openlineage "
                     "pgvector postgres",
                     "all-python-versions": "['3.9']",
                     "all-python-versions-list-as-string": "3.9",
@@ -331,9 +331,9 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
                     "upgrade-to-newer-dependencies": "false",
                     "core-test-types-list-as-string": "API Always",
                     "providers-test-types-list-as-string": "Providers[amazon] "
-                    "Providers[common.compat,common.sql,fab,openlineage,pgvector,postgres] Providers[google]",
+                    "Providers[common.compat,common.sql,databricks,fab,openlineage,pgvector,postgres] Providers[google]",
                     "individual-providers-test-types-list-as-string": "Providers[amazon] Providers[common.compat] Providers[common.sql] "
-                    "Providers[fab] Providers[google] Providers[openlineage] Providers[pgvector] "
+                    "Providers[databricks] Providers[fab] Providers[google] Providers[openlineage] Providers[pgvector] "
                     "Providers[postgres]",
                     "needs-mypy": "true",
                     "mypy-checks": "['mypy-airflow', 'mypy-providers']",

@vincbeck vincbeck force-pushed the vincbeck/api_connexion branch from 2db2b56 to 497b33a Compare February 27, 2025 20:06
@vincbeck
Copy link
Contributor Author

@vincbeck Can you also move this PR to the apache/main repo such that I can push some fixes on it? Otherwise you need to make it all yourself from MWAA team :-D

Good idea :)

@vincbeck
Copy link
Contributor Author

vincbeck commented Feb 27, 2025

Not sure how to do that tho?

@vincbeck vincbeck force-pushed the vincbeck/api_connexion branch from 497b33a to da5c94a Compare February 27, 2025 20:13
@jscheffl jscheffl mentioned this pull request Feb 27, 2025
@jscheffl
Copy link
Contributor

Not sure how to do that tho?

Here we are: #47171
You need to git branch delete-api_connexion and git push --set-upstream upstream delete-api_connexion

@vincbeck vincbeck closed this Feb 27, 2025
@pierrejeambrun
Copy link
Member

pierrejeambrun commented Feb 28, 2025

the new openapi yaml is auto-generated so we likely need to copy it over in pre-commit

I think updating the breeze release command to use the new v1-generated.yaml should do it ? I'm not sure I understand need to copy it over in pre-commit part.

we thought about changing generator to a newer version (which I think it's an ideal opportunity to do so)

+100

@vincbeck vincbeck deleted the vincbeck/api_connexion branch April 2, 2025 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API legacy api Whether legacy API changes should be allowed in PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants