Skip to content

Conversation

@potiuk
Copy link
Member

@potiuk potiuk commented Dec 30, 2023

Some recent changes, adding removed and suspended state for breeze caused significant slow-down of autocompletion retrieval - as it turned out, because we loaded and parsed all provider yaml files during auto-completion - in order to determine list of providers available for some commands.

We already planned to replace the several states (suspended, not-ready, removed) with a single state field - by doing it and addding the field to pre-commit generated "provider_dependencies.json" we could switch to parsing the single provider_dependencies.json file and retrieve provider list from there following the state stored in that json file.

This also simplifies state management following the recently added state diagram by following the same state lifecycle:

"not-ready" -> "ready" -> "suspended" -> "removed"


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

Some recent changes, adding removed and suspended state for breeze
caused significant slow-down of autocompletion retrieval - as it
turned out, because we loaded and parsed all provider yaml files
during auto-completion - in order to determine list of providers
available for some commands.

We already planned to replace the several states (suspended,
not-ready, removed) with a single state field - by doing it and
addding the field to pre-commit generated "provider_dependencies.json"
we could switch to parsing the single provider_dependencies.json
file and retrieve provider list from there following the state stored
in that json file.

This also simplifies state management following the recently
added state diagram by following the same state lifecycle:

"not-ready" -> "ready" -> "suspended" -> "removed"
@potiuk
Copy link
Member Author

potiuk commented Dec 30, 2023

This had been discussed in #36391 and it turned out that complex state management had real impact on auto-complete:

The auto-completion time went down from 1.5 s to 0.365 s

Originally most of that autocomplete time was spent in parsing yaml file

Before: (the 70% of the graph on the left side is reading yaml files).

breeze

After (this is what's left - only the right side):

breeze_new

@potiuk
Copy link
Member Author

potiuk commented Dec 30, 2023

BTW. py-spy is great :)

@potiuk
Copy link
Member Author

potiuk commented Dec 30, 2023

Merging. the Mssql failures already fixed in main

@potiuk potiuk merged commit 6937ae7 into main Dec 30, 2023
potiuk added a commit to potiuk/airflow that referenced this pull request Dec 30, 2023
After speeding up breeze in apache#36499 there are few small fixes needed
for suspended/removed providers.
potiuk added a commit that referenced this pull request Dec 30, 2023
Some recent changes, adding removed and suspended state for breeze
caused significant slow-down of autocompletion retrieval - as it
turned out, because we loaded and parsed all provider yaml files
during auto-completion - in order to determine list of providers
available for some commands.

We already planned to replace the several states (suspended,
not-ready, removed) with a single state field - by doing it and
addding the field to pre-commit generated "provider_dependencies.json"
we could switch to parsing the single provider_dependencies.json
file and retrieve provider list from there following the state stored
in that json file.

This also simplifies state management following the recently
added state diagram by following the same state lifecycle:

"not-ready" -> "ready" -> "suspended" -> "removed"
potiuk added a commit that referenced this pull request Dec 30, 2023
After speeding up breeze in #36499 there are few small fixes needed
for suspended/removed providers.
potiuk added a commit that referenced this pull request Dec 30, 2023
After speeding up breeze in #36499 there are few small fixes needed
for suspended/removed providers.
@potiuk potiuk added this to the Airflow 2.8.1 milestone Dec 30, 2023
@potiuk potiuk added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Dec 30, 2023
@potiuk potiuk deleted the simplify-provider-state branch October 1, 2024 08:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants