-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Speed up autocompletion of Breeze by simplifying provider state #36499
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Some recent changes, adding removed and suspended state for breeze caused significant slow-down of autocompletion retrieval - as it turned out, because we loaded and parsed all provider yaml files during auto-completion - in order to determine list of providers available for some commands. We already planned to replace the several states (suspended, not-ready, removed) with a single state field - by doing it and addding the field to pre-commit generated "provider_dependencies.json" we could switch to parsing the single provider_dependencies.json file and retrieve provider list from there following the state stored in that json file. This also simplifies state management following the recently added state diagram by following the same state lifecycle: "not-ready" -> "ready" -> "suspended" -> "removed"
|
This had been discussed in #36391 and it turned out that complex state management had real impact on auto-complete: The auto-completion time went down from 1.5 s to 0.365 s Originally most of that autocomplete time was spent in parsing yaml file Before: (the 70% of the graph on the left side is reading yaml files). After (this is what's left - only the right side): |
|
BTW. py-spy is great :) |
|
Merging. the Mssql failures already fixed in main |
After speeding up breeze in apache#36499 there are few small fixes needed for suspended/removed providers.
Some recent changes, adding removed and suspended state for breeze caused significant slow-down of autocompletion retrieval - as it turned out, because we loaded and parsed all provider yaml files during auto-completion - in order to determine list of providers available for some commands. We already planned to replace the several states (suspended, not-ready, removed) with a single state field - by doing it and addding the field to pre-commit generated "provider_dependencies.json" we could switch to parsing the single provider_dependencies.json file and retrieve provider list from there following the state stored in that json file. This also simplifies state management following the recently added state diagram by following the same state lifecycle: "not-ready" -> "ready" -> "suspended" -> "removed"
After speeding up breeze in #36499 there are few small fixes needed for suspended/removed providers.
After speeding up breeze in #36499 there are few small fixes needed for suspended/removed providers.
Some recent changes, adding removed and suspended state for breeze caused significant slow-down of autocompletion retrieval - as it turned out, because we loaded and parsed all provider yaml files during auto-completion - in order to determine list of providers available for some commands.
We already planned to replace the several states (suspended, not-ready, removed) with a single state field - by doing it and addding the field to pre-commit generated "provider_dependencies.json" we could switch to parsing the single provider_dependencies.json file and retrieve provider list from there following the state stored in that json file.
This also simplifies state management following the recently added state diagram by following the same state lifecycle:
"not-ready" -> "ready" -> "suspended" -> "removed"
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.