Skip to content

Conversation

@potiuk
Copy link
Member

@potiuk potiuk commented Dec 30, 2023

Some recent changes, adding removed and suspended state for breeze caused significant slow-down of autocompletion retrieval - as it turned out, because we loaded and parsed all provider yaml files during auto-completion - in order to determine list of providers available for some commands.

We already planned to replace the several states (suspended, not-ready, removed) with a single state field - by doing it and addding the field to pre-commit generated "provider_dependencies.json" we could switch to parsing the single provider_dependencies.json file and retrieve provider list from there following the state stored in that json file.

This also simplifies state management following the recently added state diagram by following the same state lifecycle:

"not-ready" -> "ready" -> "suspended" -> "removed"


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

Some recent changes, adding removed and suspended state for breeze
caused significant slow-down of autocompletion retrieval - as it
turned out, because we loaded and parsed all provider yaml files
during auto-completion - in order to determine list of providers
available for some commands.

We already planned to replace the several states (suspended,
not-ready, removed) with a single state field - by doing it and
addding the field to pre-commit generated "provider_dependencies.json"
we could switch to parsing the single provider_dependencies.json
file and retrieve provider list from there following the state stored
in that json file.

This also simplifies state management following the recently
added state diagram by following the same state lifecycle:

"not-ready" -> "ready" -> "suspended" -> "removed"
@potiuk
Copy link
Member Author

potiuk commented Dec 30, 2023

This had been discussed in #36391 and it turned out that complex state management had real impact on auto-complete:

The auto-completion time went down from 1.5 s to 0.365 s

Originally most of that autocomplete time was spent in parsing yaml file

Before:

breeze

After:

breeze_new

@potiuk
Copy link
Member Author

potiuk commented Dec 30, 2023

Closing in favour of #36499 - because this change should be done in "apache" repo

@potiuk potiuk closed this Dec 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant