Add feture of "not-ready" provider.

[potiuk]

This PR adds possibility of marking the provider as "not ready" in the provider.yaml (by setting optional field as "not-ready" to `true".

Setting provider as "not-ready", removes it by default from all the release management commands - preparing documentation files preparing provider packages, publishing docs.

You can include such providers via --include-not-ready-providers flag (or setting INCLUDE_NOT_READY_PROVIDERS environment variable to true).

This flag is set to True in our CI, so that we can make sure the providers in-progress are also being tested and verified, but when release manager prepares packages, those providers are not prepared.

That will help in early stage of a lifecycle of a provider when we already want to iterate and test it continuously, but - for example the API of such provider is not yet stable or when we are in progress of moving functionality for such provider from core.

This PR also marks fab providers as "not-ready" as it is still early days and we want to exclude it for now from any kind of release process.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

CC: @bolkedebruin - respnding to some of the needs we had in the past. In the future things like "common.io" where the API is not ready to be released yet could be marked as "not-ready". I already marked "fab" provider as "not-ready" @vincbeck

CC: @eladkal

This is the way how we can exclude some providers from being released in the "upcoming" wave of providers. The nice thing is that we should be able to mark provider as "not-ready" even if it is already released, but - for-example- we are in progress of doing something and we think we should hold back with releasing the provider.

[potiuk]

I also added a chapter in the Provider release documentation about all possible states of provider packages. Now we have 4:

• default
• not-ready
• suspended
• removed

And I thought it might be worthwile to capture the differences between those 4 states in a single place. Release documentation seems like a good place for it.

[Taragolis]

And I thought it might be worthwile to capture the differences between those 4 states in a single place. Release documentation seems like a good place for it.

I think it also would be nice to split into different sections list of providers as follow up in https://airflow.apache.org/docs/#providers-packagesdocsapache-airflow-providersindexhtml

At least move removed/suspended into the separate list

[potiuk]

I think it also would be nice to split into different sections list of providers as follow up in https://airflow.apache.org/docs/#providers-packagesdocsapache-airflow-providersindexhtml

At least move removed/suspended into the separate list

Good point. That list is maintained separately - because it is part of the airlfow-site repository. And I think we should just remember to clean-it-up after we suspend/remove providers (getting the list of suspended/removed providers in this repo might be a bit more complex, so likely manual synchronization is better).

Let me add a separate PR for that in the release management process.

[Taragolis]

I think we might be close to replace all states attributes by a single as enum (as follow up), e.g.:

"state": {
        "type": "string",
        "enum": ["ready", "not-ready", "suspended", "removed"]
      },

I think all states are mutually exclusive and lifecycle for abstract package would be

graph TD;
    not-ready-->ready;
    ready-->not-ready;
    ready-->suspended;
    ready-->removed;
    suspended-->removed;

Loading

[potiuk]

Yes. We could do it this way as well. I will leave that exercise to you :)

[hussein-awala]

It's a good idea

[potiuk]

BTW. Here is the PR about state updating by Release Manager #36392

[potiuk]

BTW. I like the diagram and added it to the docs now. But I fixed it. There is one more transition possible - ready -> not-ready. The way we have it defined now, we can mark a package as "not-ready" even if it has been previously released and it will be all-fine with tests and ci - but it will be skipped by default when releasing new wave.

[potiuk]

Looks a bit different now :)

[potiuk]

That is a bit TOO much though 👎

[potiuk]

How about this ?

[potiuk]

Ok. This one seems to be nicest:

[potiuk]

Or this for symmetry

[hussein-awala]

Looks good

[vincbeck]

Quick question: why marking fab provider as not ready? To me, the code migration is done and the only remaining piece is #36232 but no breaking change is expected now onwards in fab provider.

[potiuk]

Quick question: why marking fab provider as not ready? To me, the code migration is done and the only remaining piece is #36232 but no breaking change is expected now onwards in fab provider.

"Not ready" is only used to skip the package from being released. The thing is that we do not want to release FAB provider just yet. And when we prepare providers for release we prepare all packages that are eligible to be released. But it makes no sense to relase FAB yet - as we might find bugs and issues that make such relased package not really suitable and we will have to yank them. We had somewhat similar problem with common.io which we released quite a bit too early as it had some breaking changes that made them unusable.

Result:

I think we should removre the "not-ready" flag some two weeks before Airflow 2.9 alpha/beta relase, to give us enough time to relase it in the regular provider's wave a week/two before Airflow needs it.

[vincbeck]

I see, thanks for the explanations, it makes sense to me

[potiuk]

BTW. It's been improved since and it's easier to manage the state now with single state field in provider.yaml :D