[v3-0-test] Backport gh docs publish breeze workflows

dev/README_RELEASE_AIRFLOW.md

@@ -23,10 +23,12 @@
 - [Selecting what to put into the release](#selecting-what-to-put-into-the-release)
   - [Selecting what to cherry-pick](#selecting-what-to-cherry-pick)
   - [Making the cherry picking](#making-the-cherry-picking)
+  - [Collapse Cadwyn Migrations](#collapse-cadwyn-migrations)
   - [Reviewing cherry-picked PRs and assigning labels](#reviewing-cherry-picked-prs-and-assigning-labels)
 - [Prepare the Apache Airflow Package RC](#prepare-the-apache-airflow-package-rc)
   - [Update the milestone](#update-the-milestone)
   - [Build RC artifacts](#build-rc-artifacts)
+  - [Publish release candidate documentation (staging)](#publish-release-candidate-documentation-staging)
   - [Prepare production Docker Image RC](#prepare-production-docker-image-rc)
   - [Prepare Vote email on the Apache Airflow release candidate](#prepare-vote-email-on-the-apache-airflow-release-candidate)
 - [Verify the release candidate by PMC members](#verify-the-release-candidate-by-pmc-members)
@@ -42,8 +44,7 @@
   - [Publish release to SVN](#publish-release-to-svn)
   - [Manually prepare production Docker Image](#manually-prepare-production-docker-image)
   - [Verify production images](#verify-production-images)
-  - [Publish documentation](#publish-documentation)
-  - [Wait and make sure documentation is published on the website before proceeding](#wait-and-make-sure-documentation-is-published-on-the-website-before-proceeding)
+  - [Publish final documentation](#publish-final-documentation)
   - [Notify developers of release](#notify-developers-of-release)
   - [Send announcements about security issues fixed in the release](#send-announcements-about-security-issues-fixed-in-the-release)
   - [Add release data to Apache Committee Report Helper](#add-release-data-to-apache-committee-report-helper)
@@ -140,6 +141,15 @@ to keep a reference to the original commit we cherry picked from. ("cherry picke
 git cherry-pick <hash-commit> -x
 ```
 
+## Collapse Cadwyn Migrations
+
+Before cutting an RC, bump the HEAD date of Cadwyn versioned API (execution api for now: `airflow-core/src/airflow/api_fastapi/execution_api`)
+to reflect the tentative release date of Airflow. All the Cadwyn migrations in between the tentative release date and last release date
+should be collapsed.
+
+Refer https://github.com/apache/airflow/pull/49116 as a good example.
+
+
 ## Reviewing cherry-picked PRs and assigning labels
 
 We have the tool that allows to review cherry-picked PRs and assign the labels
@@ -331,6 +341,66 @@ pipx install -e ./dev/breeze
     --current-release ${VERSION}
     ```
 
+## Publish release candidate documentation (staging)
+
+Documentation is an essential part of the product and should be made available to users.
+In our cases, documentation for the pre-release versions is published in `staging` S3 bucket.
+The documentation source code and build tools are available in the `apache/airflow` repository, so
+you need to run several workflows to publish the documentation. More details about it can be found in
+[Docs README](../docs/README.md) showing the architecture and workflows including manual workflows for
+emergency cases.
+
+We have two options publishing the documentation 1. Using breeze commands 2. Manually using GitHub Actions.:
+
+### Using breeze commands
+
+You can use the `breeze` command to publish the documentation.
+The command does the following:
+
+1. Triggers [Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml).
+2. Triggers workflow in apache/airflow-site to refresh
+3. Triggers S3 to GitHub Sync
+
+```shell script
+  breeze workflow-run publish-docs --ref <tag> --site-env <staging/live/auto>
+```
+
+The `--ref` parameter should be the tag of the release candidate you are publishing.
+
+The `--site-env` parameter should be set to `staging` for pre-release versions or `live` for final releases. the default option is `auto`
+if the tag is rc it publishes to `staging` bucket, otherwise it publishes to `live` bucket.
+
+Other available parameters can be found with:
+
+```shell script
+breeze workflow-run publish-docs --help
+```
+
+### Manually using GitHub Actions
+
+There are two steps to publish the documentation:
+
+1. Publish the documentation to the `staging` S3 bucket.
+
+The release manager publishes the documentation using GitHub Actions workflow
+[Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). By default `auto` selection should publish to the `staging` bucket - based on
+the tag you use - pre-release tags go to staging. But you can also override it and specify the destination
+manually to be `live` or `staging`.
+
+You should specify 'apache-airflow docker-stack' passed as packages to be
+built.
+
+After that step, the provider documentation should be available under https://airflow.stage.apache.org//
+URL (RC PyPI packages are build with the staging urls) but stable links and drop-down boxes are not updated yet.
+
+2. Invalidate Fastly cache, update version drop-down and stable links with the new versions of the documentation.
+
+In order to do it, you need to run the [Build docs](https://github.com/apache/airflow-site/actions/workflows/build.yml)
+workflow in `airflow-site` repository - but make sure to use `staging` branch.
+
+After that workflow completes, the new version should be available in the drop-down list and stable links
+should be updated, also Fastly cache will be updated
+
 ## Prepare production Docker Image RC
 
 Production Docker images should be manually prepared and pushed by the release manager or another committer
@@ -816,57 +886,65 @@ docker pull apache/airflow:${VERSION}
 breeze prod-image verify --image-name apache/airflow:${VERSION}
 ```
 
-## Publish documentation
+## Publish final documentation
 
 Documentation is an essential part of the product and should be made available to users.
-In our cases, documentation for the released versions is published in a separate repository - [`apache/airflow-site`](https://github.com/apache/airflow-site), but the documentation source code and build tools are available in the `apache/airflow` repository, so you have to coordinate between the two repositories to be able to build the documentation.
+In our cases, documentation for the released versions is published in the `live` S3 bucket, and the site is
+kept in a separate repository - [`apache/airflow-site`](https://github.com/apache/airflow-site),
+but the documentation source code and build tools are available in the `apache/airflow` repository, so
+you need to run several workflows to publish the documentation. More details about it can be found in
+[Docs README](../docs/README.md) showing the architecture and workflows including manual workflows for
+emergency cases.
 
-Documentation for providers can be found in the ``/docs/apache-airflow`` directory.
+We have two options publishing the documentation 1. Using breeze commands 2. Manually using GitHub Actions.:
 
-- First, copy the airflow-site repository and set the environment variable ``AIRFLOW_SITE_DIRECTORY``.
+### Using breeze commands
 
-    ```shell script
-    git clone https://github.com/apache/airflow-site.git airflow-site
-    cd airflow-site
-    git checkout -b ${VERSION}-docs
-    export AIRFLOW_SITE_DIRECTORY="$(pwd)"
-    ```
+You can use the `breeze` command to publish the documentation.
+The command does the following:
 
-- Then you can go to the directory and build the necessary documentation packages
+1. Triggers [Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml).
+2. Triggers workflow in apache/airflow-site to refresh
+3. Triggers S3 to GitHub Sync
 
-    ```shell script
-    cd "${AIRFLOW_REPO_ROOT}"
-    breeze build-docs apache-airflow docker-stack --clean-build
-    ```
+```shell script
+  breeze workflow-run publish-docs --ref <tag> --site-env <staging/live/auto>
+```
 
-- Now you can preview the documentation.
+The `--ref` parameter should be the tag of the final candidate you are publishing.
 
-    ```shell script
-    ./docs/start_doc_server.sh
-    ```
+The `--site-env` parameter should be set to `staging` for pre-release versions or `live` for final releases. the default option is `auto`
+if the tag is rc it publishes to `staging` bucket, otherwise it publishes to `live` bucket.
 
-- Copy the documentation to the ``airflow-site`` repository, create commit, push changes, open a PR and merge it when the build is green.
+Other available parameters can be found with:
 
-    ```shell script
-    breeze release-management publish-docs apache-airflow docker-stack
-    breeze release-management add-back-references apache-airflow --airflow-site-directory "${AIRFLOW_SITE_DIRECTORY}"
-    breeze sbom update-sbom-information --airflow-version ${VERSION} --airflow-site-directory ${AIRFLOW_SITE_DIRECTORY} --force --all-combinations --run-in-parallel
-    cd "${AIRFLOW_SITE_DIRECTORY}"
-    git add .
-    git commit -m "Add documentation for Apache Airflow ${VERSION}"
-    git push
-    # and finally open a PR
-    ```
+```shell
+breeze workflow-run publish-docs --help
+```
+
+### Manually using GitHub Actions
+
+There are two steps to publish the documentation:
+
+1. Publish the documentation to S3 bucket.
+
+The release manager publishes the documentation using GitHub Actions workflow
+[Publish Docs to S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). By
+default `auto` selection should publish to the `live` bucket - based on
+the tag you use - pre-release tags go to staging. But you can also override it and specify the destination
+manually to be `live` or `staging`.
 
-The `--run-in-parallel` switch allows to speed up SBOM generation significantly, but it might take a lot
-of memory - if you are running into memory issues you can limit parallelism by setting `--parallelism N`
-where N is a number of parallel `cdxgen` servers that should be started.
+After that step, the provider documentation should be available under the https://airflow.apache.org/ URL
+(also linked directly from the PyPI packages) but stable links and drop-down boxes should not be yet updated.
+That allows the Release Manager to verify if the documentation is published.
 
-## Wait and make sure documentation is published on the website before proceeding
+2. Invalidate Fastly Cache, update version drop-down and stable links with the new versions of the documentation.
 
-This is important as it takes time for the documentation to be published. You should exercise some patient
-here before proceeding with the next steps.
+In order to do it, you need to run the [Build docs](https://github.com/apache/airflow-site/actions/workflows/build.yml)
+workflow in `airflow-site` repository. Make sure to use `main` as the branch to run the workflow.
 
+After that workflow completes, the new version should be available in the drop-down list and stable links
+should be updated, also Fastly cache will be invalidated.
 
 ## Notify developers of release
 
@@ -1007,7 +1085,7 @@ This includes:
 - For major/minor release, update version in `airflow/__init__.py` and `docs/docker-stack/` to the next likely minor version release.
 - Sync `RELEASE_NOTES.rst` (including deleting relevant `newsfragments`) and `README.md` changes.
 - Updating `Dockerfile` with the new version.
-- Updating `airflow_bug_report.yml` issue template in `.github/ISSUE_TEMPLATE/` with the new version.
+- Updating `1-airflow_bug_report.yml` issue template in `.github/ISSUE_TEMPLATE/` with the new version.
 
 ## Update default Airflow version in the helm chart
 
@@ -1046,7 +1124,7 @@ Clients can be found here:
 ### API Clients versioning policy
 
 Clients and Core versioning are completely decoupled. Clients also follow SemVer and are updated when core introduce changes relevant to the clients.
-Most of the time, if the [openapi specification](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/api_fastapi/core_api/openapi/v1-rest-api-generated.yaml) has
+Most of the time, if the [openapi specification](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/api_fastapi/core_api/openapi/v2-rest-api-generated.yaml) has
 changed, clients need to be released.
 
 To determine if you should release API clients, you can run from the airflow repository: