Skip to content

Start working on the docs for the Task SDK #47357

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ashb wants to merge 1 commit into from
Closed

Start working on the docs for the Task SDK #47357

ashb wants to merge 1 commit into from

Conversation

@ashb
Copy link
Member

@ashb ashb commented Mar 4, 2025 •
edited
Loading

In order to have a nicer experience (i.e. rather than just being given an
alphabetical list of classes/functions) I have chosen to not use the auto
generate feature of the AutoAPI extension, but instead to precisely control
the order and grouping of the classes.

For this to be complete we will likely need some testing that compares the
items on the generated objects.inv with the things we re-export from
airflow/sdk/__init__.py.

The airflow/sdk/__init__.pyi type stub is sadly needed, as without it (and
with the imports only inside the if TYPE_CHECKING block) we weren't getting
the re-exported classes showing up where we want them, specifically we want
users to import things directly from airflow.sdk, not the submodule.

It was generated with stubgen -o task_sdk/src -m airflow.sdk

Test this with sphinx-build -b html -Tv task_sdk/docs/ task_sdk/docs/_build

Closes #43010

Here's what it looks like right now:

Screenshot 2025-03-04 at 18 00 34

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@kaxil kaxil moved this from Next 2 Weeks to In Progress in AIP-72 - Task Execution Interface and SDK Apr 9, 2025
@kaxil kaxil force-pushed the prototype-task-sdk-docs branch from 6228746 to 1a25351 Compare April 9, 2025 15:13
@kaxil kaxil added affected_version:3.0.0rc priority:medium Bug that should be fixed before next release but would not block a release labels Apr 9, 2025
@kaxil kaxil force-pushed the prototype-task-sdk-docs branch from 1a25351 to 2957754 Compare April 9, 2025 15:17
@ashb @kaxil
Start working on the docs for the Task SDK
a02aade 
In order to have a nicer experience (i.e. rather than just being given an
alphabetical list of classes/functions) I have chosen to not use the auto
generate feature of the AutoAPI extension, but instead to precisely control
the order and grouping of the classes.

For this to be complete we will likely need some testing that compares the
items on the generated `objects.inv` with the things we re-export from
`airflow/sdk/__init__.py`.

The `airflow/sdk/__init__.pyi` type stub is sadly needed, as without it (and
with the imports only inside the `if TYPE_CHECKING` block) we weren't getting
the re-exported classes showing up where we want them, specifically we want
users to import things directly from airflow.sdk, not the submodule.

It was generated with `stubgen -o task_sdk/src -m airflow.sdk`

Test this with `sphinx-build -b html -Tv task_sdk/docs/ task_sdk/docs/_build`
@kaxil kaxil force-pushed the prototype-task-sdk-docs branch from 2957754 to a02aade Compare April 9, 2025 15:19
@kaxil kaxil added priority:high High priority bug that should be patched quickly but does not require immediate new release and removed priority:medium Bug that should be fixed before next release but would not block a release labels Apr 9, 2025
@kaxil kaxil mentioned this pull request Apr 11, 2025
Merged
@kaxil kaxil added this to the Airflow 3.1+ milestone Apr 16, 2025
@kaxil kaxil added priority:medium Bug that should be fixed before next release but would not block a release and removed priority:high High priority bug that should be patched quickly but does not require immediate new release labels Apr 16, 2025
@eladkal eladkal modified the milestones: Airflow 3.1+, Airflow 3.0.1 Apr 17, 2025
@cmarteepants cmarteepants mentioned this pull request Apr 23, 2025
Closed
@kaxil kaxil modified the milestones: Airflow 3.0.1, Airflow 3.0.2 May 6, 2025
@sunank200 sunank200 self-assigned this May 28, 2025
@sunank200 sunank200 mentioned this pull request May 28, 2025
Merged
@phanikumv
Copy link
Contributor

phanikumv commented Jun 2, 2025

We are handling this in #51153

@phanikumv phanikumv closed this Jun 2, 2025
kaxil pushed a commit that referenced this pull request Jun 11, 2025
@sunank200
Docs setup for the Task SDK (#51153)
ebfc0de 
This PR is a continuation of #47357

To enhance the experience (i.e., rather than just being given an alphabetical list of classes/functions), I have opted not to use the AutoAPI extension's auto-generate feature, allowing me to precisely control the order and grouping of the classes.

I have organized Task-SDK API into these sections:
* **Defining DAGs**: `DAG`, `dag`
* **Decorators**: `task`, `task_group`, `setup`, `teardown`
* **Bases**: `BaseOperator`, `BaseSensorOperator`, `BaseNotifier`, `BaseOperatorLink`, `XComArg`
* **Connections & Variables**: `Connection`, `Variable`
* **Tasks & Operators**: `TaskGroup`, `get_current_context`, `get_parsing_context`, `Param`, `chain`, `chain_linear`, `cross_downstream`
* **Assets**: `Asset`, `AssetAlias`, `AssetAll`, `AssetAny`, `AssetWatcher`, `Metadata`

I also ensured that all documented references use the top-level SDK path (e.g., `airflow.sdk.DAG`) instead of exposing the underlying module paths (e.g., `airflow.sdk.definitions.dag.DAG`).

I have incorporated a test that compares the items in the generated `objects.inv` with the elements I re-export from `airflow/sdk/__init__.py`.

The `airflow/sdk/__init__.pyi` type stub is unfortunately necessary, as without it (and with the imports solely within the `if TYPE_CHECKING` block), the re-exported classes were not appearing where I wanted them. Specifically, I want users to import items directly from `airflow.sdk`, not the submodule.

        stubgen -o task-sdk/src -m airflow.sdk

Test this with

     breeze build-docs task-sdk

or 

    uv run --group docs build-docs task-sdk

**Screenshots**
<img width="1720" alt="Screenshot 2025-06-02 at 6 15 40 PM" src="https://github.com/user-attachments/assets/7ce06d9f-1af3-4156-acf7-642d5f37f907" />
<img width="1726" alt="Screenshot 2025-06-02 at 7 04 10 PM" src="https://github.com/user-attachments/assets/adfa99b3-1c40-4c37-b523-4d6ee27381b2" />
<img width="1707" alt="Screenshot 2025-06-02 at 6 44 01 PM" src="https://github.com/user-attachments/assets/d5ccbabc-dfbd-465d-ae32-3697acdce827" />
<img width="1728" alt="Screenshot 2025-06-11 at 1 06 26 AM" src="https://github.com/user-attachments/assets/c8a5792b-59ad-4855-8937-4877a31a9a55" />


closes: #43010
closes: #51283
sunank200 added a commit to astronomer/airflow that referenced this pull request Jul 2, 2025
@sunank200
Docs setup for the Task SDK (apache#51153)
0cc4bfe 
This PR is a continuation of apache#47357

To enhance the experience (i.e., rather than just being given an alphabetical list of classes/functions), I have opted not to use the AutoAPI extension's auto-generate feature, allowing me to precisely control the order and grouping of the classes.

I have organized Task-SDK API into these sections:
* **Defining DAGs**: `DAG`, `dag`
* **Decorators**: `task`, `task_group`, `setup`, `teardown`
* **Bases**: `BaseOperator`, `BaseSensorOperator`, `BaseNotifier`, `BaseOperatorLink`, `XComArg`
* **Connections & Variables**: `Connection`, `Variable`
* **Tasks & Operators**: `TaskGroup`, `get_current_context`, `get_parsing_context`, `Param`, `chain`, `chain_linear`, `cross_downstream`
* **Assets**: `Asset`, `AssetAlias`, `AssetAll`, `AssetAny`, `AssetWatcher`, `Metadata`

I also ensured that all documented references use the top-level SDK path (e.g., `airflow.sdk.DAG`) instead of exposing the underlying module paths (e.g., `airflow.sdk.definitions.dag.DAG`).

I have incorporated a test that compares the items in the generated `objects.inv` with the elements I re-export from `airflow/sdk/__init__.py`.

The `airflow/sdk/__init__.pyi` type stub is unfortunately necessary, as without it (and with the imports solely within the `if TYPE_CHECKING` block), the re-exported classes were not appearing where I wanted them. Specifically, I want users to import items directly from `airflow.sdk`, not the submodule.

        stubgen -o task-sdk/src -m airflow.sdk

Test this with

     breeze build-docs task-sdk

or

    uv run --group docs build-docs task-sdk

**Screenshots**
<img width="1720" alt="Screenshot 2025-06-02 at 6 15 40 PM" src="https://github.com/user-attachments/assets/7ce06d9f-1af3-4156-acf7-642d5f37f907" />
<img width="1726" alt="Screenshot 2025-06-02 at 7 04 10 PM" src="https://github.com/user-attachments/assets/adfa99b3-1c40-4c37-b523-4d6ee27381b2" />
<img width="1707" alt="Screenshot 2025-06-02 at 6 44 01 PM" src="https://github.com/user-attachments/assets/d5ccbabc-dfbd-465d-ae32-3697acdce827" />
<img width="1728" alt="Screenshot 2025-06-11 at 1 06 26 AM" src="https://github.com/user-attachments/assets/c8a5792b-59ad-4855-8937-4877a31a9a55" />

closes: apache#43010
closes: apache#51283
(cherry picked from commit ebfc0de)
@sunank200 sunank200 mentioned this pull request Jul 2, 2025
Merged
kaxil pushed a commit that referenced this pull request Jul 2, 2025
@sunank200
Docs setup for the Task SDK (#51153) (#52682)
9afb410 
This PR is a continuation of #47357

To enhance the experience (i.e., rather than just being given an alphabetical list of classes/functions), I have opted not to use the AutoAPI extension's auto-generate feature, allowing me to precisely control the order and grouping of the classes.

I have organized Task-SDK API into these sections:
* **Defining DAGs**: `DAG`, `dag`
* **Decorators**: `task`, `task_group`, `setup`, `teardown`
* **Bases**: `BaseOperator`, `BaseSensorOperator`, `BaseNotifier`, `BaseOperatorLink`, `XComArg`
* **Connections & Variables**: `Connection`, `Variable`
* **Tasks & Operators**: `TaskGroup`, `get_current_context`, `get_parsing_context`, `Param`, `chain`, `chain_linear`, `cross_downstream`
* **Assets**: `Asset`, `AssetAlias`, `AssetAll`, `AssetAny`, `AssetWatcher`, `Metadata`

I also ensured that all documented references use the top-level SDK path (e.g., `airflow.sdk.DAG`) instead of exposing the underlying module paths (e.g., `airflow.sdk.definitions.dag.DAG`).

I have incorporated a test that compares the items in the generated `objects.inv` with the elements I re-export from `airflow/sdk/__init__.py`.

The `airflow/sdk/__init__.pyi` type stub is unfortunately necessary, as without it (and with the imports solely within the `if TYPE_CHECKING` block), the re-exported classes were not appearing where I wanted them. Specifically, I want users to import items directly from `airflow.sdk`, not the submodule.

        stubgen -o task-sdk/src -m airflow.sdk

Test this with

     breeze build-docs task-sdk

or

    uv run --group docs build-docs task-sdk

**Screenshots**
<img width="1720" alt="Screenshot 2025-06-02 at 6 15 40 PM" src="https://github.com/user-attachments/assets/7ce06d9f-1af3-4156-acf7-642d5f37f907" />
<img width="1726" alt="Screenshot 2025-06-02 at 7 04 10 PM" src="https://github.com/user-attachments/assets/adfa99b3-1c40-4c37-b523-4d6ee27381b2" />
<img width="1707" alt="Screenshot 2025-06-02 at 6 44 01 PM" src="https://github.com/user-attachments/assets/d5ccbabc-dfbd-465d-ae32-3697acdce827" />
<img width="1728" alt="Screenshot 2025-06-11 at 1 06 26 AM" src="https://github.com/user-attachments/assets/c8a5792b-59ad-4855-8937-4877a31a9a55" />

closes: #43010
closes: #51283
(cherry picked from commit ebfc0de)
@ashb ashb deleted the prototype-task-sdk-docs branch August 2, 2025 22:31
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Sep 25, 2025
@sunank200
Docs setup for the Task SDK (#51153) (#52682)
f8300b7 
This PR is a continuation of apache/airflow#47357

To enhance the experience (i.e., rather than just being given an alphabetical list of classes/functions), I have opted not to use the AutoAPI extension's auto-generate feature, allowing me to precisely control the order and grouping of the classes.

I have organized Task-SDK API into these sections:
* **Defining DAGs**: `DAG`, `dag`
* **Decorators**: `task`, `task_group`, `setup`, `teardown`
* **Bases**: `BaseOperator`, `BaseSensorOperator`, `BaseNotifier`, `BaseOperatorLink`, `XComArg`
* **Connections & Variables**: `Connection`, `Variable`
* **Tasks & Operators**: `TaskGroup`, `get_current_context`, `get_parsing_context`, `Param`, `chain`, `chain_linear`, `cross_downstream`
* **Assets**: `Asset`, `AssetAlias`, `AssetAll`, `AssetAny`, `AssetWatcher`, `Metadata`

I also ensured that all documented references use the top-level SDK path (e.g., `airflow.sdk.DAG`) instead of exposing the underlying module paths (e.g., `airflow.sdk.definitions.dag.DAG`).

I have incorporated a test that compares the items in the generated `objects.inv` with the elements I re-export from `airflow/sdk/__init__.py`.

The `airflow/sdk/__init__.pyi` type stub is unfortunately necessary, as without it (and with the imports solely within the `if TYPE_CHECKING` block), the re-exported classes were not appearing where I wanted them. Specifically, I want users to import items directly from `airflow.sdk`, not the submodule.

        stubgen -o task-sdk/src -m airflow.sdk

Test this with

     breeze build-docs task-sdk

or

    uv run --group docs build-docs task-sdk

**Screenshots**
<img width="1720" alt="Screenshot 2025-06-02 at 6 15 40 PM" src="https://github.com/user-attachments/assets/7ce06d9f-1af3-4156-acf7-642d5f37f907" />
<img width="1726" alt="Screenshot 2025-06-02 at 7 04 10 PM" src="https://github.com/user-attachments/assets/adfa99b3-1c40-4c37-b523-4d6ee27381b2" />
<img width="1707" alt="Screenshot 2025-06-02 at 6 44 01 PM" src="https://github.com/user-attachments/assets/d5ccbabc-dfbd-465d-ae32-3697acdce827" />
<img width="1728" alt="Screenshot 2025-06-11 at 1 06 26 AM" src="https://github.com/user-attachments/assets/c8a5792b-59ad-4855-8937-4877a31a9a55" />

closes: #43010
closes: #51283
(cherry picked from commit ebfc0de97ecb7bc550a2182def9a35cd12a8699e)
GitOrigin-RevId: 9afb41032b6826ca5c8d2322778c12ee22616d44
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Oct 23, 2025
@sunank200
Docs setup for the Task SDK (#51153)
693a0e8 
This PR is a continuation of apache/airflow#47357

To enhance the experience (i.e., rather than just being given an alphabetical list of classes/functions), I have opted not to use the AutoAPI extension's auto-generate feature, allowing me to precisely control the order and grouping of the classes.

I have organized Task-SDK API into these sections:
* **Defining DAGs**: `DAG`, `dag`
* **Decorators**: `task`, `task_group`, `setup`, `teardown`
* **Bases**: `BaseOperator`, `BaseSensorOperator`, `BaseNotifier`, `BaseOperatorLink`, `XComArg`
* **Connections & Variables**: `Connection`, `Variable`
* **Tasks & Operators**: `TaskGroup`, `get_current_context`, `get_parsing_context`, `Param`, `chain`, `chain_linear`, `cross_downstream`
* **Assets**: `Asset`, `AssetAlias`, `AssetAll`, `AssetAny`, `AssetWatcher`, `Metadata`

I also ensured that all documented references use the top-level SDK path (e.g., `airflow.sdk.DAG`) instead of exposing the underlying module paths (e.g., `airflow.sdk.definitions.dag.DAG`).

I have incorporated a test that compares the items in the generated `objects.inv` with the elements I re-export from `airflow/sdk/__init__.py`.

The `airflow/sdk/__init__.pyi` type stub is unfortunately necessary, as without it (and with the imports solely within the `if TYPE_CHECKING` block), the re-exported classes were not appearing where I wanted them. Specifically, I want users to import items directly from `airflow.sdk`, not the submodule.

        stubgen -o task-sdk/src -m airflow.sdk

Test this with

     breeze build-docs task-sdk

or

    uv run --group docs build-docs task-sdk

**Screenshots**
<img width="1720" alt="Screenshot 2025-06-02 at 6 15 40 PM" src="https://github.com/user-attachments/assets/7ce06d9f-1af3-4156-acf7-642d5f37f907" />
<img width="1726" alt="Screenshot 2025-06-02 at 7 04 10 PM" src="https://github.com/user-attachments/assets/adfa99b3-1c40-4c37-b523-4d6ee27381b2" />
<img width="1707" alt="Screenshot 2025-06-02 at 6 44 01 PM" src="https://github.com/user-attachments/assets/d5ccbabc-dfbd-465d-ae32-3697acdce827" />
<img width="1728" alt="Screenshot 2025-06-11 at 1 06 26 AM" src="https://github.com/user-attachments/assets/c8a5792b-59ad-4855-8937-4877a31a9a55" />

closes: #43010
closes: #51283
GitOrigin-RevId: ebfc0de97ecb7bc550a2182def9a35cd12a8699e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kaxil kaxil Awaiting requested review from kaxil kaxil will be requested when the pull request is marked ready for review kaxil is a code owner

@amoghrajesh amoghrajesh Awaiting requested review from amoghrajesh amoghrajesh will be requested when the pull request is marked ready for review amoghrajesh is a code owner

Assignees

@sunank200 sunank200

Labels

area:task-sdk priority:medium Bug that should be fixed before next release but would not block a release

Projects

AIP-72 - Task Execution Interface and SDK
Status: Done

Milestone

Airflow 3.0.2

Development

Successfully merging this pull request may close these issues.

Docs: Replace airflow.sdk.definitions to airflow.sdk for public facing docs

5 participants

@ashb @phanikumv @sunank200 @kaxil @eladkal