Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create simple "Task API" server based on FastAPI #43009

Closed
kaxil opened this issue Oct 14, 2024 · 2 comments · Fixed by #43015
Closed

Create simple "Task API" server based on FastAPI #43009

kaxil opened this issue Oct 14, 2024 · 2 comments · Fixed by #43015
Assignees
Labels
AIP-72 Task Execution Interface aka Task SDK area:API Airflow's REST/HTTP API
Milestone

Comments

@kaxil
Copy link
Member

kaxil commented Oct 14, 2024

Need to work with Pierre on this to work out the right code and URI layout for this.

I think there have been some people who want to run them separately (it was certainly asked about by someone in my talk), so I think they should be structured as two separate FastAPI/ASGI apps that can optionally be run in a single binary (i.e. ASGI "mounting" etc.). At this stage I'm thinking that we should not separate out the dependencies/make them stand-alone projects (that can happen later if there proves to be enough desire)

@kaxil kaxil self-assigned this Oct 14, 2024
@kaxil kaxil added the AIP-72 Task Execution Interface aka Task SDK label Oct 14, 2024
@kaxil kaxil added this to the Airflow 3.0.0 milestone Oct 14, 2024
@dosubot dosubot bot added the area:API Airflow's REST/HTTP API label Oct 14, 2024
kaxil added a commit to astronomer/airflow that referenced this issue Oct 14, 2024
@potiuk
Copy link
Member

potiuk commented Oct 15, 2024

I think it really depends on how we want to treat providers - this is the big difference of dependencies. Technically - if we solve base links and connection issue, we could have the same dependencies for both - none of the task api calls nor web API should need providers at all in this case. On the other hand, if we don't solve it, then both will have providers if they are run together - but task API technically could be separate and have no provider dependencies.

I think we should aim on having the same dependencies in both for simplicity, and we should see if we can remove providers from both - but this can also be done later - either before 3.0 or after - this only changes security property of the deployment, so it won't be a breaking change.

@kaxil
Copy link
Member Author

kaxil commented Oct 15, 2024

Yea, we absolutely need to solve Operator links & Connection deps. Jens & Vikram have a proposal. One of the last dev calls (Sep 19) we discussed about routing it through an API (and stored it in DB):

kaxil added a commit to astronomer/airflow that referenced this issue Oct 15, 2024
kaxil added a commit to astronomer/airflow that referenced this issue Oct 16, 2024
kaxil added a commit to astronomer/airflow that referenced this issue Oct 16, 2024
kaxil added a commit to astronomer/airflow that referenced this issue Oct 17, 2024
closes apache#43009

The Task Execution API is created as a separate FastAPI APP so that it allows us to hook CLI and selectively run the "Execution API" with the rest of Airflow API (UI/public). This is so that users can scale the APIs separately if needed as mentioned in apache#43009
@kaxil kaxil closed this as completed in 9ce6d49 Oct 17, 2024
R7L208 pushed a commit to R7L208/airflow that referenced this issue Oct 17, 2024
closes apache#43009

The Task Execution API is created as a separate FastAPI APP so that it allows us to hook CLI and selectively run the "Execution API" with the rest of Airflow API (UI/public). This is so that users can scale the APIs separately if needed as mentioned in apache#43009
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AIP-72 Task Execution Interface aka Task SDK area:API Airflow's REST/HTTP API
Development

Successfully merging a pull request may close this issue.

2 participants