Skip to content

Conversation

@ashb
Copy link
Member

@ashb ashb commented Mar 17, 2025

This adds in Bearer auth token validation for the entire Execution API server,
and JWT generation to the Scheduler before handing off to the executor.

Some notes about how this is achieved:

  • We make heavy use of FastAPI's Dependency Injection to make this work

  • We use the svcs module to make managing the service, and viewing the
    state of it, easy

  • In .execution_api.routes there are now two routers used -- one which
    ensures that everything underneath it has a valid ExecutionAPI JWT, and
    another one which is "public" (and should contain very few routes)

  • We ran into an issue (linked in code) with __future__ annotations imports,
    and had to disable it on one specific file.

  • Refreshing of expiring Task Identity tokens is not handled yet (this will
    need middleware to support, a Dependency cannot add the response headers)

  • This could almost certainly use more tests than it has right now.

  • The API docs know that a JWT Bearer is needed
    Image

Closes #45107


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:API Airflow's REST/HTTP API area:Executors-core LocalExecutor & SequentialExecutor area:Scheduler including HA (high availability) scheduler area:task-sdk labels Mar 17, 2025
@ashb ashb force-pushed the task-execution-api-jwts branch 2 times, most recently from 961d886 to 9045edb Compare March 18, 2025 11:30
Copy link
Member

@pierrejeambrun pierrejeambrun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice I think it's a solid base to keep iterating on, multiple things could be improved but I guess this will be done in follow up PR.

just one remark but not blocking

ashb added 2 commits March 18, 2025 19:48
This adds in Bearer auth token validation for the entire Execution API server.

Some notes about how this is achieved:

* We make heavy use of FastAPI's Dependency Injection to make this work

* We use the `svcs`[1] module to make managing the service, and viewing the
  state of it, easy

* In `.execution_api.routes` there are now two routers used -- one which
  ensures that _everything_ underneath it has a valid ExecutionAPI JWT, and
  another one which is "public" (and should contain very few routes)

* We ran into an issue (linked in code) with `__future__ annotations` imports,
  and had to disable it on one specific file.

* Refreshing of expiring Task Identity tokens is not handled yet (this will
  need middleware to support, a Dependency cannot add the response headers)

[1]: pypi.org/project/svcs
Without this being set the API server now crash-loops on startup, which adds
an extra 2 minutes to the test time for no reason.
@ashb ashb force-pushed the task-execution-api-jwts branch from 9380b97 to 7bcf2e6 Compare March 18, 2025 19:48
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
@ashb ashb merged commit c4e96d4 into apache:main Mar 18, 2025
148 checks passed
@ashb ashb deleted the task-execution-api-jwts branch March 18, 2025 21:40
agupta01 pushed a commit to agupta01/airflow that referenced this pull request Mar 21, 2025
…che#47885)

* Validate JWT tokens in the ExecutionAPI for strong task identity

This adds in Bearer auth token validation for the entire Execution API server.

Some notes about how this is achieved:

- We make heavy use of FastAPI's Dependency Injection to make this work

- We use the [`svcs`][1] module to make managing the service, and viewing the
  state of it, easy

- In `.execution_api.routes` there are now two routers used -- one which
  ensures that _everything_ underneath it has a valid ExecutionAPI JWT, and
  another one which is "public" (and should contain very few routes)

- We ran into an issue (linked in code) with `__future__ annotations` imports,
  and had to disable it on one specific file.

- Refreshing of expiring Task Identity tokens is not handled yet (this will
  need middleware to support, a Dependency cannot add the response headers)

[1]: pypi.org/project/svcs

* Set `api_auth/jwt_secret` at deploy-time in our Kube tests

Without this being set the API server now crash-loops on startup, which adds
an extra 2 minutes to the test time for no reason.
nailo2c pushed a commit to nailo2c/airflow that referenced this pull request Apr 4, 2025
…che#47885)

* Validate JWT tokens in the ExecutionAPI for strong task identity

This adds in Bearer auth token validation for the entire Execution API server.

Some notes about how this is achieved:

- We make heavy use of FastAPI's Dependency Injection to make this work

- We use the [`svcs`][1] module to make managing the service, and viewing the
  state of it, easy

- In `.execution_api.routes` there are now two routers used -- one which
  ensures that _everything_ underneath it has a valid ExecutionAPI JWT, and
  another one which is "public" (and should contain very few routes)

- We ran into an issue (linked in code) with `__future__ annotations` imports,
  and had to disable it on one specific file.

- Refreshing of expiring Task Identity tokens is not handled yet (this will
  need middleware to support, a Dependency cannot add the response headers)

[1]: pypi.org/project/svcs

* Set `api_auth/jwt_secret` at deploy-time in our Kube tests

Without this being set the API server now crash-loops on startup, which adds
an extra 2 minutes to the test time for no reason.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:Executors-core LocalExecutor & SequentialExecutor area:Scheduler including HA (high availability) scheduler area:task-sdk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create JWT token issuer infrastructure for strong Task Identity

4 participants