JWT token and validation not working

[GrumpyCat51]

Official Helm Chart version

1.16.0 (latest released)

Apache Airflow version

3.0.0

Kubernetes Version

Minikube v1.35.0

Helm Chart configuration

No response

Docker Image customizations

No response

What happened

After downloading the newest chart for airflow 3.0.0, I've made no changes and deployed it through argocd on a local minikube on WSL. I then get the following warning message:

  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
2025-04-28 09:27:25 [warning  ] `api_auth/jwt_secret` was empty, using a generated one for now. Please set this in your config [airflow.api_fastapi.auth.tokens]
[2025-04-28T09:27:25.164+0000] {api_server_command.py:92} INFO - Running the uvicorn with:

followed by the error

[2025-04-28T09:27:45.295+0000] {_core.py:294} DEBUG - registered factory <function _jwt_generator at 0x7f3d57a1f920> for service type airflow.api_fastapi.auth.tokens.JWTGenerator
Stack (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/usr/local/lib/python3.12/multiprocessing/spawn.py", line 135, in _main
    return self._bootstrap(parent_sentinel)
  File "/usr/local/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/_subprocess.py", line 80, in subprocess_started
    target(sockets=sockets)
  File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/supervisors/multiprocess.py", line 63, in target
    return self.real_target(sockets)
  File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/server.py", line 66, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/lifespan/on.py", line 86, in main
    await app(scope, self.receive, self.send)
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 152, in __call__
    await self.app(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 19, in __call__
    await self.app(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 77, in __call__
    await self.app(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 100, in __call__
    await self.app(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 48, in __call__
    await self.app(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 714, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 723, in app
    await self.lifespan(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 692, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/app.py", line 62, in lifespan
    await stack.enter_async_context(
  File "/usr/local/lib/python3.12/contextlib.py", line 659, in enter_async_context
    result = await _enter(cm)
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
  File "/home/airflow/.local/lib/python3.12/site-packages/svcs/fastapi.py", line 66, in __call__
    async with self.registry, cm(app, self.registry) as state:
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/execution_api/app.py", line 91, in lifespan
    registry.register_factory(JWTGenerator, _jwt_generator)
  File "/home/airflow/.local/lib/python3.12/site-packages/svcs/_core.py", line 294, in register_factory
    log.debug(
ERROR:    Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 692, in lifespan
    async with self.lifespan_context(app) as maybe_state:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/app.py", line 62, in lifespan
    await stack.enter_async_context(
  File "/usr/local/lib/python3.12/contextlib.py", line 659, in enter_async_context
    result = await _enter(cm)
             ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/svcs/fastapi.py", line 66, in __call__
    async with self.registry, cm(app, self.registry) as state:
                              ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/execution_api/app.py", line 93, in lifespan
    registry.register_value(JWTValidator, _jwt_validator(), ping=JWTValidator.status)
                                          ^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/execution_api/app.py", line 64, in _jwt_validator
    **get_sig_validation_args(make_secret_key_if_needed=False),
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/tokens.py", line 578, in get_sig_validation_args
    return {"secret_key": get_signing_key("api_auth", "jwt_secret", make_secret_key_if_needed)}
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/tokens.py", line 535, in get_signing_key
    raise ValueError(f"The value {section}/{key} must be set!")
ValueError: The value api_auth/jwt_secret must be set!
ERROR:    Application startup failed. Exiting.
INFO:     Stopping parent process [7]

which prevents the API server from being started.

Then I've added one to the config section

config:
...
  api_auth:
    jwt_secret: '2fn4z0DVXsj3z8p1qU5nfA'
  logging:
    logging_level: 'DEBUG'
    fab_logging_level: 'DEBUG'

now the api server starts up as expected and shows this line in the logs

[2025-04-28T09:44:52.475+0000] {_core.py:339} DEBUG - registered value JWTValidator(jwks=None, issuer=None, required_claims=frozenset({'iat', 'exp', 'aud'}), audience=['urn:airflow.apache.org:task'], algorithm=['HS512'], leeway=10) for service type airflow.api_fastapi.auth.tokens.JWTValidator

Then I try to login with the default admin/admin credentials, but I'm now stuck in an endless loop of 401 Unauthorized

INFO:     10.244.2.90:34112 - "GET / HTTP/1.1" 200 OK
INFO:     10.244.2.90:34112 - "GET /static/assets/index-DVWpfKkn.js HTTP/1.1" 200 OK
INFO:     10.244.2.90:34112 - "GET /ui/config HTTP/1.1" 401 Unauthorized
INFO:     10.244.2.90:34112 - "GET /api/v2/auth/login?next=http%3A%2F%2Fairflow.local%2F HTTP/1.1" 307 Temporary Redirect
INFO:     10.244.2.90:34112 - "GET /auth/login/?next=http://airflow.local/ HTTP/1.1" 200 OK
INFO:     10.244.2.90:34112 - "GET /auth/static/appbuilder/css/bootstrap.min.css HTTP/1.1" 200 OK
INFO:     10.244.2.90:42252 - "GET /auth/static/appbuilder/css/fontawesome/fontawesome.min.css HTTP/1.1" 200 OK
INFO:     10.244.2.90:42264 - "GET /auth/static/appbuilder/css/fontawesome/solid.min.css HTTP/1.1" 200 OK
INFO:     10.244.2.90:42268 - "GET /auth/static/appbuilder/css/fontawesome/brands.min.css HTTP/1.1" 200 OK
INFO:     10.244.2.90:34112 - "GET /auth/static/appbuilder/css/bootstrap-datepicker/bootstrap-datepicker3.min.css HTTP/1.1" 200 OK
INFO:     10.244.2.90:42252 - "GET /auth/static/appbuilder/css/select2/select2.min.css HTTP/1.1" 200 OK
INFO:     10.244.2.90:42260 - "GET /auth/static/appbuilder/css/fontawesome/regular.min.css HTTP/1.1" 200 OK
INFO:     10.244.2.90:42264 - "GET /auth/static/appbuilder/css/select2/select2-bootstrap.min.css HTTP/1.1" 200 OK
INFO:     10.244.2.90:42268 - "GET /auth/static/appbuilder/css/flags/flags16.css HTTP/1.1" 200 OK
INFO:     10.244.2.90:34112 - "GET /auth/static/appbuilder/css/ab.css HTTP/1.1" 200 OK
INFO:     10.244.2.90:42252 - "GET /auth/static/dist/airflowDefaultTheme.feec4a4075c2f3d6ae01.css HTTP/1.1" 200 OK
INFO:     10.244.2.90:42284 - "GET /auth/static/appbuilder/css/fontawesome/v4-shims.min.css HTTP/1.1" 200 OK
INFO:     10.244.2.90:42260 - "GET /auth/static/dist/materialIcons.57390fa60d8f61175334.css HTTP/1.1" 200 OK
INFO:     10.244.2.90:42264 - "GET /auth/static/dist/main.edb2d40dfbbc537916e3.css HTTP/1.1" 200 OK
INFO:     10.244.2.90:42268 - "GET /auth/static/dist/loadingDots.48ab7d5b04e66f2686b0.css HTTP/1.1" 200 OK
INFO:     10.244.2.90:34112 - "GET /auth/static/dist/jquery-ui.min.css HTTP/1.1" 200 OK
INFO:     10.244.2.90:42284 - "GET /auth/static/appbuilder/js/jquery-latest.js HTTP/1.1" 200 OK
INFO:     10.244.2.90:42268 - "GET /auth/static/appbuilder/js/ab_actions.js HTTP/1.1" 200 OK
INFO:     10.244.2.90:42252 - "GET /auth/static/dist/flash.137b30cff85b5588e661.css HTTP/1.1" 200 OK
INFO:     10.244.2.90:34112 - "GET /auth/static/appbuilder/js/bootstrap.min.js HTTP/1.1" 200 OK
INFO:     10.244.2.90:42260 - "GET /auth/static/appbuilder/js/bootstrap-datepicker/bootstrap-datepicker.min.js HTTP/1.1" 200 OK
INFO:     10.244.2.90:42264 - "GET /auth/static/appbuilder/js/ab_filters.js HTTP/1.1" 200 OK
INFO:     10.244.2.90:42268 - "GET /auth/static/appbuilder/js/ab.js HTTP/1.1" 200 OK
INFO:     10.244.2.90:42284 - "GET /auth/static/appbuilder/js/select2/select2.min.js HTTP/1.1" 200 OK
INFO:     10.244.2.90:42252 - "GET /auth/static/dist/moment.624b1f00ba723d39ce06.js HTTP/1.1" 200 OK
INFO:     10.244.2.90:34112 - "GET /auth/static/dist/main.edb2d40dfbbc537916e3.js HTTP/1.1" 200 OK
INFO:     10.244.2.90:42260 - "GET /auth/static/dist/jquery-ui.min.js HTTP/1.1" 200 OK
INFO:     10.244.2.90:42260 - "GET /auth/static/appbuilder/css/webfonts/fa-solid-900.woff2 HTTP/1.1" 200 OK
INFO:     10.244.2.90:42260 - "GET /auth/static/pin_32.png HTTP/1.1" 404 Not Found
INFO:     10.244.0.1:45996 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     10.244.0.1:46010 - "GET /api/v2/version HTTP/1.1" 200 OK
[2025-04-28T09:48:19.326+0000] {override.py:1423} INFO - Updated user admin user
INFO:     10.244.2.90:34112 - "POST /auth/login/?next=http://airflow.local/ HTTP/1.1" 302 Found
INFO:     10.244.2.90:34112 - "GET / HTTP/1.1" 200 OK
INFO:     10.244.2.90:42260 - "GET /ui/config HTTP/1.1" 401 Unauthorized
INFO:     10.244.2.90:34112 - "GET /api/v2/auth/login?next=http%3A%2F%2Fairflow.local%2F HTTP/1.1" 307 Temporary Redirect
INFO:     10.244.2.90:34112 - "GET /auth/login/?next=http://airflow.local/ HTTP/1.1" 302 Found
INFO:     10.244.2.90:42260 - "GET /auth/ HTTP/1.1" 302 Found
INFO:     10.244.2.90:42252 - "GET / HTTP/1.1" 200 OK
INFO:     10.244.2.90:42252 - "GET /ui/config HTTP/1.1" 401 Unauthorized
INFO:     10.244.2.90:42252 - "GET /api/v2/auth/login?next=http%3A%2F%2Fairflow.local%2F HTTP/1.1" 307 Temporary Redirect
INFO:     10.244.2.90:42252 - "GET /auth/login/?next=http://airflow.local/ HTTP/1.1" 302 Found
INFO:     10.244.2.90:42252 - "GET /auth/ HTTP/1.1" 302 Found
INFO:     10.244.2.90:42268 - "GET / HTTP/1.1" 200 OK

I've then tried to use the new airflow.api_fastapi.auth.managers.simple.simple_auth_manager.SimpleAuthManager, set through the same configs as above, with simple_auth_manager_all_admins set to True as this should avoid any verification, but I'm still stuck in the same Unauthorized loop

What you think should happen instead

I'm not quite sure what went wrong (I suspect the JWT part), but I'd expect the default settings to work.

How to reproduce

Start up the helm chart in a minikube cluster and add the ingress from #49727

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[paolofacchinetti]

hey @GrumpyCat51 FYI it's a known issue that you can mitigate by setting AIRFLOW__API_AUTH__JWT_SECRET manually, as shown here: #49646

fix is yet to be implemented

[GrumpyCat51]

hey @GrumpyCat51 FYI it's a known issue that you can mitigate by setting AIRFLOW__API_AUTH__JWT_SECRET manually, as shown here: #49646

fix is yet to be implemented

I've added this through

  api_auth:
    jwt_secret: '2fn4z0DVXsj3z8p1qU5nfA'

but still always run into the 401 Unauthorized further down. I'm trying to investigate where the error comes from through adding more logging (and disabling certain steps)

Following https://airflow.apache.org/docs/apache-airflow/stable/security/api.html, I can successfully fetch a jwt token and run

curl -s -X GET "${ENDPOINT_URL}/ui/config" -H "Authorization: Bearer ${ACCESS_TOKEN}" 

[tirkarthi]

This seems to be relevant to #49495 and below comment might be helpful.

#49495 (comment)

[GrumpyCat51]

Thanks @tirkarthi, updating to apache-airflow-providers-fab==2.0.2rc2 has resolved the issue

[bdsoha]

The JWT secret token is bundled in #49923