Airflow API Server Internal Error (500)

[inkerinmaa]

Apache Airflow version

3.1.7

If "Other Airflow 3 version" selected, which one?

No response

What happened?

API Server errors continue to appear in Airflow 3.1.7 as well. K3S deployement via helm chart. Stack:
`INFO: 10.42.0.182:34248 - "GET / HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application

• Exception Group Traceback (most recent call last):
  | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 79, in collapse_excgroups
  | yield
  | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 183, in call
  | async with anyio.create_task_group() as task_group:
  | ^^^^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 783, in aexit
  | raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
  | Traceback (most recent call last):
  | File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
  | result = await app( # type: ignore[func-returns-value]
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in call
  | await super().call(scope, receive, send)
  | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in call
  | await self.middleware_stack(scope, receive, send)
  | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
  | raise exc
  | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
  | await self.app(scope, receive, _send)
  | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in call
  | await responder(scope, receive, send)
  | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in call
  | await super().call(scope, receive, send)
  | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in call
  | await self.app(scope, receive, self.send_with_compression)
  | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in call
  | await self.app(scope, receive, send)
  | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in call
  | with recv_stream, send_stream, collapse_excgroups():
  | ^^^^^^^^^^^^^^^^^^^^
  | File "/usr/python/lib/python3.12/contextlib.py", line 158, in exit
  | self.gen.throw(value)
  | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
  | raise exc
  | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 184, in call
  | response = await self.dispatch_func(request, call_next)
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 49, in dispatch
  | new_user, current_user = await self._refresh_user(current_token)
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 80, in _refresh_user
  | user = await resolve_user_from_token(current_token)
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/core_api/security.py", line 100, in resolve_user_from_token
  | return await get_auth_manager().get_user_from_token(token_str)
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/managers/base_auth_manager.py", line 111, in get_user_from_token
  | return self.deserialize_user(payload)
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 375, in call
  | return wrapper(self._obj, *args, **kwargs)
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 358, in wrapper
  | v = method(self, *args, **kwargs)
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 267, in deserialize_user
  | return self.session.scalars(select(User).where(User.id == int(token["sub"]))).one()
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/scoping.py", line 1891, in scalars
  | return self._proxied.scalars(
  | ^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2459, in scalars
  | return self._execute_internal(
  | ^^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
  | result: Result[Any] = compile_state_cls.orm_execute_statement(
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
  | result = conn.execute(
  | ^^^^^^^^^^^^^
  | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
  | return meth(
  | ^^^^^
  | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
  | return connection._execute_clauseelement(
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
  | ret = self._execute_context(
  | ^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
  | conn = self._revalidate_connection()
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection
  | self._invalid_transaction()
  | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction
  | raise exc.PendingRollbackError(
  | sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b)
  +------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in call
await super().call(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in call
await self.middleware_stack(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in call
await responder(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in call
await super().call(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in call
await self.app(scope, receive, self.send_with_compression)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in call
await self.app(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in call
with recv_stream, send_stream, collapse_excgroups():
^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/contextlib.py", line 158, in exit
self.gen.throw(value)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 184, in call
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 49, in dispatch
new_user, current_user = await self._refresh_user(current_token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 80, in _refresh_user
user = await resolve_user_from_token(current_token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/core_api/security.py", line 100, in resolve_user_from_token
return await get_auth_manager().get_user_from_token(token_str)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/managers/base_auth_manager.py", line 111, in get_user_from_token
return self.deserialize_user(payload)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 375, in call
return wrapper(self._obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 358, in wrapper
v = method(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 267, in deserialize_user
return self.session.scalars(select(User).where(User.id == int(token["sub"]))).one()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/scoping.py", line 1891, in scalars
return self._proxied.scalars(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2459, in scalars
return self._execute_internal(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
result: Result[Any] = compile_state_cls.orm_execute_statement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
result = conn.execute(
^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
return meth(
^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
conn = self._revalidate_connection()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection
self._invalid_transaction()
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction
raise exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b)`

What you think should happen instead?

No response

How to reproduce

3.1.7 Airflow deployment via official Helm chart

Operating System

Debian 12

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[tirkarthi]

Looks similar to #61518 .

[amankum2004]

@inkerinmaa Hello, if no one is working on this issue, can i work on this ?
Can it be assigned to me ?

[inkerinmaa]

Hello, I had a look at #61518 - looks similar.

[inkerinmaa]

Just for information:
tested with apache-airflow-providers-fab==3.1.2 and new 3.3.0. Same issue. Works for a while, then rollback transcation.
With 3.3.0 new issue:
Just tested (Airflow 3.1.7, fab 3.3.0) - opened UI. Trying to open security settings, nothing opens. Logout - trying to login back: INFO: 10.42.0.182:53530 - "GET /airflow/auth/static/appbuilder/css/webfonts/fa-solid-900.woff2 HTTP/1.1" 304 Not Modified
2026-02-16T07:21:07.856279Z [error ] Exception on /login/ [POST] [airflow.providers.fab.www.app] loc=app.py:1744
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
self.dialect.do_execute(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2529, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1823, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1799, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/flask_appbuilder/security/decorators.py", line 36, in wrapped_view
response = make_response(view(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/flask_appbuilder/security/views.py", line 590, in login
user = self.appbuilder.sm.auth_user_db(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1967, in auth_user_db
user = self.find_user(username=username)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1437, in find_user
return self.session.scalars(
^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1778, in scalars
return self.execute(
^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1717, in execute
result = conn._execute_20(statement, params or {}, execution_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
self.handle_dbapi_exception(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2134, in handle_dbapi_exception
util.raise(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/util/compat.py", line 211, in raise
raise exception
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
self.dialect.do_execute(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

[SQL: SELECT ab_user.id, ab_user.first_name, ab_user.last_name, ab_user.username, ab_user.password, ab_user.active, ab_user.email, ab_user.last_login, ab_user.login_count, ab_user.fail_login_count, ab_user.created_on, ab_user.changed_on, ab_user.created_by_fk, ab_user.changed_by_fk
FROM ab_user
WHERE lower(ab_user.username) = lower(%(lower_1)s)]
[parameters: {'lower_1': 'admin'}]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
INFO: 10.42.0.182:53530 - "POST /airflow/auth/login/ HTTP/1.1" 500 Internal Server Error
INFO: 10.42.0.1:42308 - "GET /api/v2/version HTTP/1.1" 200 OK``

[Karabiy]

Keeps happening, still stable only 3.0.2

      pip install --no-cache-dir "apache-airflow-providers-fab==3.0.2" \
      && exec airflow api-server

Only stable version so far discovered personally