Cannot upgrade Airflow 3.1.0 to 3.1.7, session seems to be empty

[jlandercy]

I would like to upgrade my Airflow instances to the latest version of Airflow 3.1.7.

But when I build the new image, everything starts but the API fails with the following error:

[2026-03-01T09:48:38.180881Z] {_utils.py:59} ERROR - Exception when querying database (MessagePack data is malformed: trailing characters (byte 1)).Retrying (1/3) in 0.30s.
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/flask_session/_utils.py", line 52, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask_session/sqlalchemy/sqlalchemy.py", line 152, in _retrieve_session_data
    return self.serializer.decode(serialized_session_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/session.py", line 35, in loads
    return decoder.decode(data)
           ^^^^^^^^^^^^^^^^^^^^
msgspec.DecodeError: MessagePack data is malformed: trailing characters (byte 1)
[2026-03-01T09:48:38.483113Z] {_utils.py:59} ERROR - Exception when querying database (MessagePack data is malformed: trailing characters (byte 1)).Retrying (2/3) in 0.60s.
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/flask_session/_utils.py", line 52, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask_session/sqlalchemy/sqlalchemy.py", line 152, in _retrieve_session_data
    return self.serializer.decode(serialized_session_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/session.py", line 35, in loads
    return decoder.decode(data)
           ^^^^^^^^^^^^^^^^^^^^
msgspec.DecodeError: MessagePack data is malformed: trailing characters (byte 1)
[2026-03-01T09:48:39.085030Z] {app.py:1744} ERROR - Exception on /login/ [GET]
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2528, in wsgi_app
    ctx.push()
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/ctx.py", line 375, in push
    self.session = session_interface.open_session(self.app, self.request)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask_session/base.py", line 350, in open_session
    saved_session_data = self._retrieve_session_data(store_id)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask_session/_utils.py", line 56, in wrapper
    raise e
  File "/home/airflow/.local/lib/python3.12/site-packages/flask_session/_utils.py", line 52, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask_session/sqlalchemy/sqlalchemy.py", line 152, in _retrieve_session_data
    return self.serializer.decode(serialized_session_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/session.py", line 35, in loads
    return decoder.decode(data)
           ^^^^^^^^^^^^^^^^^^^^
msgspec.DecodeError: MessagePack data is malformed: trailing characters (byte 1)
INFO:     172.18.0.4:58526 - "GET /auth/login/?next=https://app.landercy.be/ HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
  + Exception Group Traceback (most recent call last):
  |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 79, in collapse_excgroups
  |     yield
  |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 183, in __call__
  |     async with anyio.create_task_group() as task_group:
  |                ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 783, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Exception Group Traceback (most recent call last):
    |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 184, in __call__
    |     response = await self.dispatch_func(request, call_next)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 237, in cleanup_session_middleware
    |     response = await call_next(request)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 159, in call_next
    |     raise app_exc
    |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 144, in coro
    |     await self.app(scope, receive_or_disconnect, send_no_error)
    |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
    |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
    |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    |     raise exc
    |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    |     await app(scope, receive, sender)
    |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 716, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 736, in app
    |     await route.handle(scope, receive, send)
    |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 462, in handle
    |     await self.app(scope, receive, send)
    |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/wsgi.py", line 81, in __call__
    |     await responder(receive, send)
    |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/wsgi.py", line 106, in __call__
    |     async with anyio.create_task_group() as task_group:
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 783, in __aexit__
    |     raise BaseExceptionGroup(
    | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
    +-+---------------- 1 ----------------
      | Traceback (most recent call last):
      |   File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
      |     result = await app(  # type: ignore[func-returns-value]
      |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in __call__
      |     await super().__call__(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
      |     await self.middleware_stack(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
      |     raise exc
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
      |     await self.app(scope, receive, _send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in __call__
      |     await responder(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in __call__
      |     await super().__call__(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in __call__
      |     await self.app(scope, receive, self.send_with_compression)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
      |     await self.app(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 189, in __call__
      |     raise app_exc
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 144, in coro
      |     await self.app(scope, receive_or_disconnect, send_no_error)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
      |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
      |     raise exc
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
      |     await app(scope, receive, sender)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 716, in __call__
      |     await self.middleware_stack(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 736, in app
      |     await route.handle(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 462, in handle
      |     await self.app(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in __call__
      |     await super().__call__(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
      |     await self.middleware_stack(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
      |     raise exc
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
      |     await self.app(scope, receive, _send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in __call__
      |     with recv_stream, send_stream, collapse_excgroups():
      |                                    ^^^^^^^^^^^^^^^^^^^^
      |   File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__
      |     self.gen.throw(value)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
      |     raise exc
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/wsgi.py", line 109, in __call__
      |     await anyio.to_thread.run_sync(self.wsgi, environ, self.start_response)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 63, in run_sync
      |     return await get_async_backend().run_sync_in_worker_thread(
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2502, in run_sync_in_worker_thread
      |     return await future
      |            ^^^^^^^^^^^^
      |   File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 986, in run
      |     result = context.run(func, *args)
      |              ^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/wsgi.py", line 147, in wsgi
      |     for chunk in self.app(environ, start_response):
      |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2552, in __call__
      |     return self.wsgi_app(environ, start_response)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2532, in wsgi_app
      |     response = self.handle_exception(e)
      |                ^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1727, in handle_exception
      |     server_error = self.ensure_sync(handler)(server_error)
      |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/views.py", line 75, in show_traceback
      |     return render_template("airflow/traceback.html"), 500
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/airflow/.local/lib/python3.12/site-packages/flask/templating.py", line 147, in render_template
      |     return _render(app, template, context)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/airflow/.local/lib/python3.12/site-packages/flask/templating.py", line 128, in _render
      |     app.update_template_context(context)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 997, in update_template_context
      |     context.update(func())
      |                    ^^^^^^
      |   File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/utils.py", line 405, in _user_context_processor
      |     return dict(current_user=_get_user())
      |                              ^^^^^^^^^^^
      |   File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/utils.py", line 370, in _get_user
      |     current_app.login_manager._load_user()
      |   File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/login_manager.py", line 362, in _load_user
      |     user_id = session.get("_user_id")
      |               ^^^^^^^^^^^
      | AttributeError: 'NoneType' object has no attribute 'get'
      +------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in __call__
    await super().__call__(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in __call__
    await responder(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in __call__
    await super().__call__(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in __call__
    await self.app(scope, receive, self.send_with_compression)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 189, in __call__
    raise app_exc
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 144, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 716, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 736, in app
    await route.handle(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 462, in handle
    await self.app(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in __call__
    await super().__call__(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in __call__
    with recv_stream, send_stream, collapse_excgroups():
                                   ^^^^^^^^^^^^^^^^^^^^
  File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
    raise exc
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/wsgi.py", line 109, in __call__
    await anyio.to_thread.run_sync(self.wsgi, environ, self.start_response)
  File "/home/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 63, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2502, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 986, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/wsgi.py", line 147, in wsgi
    for chunk in self.app(environ, start_response):
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2552, in __call__
    return self.wsgi_app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2532, in wsgi_app
    response = self.handle_exception(e)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1727, in handle_exception
    server_error = self.ensure_sync(handler)(server_error)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/views.py", line 75, in show_traceback
    return render_template("airflow/traceback.html"), 500
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/templating.py", line 147, in render_template
    return _render(app, template, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/templating.py", line 128, in _render
    app.update_template_context(context)
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 997, in update_template_context
    context.update(func())
                   ^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/utils.py", line 405, in _user_context_processor
    return dict(current_user=_get_user())
                             ^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/utils.py", line 370, in _get_user
    current_app.login_manager._load_user()
  File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/login_manager.py", line 362, in _load_user
    user_id = session.get("_user_id")
              ^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'

It seems the session for the user does not exist and return None. My dependencies are up-to-date:

apache-airflow==3.1.7
apache-airflow-core==3.1.7
apache-airflow-providers-fab==3.3.0
apache-airflow-providers-common-compat>=1.7.4
apache-airflow-providers-common-io>=1.6.3
apache-airflow-providers-common-sql>=1.28.1
apache-airflow-providers-papermill>=3.11.3
apache-airflow-providers-postgres>=6.3.0
apache-airflow-providers-smtp>=2.3.1
apache-airflow-providers-standard>=1.9.0
apache-airflow-task-sdk>=1.1.2

I have done several research on the internet but it does not help for this specific error. I am wondering what am I doing wrong?

Any help is appreciated. Thank you further.

Jean

[jlandercy]

Ok, after digging a bit more it seems the session payload has changed between the two version.

After TRUNCATE session; the error vanished but the FAB GitHub Auth is failing now.