Cache user object fetched per request in FAB auth manager for improved performance.

[tirkarthi]

When a request is authenticated the user object is created from jwt token twice in the middleware layer and in the GetUserDep dependency resolution. Each token maps to a single user and hence cache the token's id to the user object so that it's reused.

TTLCache from cachetools is used to ensure the cached user objects expire to be fetched again. The TTL value is configurable through fab.cache_ttl configuration with a default of 30 seconds. Each worker and instance of auth manager has its own cache and they are not persisted to be reset as workers restart. In case this needs to be disabled then users can set the cache_ttl as 0. I have added test to ensure the cache is expired. In case of deployments with frequent roles and permission changes users can set this to lower value at the cost of frequent db queries. For deployments with lesser number of changes to roles and permissions this can be set to a higher value for better performance. This should mitigate the need to invalidate caches on roles and permission changes. If required this can be documented with more detail in changelog just to avoid confusion.

https://cachetools.readthedocs.io/en/latest/#cachetools.cachedmethod

closes #60265

[vincbeck]

As brought up in the related issue, we need to think about the case when permissions are updated

We need to think about permissions change

[tirkarthi]

Mypy related error as below but not reproducible locally. Do I need to add stub pypi packages in pyproject.toml?

   providers/fab/src/airflow/providers/fab/auth_manager/fab_auth_manager.py:25: error: Library stubs not installed for "cachetools"  [import-untyped]
      from cachetools import TTLCache, cachedmethod
      ^
  providers/fab/src/airflow/providers/fab/auth_manager/fab_auth_manager.py:25: note: Hint: "python3 -m pip install types-cachetools"
  providers/fab/src/airflow/providers/fab/auth_manager/fab_auth_manager.py:25: note: (or run "mypy --install-types" to install all missing stub packages)
  providers/fab/src/airflow/providers/fab/auth_manager/fab_auth_manager.py:25: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
  Found 1 error in 1 file (checked 4075 source files)

  Error 1 returned
  You are running mypy with the folders selected. If you want to reproduce it locally, you need to run the following command:

  prek --hook-stage manual mypy-<folder> --all-files

  If you see strange stacktraces above, and can't reproduce it, please run this command and try again:

  breeze ci-image build --python 3.10

  You can also run `breeze down --cleanup-mypy-cache` to clean up the cache used.

Running mypy locally

mypy --version                                                               
mypy 1.18.2 (compiled: yes)
pip freeze | grep -i cachetools                            
cachetools==6.2.4
mypy providers/fab/src/airflow/providers/fab/auth_manager/fab_auth_manager.py
Success: no issues found in 1 source file

[uranusjr]

Single-file type checking is limited. Try using the suggested prek command locally and see if you can reproduce the error.

[tirkarthi]

@uranusjr Thanks, I am able to reproduce it locally. I am adding cachetools dependency. I do see ignore_missing_imports=True in pyproject.yaml but see types added in devel-common/pyproject.toml for mypy. It seems adding types-cachetools in devel-common seems to help.

airflow/pyproject.toml

Lines 1004 to 1006 in fb21525

	## mypy settings ##
	[tool.mypy]
	ignore_missing_imports = true

airflow/devel-common/pyproject.toml

Lines 107 to 130 in fb21525

	"mypy" = [
	# Mypy dependencies
	# TODO: upgrade to newer versions of MyPy continuously as they are released
	"mypy==1.19.1",
	"types-Deprecated>=1.2.9.20240311",
	"types-Markdown>=3.6.0.20240316",
	"types-PyMySQL>=1.1.0.20240425",
	"types-PyYAML>=6.0.12.20240724",
	"types-aiofiles>=23.2.0.20240403",
	"types-certifi>=2021.10.8.3",
	"types-croniter>=2.0.0.20240423",
	"types-docutils>=0.21.0.20240704",
	# TODO: Bump to >= 4.0.0 once https://github.com/apache/airflow/issues/54079
	"types-paramiko>=3.4.0.20240423,<4.0.0",
	"types-protobuf>=5.26.0.20240422",
	"types-python-dateutil>=2.9.0.20240316",
	"types-python-slugify>=8.0.2.20240310",
	"types-pytz>=2024.1.0.20240417",
	"types-redis>=4.6.0.20240425",
	"types-requests>=2.31.0",
	"types-setuptools>=80.0.0.20250429",
	"types-tabulate>=0.9.0.20240106",
	"types-toml>=0.10.8.20240310",
	]

prek --color always --verbose --hook-stage manual "mypy-providers" --all-files
Running hooks for `providers`:
Run mypy for providers (manual)..........................................Failed
- hook id: mypy-providers
- duration: 282.93s
- exit code: 1

  Running mypy with @/files/mypy_files.txt
  You cand check the list of files in: /home/karthikeyan/stuff/python/airflow/files/mypy_files.txt
  Running mypy with @/files/mypy_files.txt
  Regenerating provider dependencies file
  Refreshed 97 providers with 1983 Python files.

  Written /home/karthikeyan/stuff/python/airflow/generated/provider_dependencies.json

  Docker container engine detected.
  Attempting to generate provider dependencies. 2 pyproject.toml file(s) changed 
  since last check.

  Using 'uv' to install Airflow

  Using airflow version from current sources

  providers/fab/src/airflow/providers/fab/auth_manager/fab_auth_manager.py:25: error:
  Library stubs not installed for "cachetools"  [import-untyped]
      from cachetools import TTLCache, cachedmethod
      ^
  providers/fab/src/airflow/providers/fab/auth_manager/fab_auth_manager.py:25: note: Hint: "python3 -m pip install types-cachetools"
  providers/fab/src/airflow/providers/fab/auth_manager/fab_auth_manager.py:25: note: (or run "mypy --install-types" to install all missing stub packages)
  providers/fab/src/airflow/providers/fab/auth_manager/fab_auth_manager.py:25: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
  Found 1 error in 1 file (checked 4075 source files)
  Error 1 returned
  If you see strange stacktraces above, and can't reproduce it, please run this command and try again:

  breeze ci-image build --python 3.10

  You can also run `breeze down --cleanup-mypy-cache` to clean up the cache used.

prek --color always --verbose --hook-stage manual "mypy-providers" --all-files
Running hooks for `providers`:
Run mypy for providers (manual)..........................................Passed
- hook id: mypy-providers
- duration: 261.46s

  Running mypy with @/files/mypy_files.txt
  You cand check the list of files in: /home/karthikeyan/stuff/python/airflow/files/mypy_files.txt
  Running mypy with @/files/mypy_files.txt
  Docker container engine detected.

  Using 'uv' to install Airflow

  Using airflow version from current sources

  Success: no issues found in 4075 source files

[tirkarthi]

Thanks @vincbeck for the review.

cc: @potiuk for also extra eyes on the security aspect.

[potiuk]

LGTM.

CC: @jscheffl -> good idea with the warning in release notes. Maybe we should use more of those - to just recall some non-obvious things (though in this case it's pretty clear it's a new feature and unless we have a breaking change, we will end up with 3.2.0 for FAB provider.

[jscheffl]

LGTM.

CC: @jscheffl -> good idea with the warning in release notes. Maybe we should use more of those - to just recall some non-obvious things (though in this case it's pretty clear it's a new feature and unless we have a breaking change, we will end up with 3.2.0 for FAB provider.

Kudos to @eladkal who pinged me on slack about this.

Actually his proposal was that we should consider a # use next version for such forward looking config keys as well :-D

[potiuk]

Actually his proposal was that we should consider a # use next version for such forward looking config keys as well :-D

We could, yes.