airflow error:AttributeError: module 'airflow.utils.log' has no attribute 'file_processor_handler'

[lsabreu96]

Specs:
Docker image: apache/airflow:2.2.2-python3.7
I'm spinning the environment up using the docker-compose and specifying the package as such

_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:- airflow-dbt-python==0.15.2 dbt-redshift==1.3.0}

The error is

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
google-ads 14.0.0 requires PyYAML<6.0,>=5.1, but you have pyyaml 6.0 which is incompatible.
apache-airflow 2.2.2 requires markupsafe<2.0,>=1.1.1, but you have markupsafe 2.1.2 which is incompatible.
Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.2 agate-1.6.3 airflow-dbt-python-0.15.2 dbt-core-1.3.2 dbt-extractor-0.4.1 dbt-postgres-1.3.2 dbt-redshift-1.3.0 future-0.18.3 hologram-0.0.15 leather-0.3.4 logbook-1.5.3 mashumaro-3.0.4 minimal-snowplow-tracker-0.0.2 networkx-2.6.3 parsedatetime-2.4 pathspec-0.9.0 pytimeparse-1.1.8 pyyaml-6.0 sqlparse-0.4.3
WARNING: You are using pip version 21.3.1; however, version 22.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/logging/config.py", line 388, in resolve
    found = getattr(found, frag)
AttributeError: module 'airflow.utils.log' has no attribute 'file_processor_handler'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/logging/config.py", line 390, in resolve
    self.importer(used)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/log/file_processor_handler.py", line 25, in <module>
    from airflow.utils.helpers import parse_template_string
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/helpers.py", line 27, in <module>
    from flask import url_for
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/__init__.py", line 14, in <module>
    from jinja2 import escape
ImportError: cannot import name 'escape' from 'jinja2' (/home/airflow/.local/lib/python3.7/site-packages/jinja2/__init__.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/logging/config.py", line 563, in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.7/logging/config.py", line 713, in configure_handler
    klass = self.resolve(cname)
  File "/usr/local/lib/python3.7/logging/config.py", line 397, in resolve
    raise v
  File "/usr/local/lib/python3.7/logging/config.py", line 390, in resolve
    self.importer(used)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/log/file_processor_handler.py", line 25, in <module>
    from airflow.utils.helpers import parse_template_string
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/helpers.py", line 27, in <module>
    from flask import url_for
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/__init__.py", line 14, in <module>
    from jinja2 import escape
ValueError: Cannot resolve 'airflow.utils.log.file_processor_handler.FileProcessorHandler': cannot import name 'escape' from 'jinja2' (/home/airflow/.local/lib/python3.7/site-packages/jinja2/__init__.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__init__.py", line 46, in <module>
    settings.initialize()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/settings.py", line 483, in initialize
    LOGGING_CLASS_PATH = configure_logging()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/logging_config.py", line 74, in configure_logging
    raise e
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/logging_config.py", line 69, in configure_logging
    dictConfig(logging_config)
  File "/usr/local/lib/python3.7/logging/config.py", line 800, in dictConfig
    dictConfigClass(config).configure()
  File "/usr/local/lib/python3.7/logging/config.py", line 571, in configure
    '%r' % name) from e
ValueError: Unable to configure handler 'processor'

But I'm receiving the above error. Under some investigation, the most useful threads seemed to be this one, which is related to some naming conflict.

I pinned the version as such
psutil==4.4.2

But no success still. Also tried pinning PyYaml to be under 6, but also without success

Any thoughts ?

[tomasfarias]

Hi @lsabreu96, thanks for reporting an issue.

Looking at this line:

ImportError: cannot import name 'escape' from 'jinja2' (/home/airflow/.local/lib/python3.7/site-packages/jinja2/__init__.py)

I would imagine this is the related issue: https://stackoverflow.com/questions/71718167/importerror-cannot-import-name-escape-from-jinja2. Have you tried upgrading to Airflow 2.3 which bumps the Flask=1.1.2 constraint to Flask=2.2.2? That should take care of the issue if my assessment is correct. If you are looking for reference versions, we test Airflow versions 2.4.3 and 2.5.1, with latest dbt (1.4) and Python >=3.7, and haven't had any issues.

This could be an incompatibility between the requirements between Airflow and dbt (as both appear to require jinja2), so there's not much we can do from our end.