Custom logging handler not sending logs to ClickHouse when set via LOGGING_CONFIG_CLASS

[serviliosouza]

I’m trying to configure a custom logging handler in Airflow to send logs directly to ClickHouse. I followed the official documentation on advanced logging configuration:

https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/advanced-logging-configuration.html

Environment

Image: apache/airflow:3.1.0-python3.12

Executor: CeleryExecutor (with Redis + Postgres)

Docker Compose deployment

Custom handler defined in /opt/airflow/plugins/clickhouse_logging.py

Config in /opt/airflow/plugins/log_config.py, referenced with:

AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS=log_config.LOGGING_CONFIG (compose container environment variable)

What works:

Direct Python test inside container with ClickHouseHTTPHandler() correctly inserts rows into ClickHouse.

If I manually add the handler in a DAG (using logging.getLogger("airflow.task").addHandler(ClickHouseHTTPHandler()) before send logs), logs are sent successfully to clickhouse.

What doesn’t work:

When relying on the global LOGGING_CONFIG_CLASS setup, no logs are pushed to ClickHouse.

Airflow does parse the config file: if I change the class name to a non-existent one, Airflow raises the expected error, but apparently it does not append the handler to the loggers.

Tests already done:

Verified sys.path includes /opt/airflow/plugins.

Compared DEFAULT_LOGGING_CONFIG vs custom LOGGING_CONFIG.

Tested ClickHouseHTTPHandler manually (works).

Confirmed Airflow recognizes the LOGGING_CONFIG_CLASS path (with command airflow config get-value logging logging_config_class).

Verified connectivity and schema: inserts succeed manually into db_logs.tb_logs_airflow.

Question:

Why are logs not being pushed when configured via LOGGING_CONFIG_CLASS, even though:

The handler is discoverable and testable.

Manual DAG-level attachment works.

Airflow parses the config without errors.

Any insights on whether there’s a nuance in how Airflow attaches custom handlers globally (vs. per DAG) would be greatly appreciated.

Thanks in advance!

Solving the problem

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.