How to limit the maximum log size for a single task run in Airflow?

[HuanjieGuo]

I have a disk to store the log for the task run.

Sometimes the task may contain some long-running log logic, and it can produce a 5Gb size log file.

This is very dangerous for a single task to produce such a large log file because it will make the disk full.

My question is how can we limit the maximum log size for a run in the airflow cluster?

[dondaum]

Not that I know. 

IMO producing a log file with 5gb should be avoided.

Since Airflow 2.6.0 you can delete local log files whenever using remote logging. Local files are deleted after they are uploaded to the remote location. That might free up some space again. 

https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#delete-local-logs

[HuanjieGuo]

Thanks @dondaum , I understand that producing a log file with 5 GB should be avoided.

However, when we provide a schedule cluster for other users to write their pipelines, we should also think about how to isolate the resource.

The cluster can be out of service just due to a little bug in a task, it is unreasonable.

[HuanjieGuo]

@potiuk does this feature make sense to the community? we can try to contribute it.

[potiuk]

Sure you can contribute @HuanjieGuo - but that should be documentation-only change at most.

I think there is nothing to implement, possibly a documentation how to configure "task" log handler to use "NonCachingRotatingFileHandler" and configure it to have max size/number of chunks https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/advanced-logging-configuration.html#custom-logger-for-operators-hooks-and-tasks

Airflow uses standard logging facility of Python and it is configurable via local settings as described above. In this case simply using handler that limits size and number of files tha the log rotates throgh as "task" handler should do the job. RotatingFileHandler is a standard handler in Python and we even have a handler that is setting the "non-cache" bit on the files created so that they are not cached by the filesystem - we are using it by default in DAG file processor to limit ever growing cache of log of files on POSIX systems: https://github.com/apache/airflow/blob/main/airflow/config_templates/airflow_local_settings.py#L159

[HuanjieGuo]

Thx @potiuk , the information is real helpful.

I read the code and find that, currently the log mechanism and init parameters for FileTaskHandler and NonCachingRotatingFileHandler is totally different.

I opened a PR to change the handler in FileTaskHandler from NonCachingFileHandler to NonCachingRotatingFileHandler, all the added parameters have default values (will not break the existed case.)

Pls help review: https://github.com/apache/airflow/pull/41064/files