Permissions of new log directory not being set to parent folders. #37200
Description
Apache Airflow version
2.8.1
If "Other Airflow 2 version" selected, which one?
No response
What happened?
When creating log dir/file structure, permissions are set only for the leaf folder.
As per pathlib documentation:
`Create a new directory at this given path. If mode is given, it is combined with the process’ umask value to determine the file mode and access flags. If the path already exists, FileExistsError is raised.
If parents is true, any missing parents of this path are created as needed; they are created with the default permissions without taking mode into account (mimicking the POSIX mkdir -p command).`
Only the leaf folder is created with the specified permisions. Therefore, if logging structure is specified and contains other than leaf folders (such as the default one), the permissions to not get propagated, resulting in a permission denied error if, for example, a celery worker running on a different machine (while in the same group with same GID) tries to write to a directory already created by a task created by a worker on a different machine.
What you think should happen instead?
No response
How to reproduce
- Use file logging in airflow.
- Use the default formatting of log files.
- Create a worker on a remote machine (in my case via celery on redis) with its specific queue.
- Run any task on this queue. Two things can happen:
a.) If this task is the first in the DAG, other workers won't be able to write into logs of this attempt
b.) If this task is not the first in the DAG, then the logs of this task won't be able to be written.
Operating System
Debian GNU/Linux 11 (bullseye)
Versions of Apache Airflow Providers
No response
Deployment
Virtualenv installation
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct