Skip to content

Explicitly close log file descriptor in the supervise function #51627

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
ashb merged 8 commits into from
Jun 12, 2025
Merged

Explicitly close log file descriptor in the supervise function #51627

ashb merged 8 commits into from
Jun 12, 2025

Conversation

@adilkhash
Copy link
Contributor

@adilkhash adilkhash commented Jun 11, 2025

Airflow doesn't close log file descriptor properly hence leading to too many open files error from the operating system.

closes: #51624

@kaxil kaxil added this to the Airflow 3.0.3 milestone Jun 11, 2025
@adilkhash
Copy link
Contributor Author

adilkhash commented Jun 11, 2025 •
edited
Loading

@kaxil hi, do I need to write tests here? (looks like as a non-trivial case)

@kaxil
Copy link
Member

kaxil commented Jun 11, 2025 •
edited
Loading

Yeah but Mypy is failing though:


task-sdk/src/airflow/sdk/execution_time/supervisor.py:1580: error: Incompatible
types in assignment (expression has type "TextIOWrapper", variable has type
"Optional[BufferedWriter]")  [assignment]
                log_file_descriptor = log_file.open("a", buffering=1)
                                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
task-sdk/src/airflow/sdk/execution_time/supervisor.py:1581: error: Argument 1
to "WriteLogger" has incompatible type "Optional[BufferedWriter]"; expected
"Optional[TextIO]"  [arg-type]
    ...ing_logger: WrappedLogger = structlog.WriteLogger(log_file_descriptor)
                                                         ^~~~~~~~~~~~~~~~~~~
Found 2 errors in 1 file (checked 110 source files)

Error 1 returned

@kaxil kaxil added the affected_version:3.0 Issues Reported for 3.0 label Jun 11, 2025
ashb
ashb reviewed Jun 12, 2025
View reviewed changes
@adilkhash @ashb
removed noisy log message
01b513d 
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
@adilkhash
Copy link
Contributor Author

adilkhash commented Jun 12, 2025

@ashb @uranusjr thanks for quick review. failing tests were fixed

@ashb ashb merged commit 4a0a89b into apache:main Jun 12, 2025
71 checks passed
@ashb ashb added the backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch label Jun 12, 2025
github-actions bot pushed a commit that referenced this pull request Jun 12, 2025
@adilkhash
[v3-0-test] Explicitly close log file descriptor in the supervise fun…
1a7beb2 
…ction (#51627)

We didn't close log file descriptor properly hence leading to too many open files error from the operating system.
(cherry picked from commit 4a0a89b)

Co-authored-by: Adylzhan Khashtamov <adil.khashtamov@gmail.com>
@github-actions
Copy link

github-actions bot commented Jun 12, 2025

Backport successfully created: v3-0-test

Status Branch Result
✅ v3-0-test PR Link

kaxil pushed a commit that referenced this pull request Jun 12, 2025
@github-actions @adilkhash
[v3-0-test] Explicitly close log file descriptor in the supervise fun…
86397cf 
…ction (#51627) (#51654)

We didn't close log file descriptor properly hence leading to too many open files error from the operating system.
(cherry picked from commit 4a0a89b)

Co-authored-by: Adylzhan Khashtamov <adil.khashtamov@gmail.com>
choo121600 pushed a commit to choo121600/airflow that referenced this pull request Jun 14, 2025
@adilkhash @choo121600
Explicitly close log file descriptor in the supervise function (apach…
b7eb4a8 
…e#51627)

We didn't close log file descriptor properly hence leading to too many open files error from the operating system.
@adilkhash adilkhash deleted the adilkhash-patch-1 branch October 7, 2025 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashb ashb ashb approved these changes

@uranusjr uranusjr uranusjr approved these changes

@kaxil kaxil Awaiting requested review from kaxil kaxil is a code owner

@amoghrajesh amoghrajesh Awaiting requested review from amoghrajesh amoghrajesh is a code owner

Assignees

No one assigned

Labels

affected_version:3.0 Issues Reported for 3.0 backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch

Projects

None yet

Milestone

Airflow 3.0.3

Development

Successfully merging this pull request may close these issues.

File descriptor leak: task log files not closed, causing 'Too many open files' in Airflow

4 participants

@adilkhash @kaxil @ashb @uranusjr