fix(storage): close the upload file handle when the request fails - #1575
Open
tushardev-365 wants to merge 1 commit into
Open
fix(storage): close the upload file handle when the request fails#1575tushardev-365 wants to merge 1 commit into
tushardev-365 wants to merge 1 commit into
Conversation
When a caller passes a path rather than an open file, _upload_or_update and upload_to_signed_url open the handle themselves and hand it to _request inside the files dict. The close lived after the except HTTPStatusError block, so it only ran when the request succeeded: any non-2xx raised StorageApiError straight past it and the handle was left open, with no way for the caller to reach it. It leaks whenever the exception is retained (logging with exc_info, collecting failures in a batch upload loop, an error reporter), because the traceback keeps the frame holding the reader alive. Python also emits ResourceWarning: unclosed file on that path. Move the close into a finally so it runs on both paths. The condition is unchanged, so only handles storage3 opened itself are closed and a caller-supplied stream is still left alone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
When a caller passes a path instead of an already-open file, storage3 opens the handle itself:
_requestthen closed it after theexcept HTTPStatusErrorblock, so the close only ran when the request succeeded. Any non-2xx raisedStorageApiErrorstraight past it and the handle stayed open — and since it only ever existed inside_request's localfilesdict, the caller has no way to close it either.It leaks whenever the exception is retained: logging with
exc_info, collecting failures in a batch-upload loop,raise ... from, an error reporter. The traceback keeps the frame holding the reader alive. Python also emitsResourceWarning: unclosed fileon that path.Counting open fds against a mock transport returning 409 (the most common upload failure), with the error retained each time:
Both
upload()/update()(via_upload_or_update) andupload_to_signed_url()are affected.Fix
Move the close into a
finallyso it runs on both paths. The condition is unchanged — still onlyBufferedReader, i.e. only handles storage3 opened itself, so a caller-supplied stream is still left alone to close as they see fit.Tests
test_upload_closes_file_handle_on_error(async + sync): mock the transport into an error, then assert the handle storage3 opened is closed. Reverting only the source change fails it on.closed is False.Note
The diff on
file_api.pylooks larger than it is — wrapping the body intryreindents it. The only behavioural change is where the close happens._syncis generated, so the change was made in_asyncand regenerated withmake build-sync; unrelated regenerated files are excluded.This does not touch the
resp.textline a few lines above (#1563), which has PRs open already — different defect, different code path.