Skip to content

fix(streaming): return empty bytes when reading at the end of an S3 object - #8511

Merged
leandrodamascena merged 2 commits into
aws-powertools:developfrom
chancesong10:fix/streaming-read-at-end-of-object
Oct 2, 2026
Merged

leandrodamascena merged 2 commits into
aws-powertools:developfrom
chancesong10:fix/streaming-read-at-end-of-object

Conversation

@chancesong10

Copy link
Copy Markdown
Contributor

Issue number: closes #8510

Summary

Changes

_S3SeekableIO opens each stream with Range: bytes=<position>-, and S3 answers 416 InvalidRange when that position is at or past the end of the object, which includes any range on an empty object.

  • raw_stream catches ClientError with code InvalidRange and uses an empty PowertoolsStreamingBody, so reads at the end return b"" like a regular file. Any other error is still raised. This keeps one GetObject per stream, with no extra HeadObject to check the size first.
  • close() only closes _raw_stream if a stream is already open. Before, it went through the raw_stream property, so closing an unread object made a GetObject just to close it, and closing at the end of the object raised.

PowertoolsStreamingBody is now imported at runtime rather than only under TYPE_CHECKING, since it's used to build the empty body.

Tests added to tests/functional/streaming/_boto3/test_s3_seekable_io.py:

  • test_read_at_end_of_object_returns_empty_bytes
  • test_read_empty_object_returns_empty_bytes, covering read() and iteration
  • test_raw_stream_raises_other_client_errors (NoSuchKey still raises)
  • test_context_manager_at_end_of_object

test_closed no longer stubs a get_object. It now checks that closing an unread object makes no request. The three new behaviour tests and the updated test_closed fail on develop.

User experience

Before:

obj = S3Object(bucket="my-bucket", key="data.txt")
obj.seek(0, io.SEEK_END)
obj.read()        # ClientError: InvalidRange

S3Object(bucket="my-bucket", key="photos/").read()  # empty object: ClientError: InvalidRange

with S3Object(bucket="my-bucket", key="data.txt") as obj:
    obj.seek(0, io.SEEK_END)
# ClientError: InvalidRange, raised by close() on exit

After, these return b"", b"", and exit cleanly, the same as io.BytesIO. Closing an unread object no longer calls GetObject.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

@chancesong10
chancesong10 requested a review from a team as a code owner October 1, 2026 06:43
@chancesong10
chancesong10 requested a review from svozza October 1, 2026 06:43
@boring-cyborg

boring-cyborg Bot commented Oct 1, 2026

Copy link
Copy Markdown

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 1, 2026
@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 2, 2026
@sonarqubecloud

sonarqubecloud Bot commented Oct 2, 2026

Copy link
Copy Markdown

@codecov

codecov Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.92%. Comparing base (b1af7e8) to head (164497c).

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #8511      +/-   ##
===========================================
+ Coverage    96.88%   96.92%   +0.04%     
===========================================
  Files          317      317              
  Lines        15899    15910      +11     
  Branches      1395     1398       +3     
===========================================
+ Hits         15403    15421      +18     
+ Misses         361      353       -8     
- Partials       135      136       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@leandrodamascena

Copy link
Copy Markdown
Contributor

Thanks for catching this, @chancesong10! Good find! I pushed a small follow-up after noticing that a stream closed before its first read could reopen on a later read. It now raises ValueError without another S3 request. I also added regression tests covering the read methods and iteration.

@leandrodamascena
leandrodamascena merged commit 3311cc1 into aws-powertools:develop Oct 2, 2026
@boring-cyborg

boring-cyborg Bot commented Oct 2, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request and thank you for helping improve everyone's experience!

@powertools-for-aws-oss-automation

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request and thank you for helping improve everyone's experience!

@chancesong10

Copy link
Copy Markdown
Contributor Author

Thanks @leandrodamascena, glad I could help! If it's alright, I'd like to take on #8348 (dependency injection for middleware), starting with function-based middleware as you suggested. Would you be OK with me opening a PR for it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/L Denotes a PR that changes 100-499 lines, ignoring generated files. streaming tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: S3Object raises InvalidRange when reading at the end of an object or reading an empty object

2 participants