Skip to content

Support async file objects in content= and files= - #1145

Open
tsimoshka wants to merge 4 commits into
pydantic:mainfrom
tsimoshka:async-file-uploads
Open

Support async file objects in content= and files=#1145
tsimoshka wants to merge 4 commits into
pydantic:mainfrom
tsimoshka:async-file-uploads

Conversation

@tsimoshka

@tsimoshka tsimoshka commented Aug 18, 2026

Copy link
Copy Markdown

Summary

This is my third attempt at this. This PR resolves #597 and implements chunked upload for anyio.open_file, trio.open_file and aiofiles.open when used as content= parameter for post and put request and implements support for multipart file upload for the same libraries. Previous attempts were encode/httpx#3339 and encode/httpx#3698 against encode/httpx, both went stale while the library was unmaintained. Most of the code is reworked from encode/httpx#3698, with the review comments left on encode/httpx#3339 applied.

Currently content= with an async file iterates it a line at a time instead of reading in chunks, so a file without newlines gets buffered whole, and files= with an async file loops forever on both clients, since MultipartStream.__aiter__ just delegates to the sync iter_chunks() and read() returns a coroutine, which is always truthy.

changes

  • _types.AsyncReadableFile protocol was added along with is_async_readable_file type predicate function to detect and perform type narrowing for trio/anyio/aiofiles async files
  • _types.FileContent was extended to include the _types.AsyncReadableFile protocol
  • _utils.peek_async_filelike_length was added, async wrappers expose a sync fileno() so the fd can be stat'ed, there's no tell()/seek() fallback like the sync version because those would need awaiting
  • _content.AsyncIteratorByteStream updated to use await read() for async files instead of looping over lines
  • _content.encode_content updated to attach content length header for async files when it's known upfront
  • _multipart.FileField updated with arender_data, and get_length / render_data made async-aware, sync Client now raises TypeError instead of looping
  • _multipart.MultipartStream updated with aiter_chunks, and __aiter__ updated to use it, both wrapped in contextlib.aclosing
  • async files must be opened in binary mode, same as sync ones, otherwise the Content-Length taken from os.fstat wouldn't match the encoded body
  • typing_extensions bumped to >=4.10 for TypeIs (already a dep for python < 3.13), lockfile updated
  • tests added to test_content.py, test_multipart.py and test_utils.py, benchmarks added to test_benchmark.py, docs and changelog updated

perf

  • content= over 4 MiB with 64-byte lines: 9.18 ms -> 0.33 ms (encoding path, in-memory file, so no disk I/O in the number)
  • the type predicate runs 0-3 times per request, never per chunk, and short circuits C callables so io.BytesIO / open() cost 57 ns instead of ~630 ns. Benchmarked against a clean main worktree, sync multipart is unchanged within noise.

Checklist

  • I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.

Review in cubic

Read `anyio`, `trio` and `aiofiles` files with `await read()` rather than
iterating a line at a time, and set `Content-Length` when the length is known.
Multipart uploads from an async file previously looped indefinitely over
un-awaited coroutines; they now stream, or raise `TypeError` on a sync `Client`.
@codspeed-hq

codspeed-hq Bot commented Aug 18, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 17 untouched benchmarks
🆕 2 new benchmarks
⏩ 7 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
🆕 Simulation test_bench_async_file_multipart_upload N/A 6.3 ms N/A
🆕 Simulation test_bench_async_file_upload N/A 6.1 ms N/A

Comparing tsimoshka:async-file-uploads (10ad151) with main (2b13fa9)

Open in CodSpeed

Footnotes

  1. 7 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed against the latest diff

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread src/httpx2/httpx2/_multipart.py
Comment thread src/httpx2/httpx2/_content.py
Comment thread tests/test_benchmark.py
Comment thread tests/test_benchmark.py Outdated
Fall back to chunked encoding when an async file reports it can't be rewound,
so `Content-Length` can't disagree with the bytes actually sent. Reject
text-mode async files passed as `content=`, matching the multipart path.
Benchmark async uploads through a descriptor so the length path is covered,
and reuse one event loop per benchmark rather than paying loop bootstrap on
every sample.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread src/httpx2/httpx2/_content.py Outdated
The truthiness guard let an empty `str` read past the binary-mode check, so an
empty text-mode file uploaded silently as zero bytes instead of raising.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support async file types in files = {} and content = ...

1 participant