Skip to content

support streaming content from S3, start using it in the webserver #2849

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 1, 2025

Conversation

syphar
Copy link
Member

@syphar syphar commented Jun 25, 2025

First step of streaming support for content. Adds streaming support to our storage backends (only for testing in the db backend, so no real streaming there). Then uses axum's streaming response feature to stream the responses to the client.

Idea is also to give up size limits since we don't have memory issues any more.

Next steps after this:

  • also support rustdoc html content by using the streaming functionality of lol-html. This needs some wrapping since lol-html doesn't do async yet. Probably using the rendering background thread & some channels.
  • start with streaming uploads too.

My guess is that all content rendered via askama can't really be streamed, perhaps I switch to a manual streaming impl for bigger streamable things like the sitemap endpoints.

@syphar syphar requested a review from a team as a code owner June 25, 2025 19:04
@github-actions github-actions bot added the S-waiting-on-review Status: This pull request has been implemented and needs to be reviewed label Jun 25, 2025
Copy link
Member

@GuillaumeGomez GuillaumeGomez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apart from the nit, looks good to me!

@syphar
Copy link
Member Author

syphar commented Jun 27, 2025

clippy errors are fixed separately in #2850

Copy link
Member

@GuillaumeGomez GuillaumeGomez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Much simpler indeed, thanks!

@syphar syphar merged commit c1135d1 into rust-lang:master Jul 1, 2025
9 of 10 checks passed
@github-actions github-actions bot added the S-waiting-on-deploy This PR is ready to be merged, but is waiting for an admin to have time to deploy it label Jul 1, 2025
@syphar syphar deleted the storage-stream branch July 1, 2025 06:42
@github-actions github-actions bot removed the S-waiting-on-review Status: This pull request has been implemented and needs to be reviewed label Jul 1, 2025
@syphar syphar removed the S-waiting-on-deploy This PR is ready to be merged, but is waiting for an admin to have time to deploy it label Jul 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants