feat: uploadCarWithStat avoids copying stat.carBytes via Blob construction #2543
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… and instead wraps the stat.carBytes Uint8Array as a BlobLike
Deploying nft-storage with
|
| Latest commit: |
b51165f
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://d8df609d.nft-storage-1at.pages.dev |
| Branch Preview URL: | https://w3up-nocopy-1711993230.nft-storage-1at.pages.dev |
travis
approved these changes
Apr 1, 2024
travis
pushed a commit
that referenced
this pull request
Apr 3, 2024
🤖 I have created a release *beep* *boop* --- ## [4.5.0](api-v4.4.3...api-v4.5.0) (2024-04-02) ### Features * POST /uploads can write uploads to w3up for web3 serving and dagcargoless filecoin deal-making ([#2522](#2522)) ([0c77cb3](0c77cb3)) * uploadCarWithStat avoids copying stat.carBytes via Blob construction ([#2543](#2543)) ([bcb67a3](bcb67a3)) ### Bug Fixes * add a fake piece hasher to try to fix staging upload issues ([#2542](#2542)) ([542983c](542983c)) * avoid car repack ([#2551](#2551)) ([b0d622e](b0d622e)) * avoid car repack ([#2552](#2552)) ([b4ff571](b4ff571)) * change to post method ([#2547](#2547)) ([65e20fb](65e20fb)) * enable `url_standard` compatbility flag ([#2540](#2540)) ([bdde10a](bdde10a)) * enable the `url_standard` compatibility flag ([#2537](#2537)) ([e822bab](e822bab)) * enabled nodejs compat ([#2545](#2545)) ([654cde7](654cde7)) * log out proof ([225cf28](225cf28)) * log proof for debugging ([#2535](#2535)) ([0ecace3](0ecace3)) * more logging ([#2536](#2536)) ([11ae646](11ae646)) * remove debugging console.logs and nodejs_compat flag ([#2553](#2553)) ([6ceb299](6ceb299)) * Revert "fix: try older w3up client" ([#2549](#2549)) ([b31847b](b31847b)) * revert debugging prs ([#2539](#2539)) ([ba22d01](ba22d01)) * try older w3up client ([#2548](#2548)) ([5d94ec5](5d94ec5)) * upgrade @ipld/dag-cbor ([#2534](#2534)) ([964a12c](964a12c)) ### Other Changes * add a bunch of console.logs ([#2544](#2544)) ([2fc34f3](2fc34f3)) * expose simple route to debug ([#2546](#2546)) ([8a9aef5](8a9aef5)) * simple test case to check limits ([#2550](#2550)) ([6a50a70](6a50a70)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
… and instead wraps the stat.carBytes Uint8Array as a BlobLike
Context
Motivation:
new Blobcould get rid of a memory allocation that could be contributing to 'out of memory'uploadCARhere and not a whole blob, there will not be downstream copies of the underlying bytes because this car BlobLike, once passed touploadCARwill get passed to aCAR.BlockStreamthat streams out blocks without re-buffering the whole car bytes in memory https://github.com/web3-storage/w3up/blob/main/packages/upload-client/src/index.js#L107