Read archive in chunks when publishing archive to registry #6979
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
spqw
force-pushed
the
package-registry-read-archive-in-chunks
branch
from
7c61991 to
264170b
Compare
spqw
changed the title
MaxDesiatov
reviewed
Oct 8, 2023
| /// Get the contents of a file without loading all of it in memory | ||
| /// | ||
| /// - Returns: The file contents as Data, or nil if missing | ||
| func readFileInChunks(path: AbsolutePath, bufferSize: Int = 1024) -> Data? { |
There was a problem hiding this comment.
Sorry, I'm not sure I understand what this is trying to achieve. How exactly would this prevent reading the whole file into memory if the resulting Data value contains the whole file in memory due to the fact that all of the chunks are appended to it to form one huge Data buffer?
There was a problem hiding this comment.
You're totally right.
My bad, this is actually far from being ready for review.
Marking it as a draft for now if that's ok.
|
Hi @spqw, would you be interested in addressing the PR review provided above? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add function to read archive in chunks when publishing archive to registry
Motivation:
This helps with memory efficiency and performance for large files.
This is implementing a TODO.
Modifications:
Before: The archive is read with
readFileContents, loading the entire file in memoryModification: The archive is read with a new function called
readFileInChunks, using anInputStream.Result:
The archive is read in chunks.