-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEATURE] Announce total length in HTTP header 'Content-Length' #19
Comments
Hello. As I have explained in CONTRIBUTING.md, It would be possible to set content-length without sacrificing memory efficiency if
Is that OK for you ? Then of course |
I have another suggestion, that won't be as accurate as monitoring the download stream but should be close enough and works with the current version of client-zip. If all your input is in Streams (I assume it's Response bodies in your case), you could instead monitor those streams.
|
Thank you for your quick and detailed response. I understand that due to the streaming design of client-zip it is not possible to know the final content length in advance. If I understood you correct, then you proposed the following solution to calculate the total content length? Using this approach, I can calculate the download progress as a percentage even with the current version of client-zip. Thank you for your hint! import { downloadZip } from 'https://cdn.jsdelivr.net/npm/client-zip/index.js';
async function download() {
const file1 = await fetch('https://4sc35swexhjkk3fe-public.s3-eu-west-1.amazonaws.com/elevator-music.mp3');
const file2 = await fetch('https://4sc35swexhjkk3fe-public.s3-eu-west-1.amazonaws.com/elevator-music.mp3');
const file3 = await fetch('https://4sc35swexhjkk3fe-public.s3-eu-west-1.amazonaws.com/elevator-music.mp3');
const file4 = await fetch('https://4sc35swexhjkk3fe-public.s3-eu-west-1.amazonaws.com/elevator-music.mp3');
const files = [file1, file2, file3, file4];
// Sum-up the content lengths
let contentLength = 0;
files.forEach(file => {
contentLength += +file.headers.get('Content-Length');
});
const response = await downloadZip(files);
const reader = response.body.getReader();
let receivedLength = 0;
const chunks = [];
while (true) {
const { done, value } = await reader.read();
if (done) {
break;
}
chunks.push(value);
receivedLength += value.length;
console.log(`Received ${receivedLength} of ${contentLength} Bytes -> ${(receivedLength/contentLength*100).toFixed()} %`);
}
const blob = new Blob(chunks);
const link = document.createElement('a');
link.href = URL.createObjectURL(blob);
link.download = 'test.zip';
link.click();
link.remove();
}
download(); Regarding your second suggestion, could you please explain how to listen to the "end" event of each stream? And yes, simply counting how many files have already been downloaded would be sufficient for my use case, since I download dozens to hundreds of small files (< 1MB). So knowing the count of already downloaded files would be a sufficient indicator of the download progress. |
OK, so first about your code: that's not exactly what I had in mind, but close enough, I suppose. You are comparing the progress of the output generation against the total size of the input files, whereas my suggestion was to monitor the progress of the input consumption using a passthrough TransformStream. In fact, the overhead from the Zip file format isn't very big, and it's predictable so you can make your method more accurate : the exact overhead, without Zip64 (for archives smaller than 4GB), is 92 bytes per file + the length of each filename in UTF8 + 22 bytes at the end. You could approximate to 100-ish bytes per file, depending on your average filename. By the way, you don't need to About the second option: sorry, I thought there were events on WHATWG streams and I was mistaken. The easiest solution I can think of is using Honestly, it sounds like more trouble than it's worth, especially since you already have a working solution. The TransformStream solution is meant to be more efficient if you were generating large archives, but I guess that's not your use case. |
Thank you for your explanations, @Touffy. You're right, I'm satisfied with this working solution. There's no need to extend For others who might want to calculate the download progress, too, this would be a potential solution: import { downloadZip } from 'https://cdn.jsdelivr.net/npm/client-zip/index.js';
async function download() {
const file1 = await fetch('https://4sc35swexhjkk3fe-public.s3-eu-west-1.amazonaws.com/elevator-music.mp3');
const file2 = await fetch('https://4sc35swexhjkk3fe-public.s3-eu-west-1.amazonaws.com/elevator-music.mp3');
const file3 = await fetch('https://4sc35swexhjkk3fe-public.s3-eu-west-1.amazonaws.com/elevator-music.mp3');
const file4 = await fetch('https://4sc35swexhjkk3fe-public.s3-eu-west-1.amazonaws.com/elevator-music.mp3');
const files = [file1, file2, file3, file4];
// Calculate total download size by summing up the HTTP header `Content-Length` + the length
// of the UTF-8 encoded file name + 92 Bytes
let totalBytes = 0;
files.forEach(file => {
const contentLength = +file.headers.get('Content-Length');
const fileNameSizeInBytes = (new TextEncoder().encode('elevator-music.mp3')).length;
totalBytes += contentLength + fileNameSizeInBytes + 92;
});
totalBytes += 22;
const response = downloadZip(files);
const reader = response.body.getReader();
let receivedBytes = 0;
const chunks = [];
while (true) {
const { done, value } = await reader.read();
if (done) {
break;
}
chunks.push(value);
receivedBytes += value.length;
console.log(`Received ${receivedBytes} of ${totalBytes} Bytes -> ${(receivedBytes/totalBytes*100).toFixed()} %`);
}
const blob = new Blob(chunks);
const link = document.createElement('a');
link.href = URL.createObjectURL(blob);
link.download = 'test.zip';
link.click();
link.remove();
}
download(); |
Fetch doesn't have a progress event so can we use xmlhttprequest which does have a progress event? Jake Archibald did a pretty good write up in 2016 : https://jakearchibald.com/2016/streams-ftw/ |
Yes, you could use an XHR to make a custom input for |
@Touffy Thanks for your feedback, i must say you have been an amazing open source contributor for this package. I hope you will one day do a paid course that teaches us about js streams/transforms and other browser related api's. I will happily hand over my money. Most if not all other JS courses teach common already well concepts., but none of them ever go in to the nitty gritty of enterprise tech such as streams, piping, blobs,readers, iterables , generators, etc. Thank you. I am sure i will have more questions to come. oh and jake@google posted an update 2020 video on streams > https://www.youtube.com/watch?v=G9PpImUEeUA |
Sorry for bring up a closed issue, but I did not want to create a duplicate one. For my particular use case, I need to know the final zip size before even starting the process. This is because I need to pipe the Because I know in advance the sizes of the inputs to I could do the calculation manually, but it's cleaner to keep it within |
Hi Sơn. You are right to re-use this issue. Like I said before, it's possible for client-zip to compute the total length in advance if all file lengths and names are known at the start and if the files will be stored without compression (encryption overhead should be predictable, though I haven't looked into it yet). Ideally, to preserve the general streaming design for large archives, you'd want to avoid reading the metadata from the actual file Responses. Instead, you should use a dedicated metadata endpoint (or, at least, HEAD requests) before starting the zipping process. Once you have collected all the metadata, there are two ways to proceed (and both could be implemented) :
|
The list of first argument of I suppose if |
I know it's a bit more complicated, but this way, you can let |
That actually sounds great for my use case! Thanks for building this with streaming in mind! I'm looking forward to have this functionality implemented! |
Actually, I was hasty when saying we can get rid of the streaming flag. That would require also knowing the file's CRC32 in advance. I know some cloud storage services can give you that information in a metadata endpoint, but that's too situational and the benefit is tiny (it would just save 16 to 28 bytes of overhead per file). So, streaming is still always on after all. The new feature will simply let you predict the total length of the zip file by passing an (synchronous only, this time) iterable of file metadata to a new I think In the case of Zip64 archives, the list of file metadata given to |
Should be solved with the newly published versions 1.5 (still distributed as ES2018 and without Zip64) and 2.2. I must say the feature came at a significant cost in kilobytes (well, only about 1kB, but client-zip being so small to begin with, that was a big increase). |
This is awesome! Sorry for asking you to add the extra 1KB to the library, but I think it would add more possibilities to the library, such as piping data from cloud providers (which already have API to return the size) into zip file. When we have the basic password encryption, |
Thank you for your library. I was wondering if it is possible to calculate the download progress as a percentage. To do this, the response object of
downloadZip()
would have to contain the HTTP headerContent-Length
, which it does not at the moment.I have already found that browsers hide HTTP headers from scripts on cross-origin requests (see here) unless the HTTP header
Access-Control-Expose-Headers
says otherwise. So, I adapted my server config to expose theContent-Length
header. As a result, the request to fetch my test file returns the HTTP headerContent-Length
(see (1) in code snippet). But, because the HTTP headerContent-Length
is still missing on the response ofdownloadZip()
(see (2) in code snippet), I cannot calculate a progress in percent (as the absolute value is unknown).So my question is: is it even possible to include the HTTP header
Content-Length
and have it contain the sum of all content lengths of the files to be downloaded?The following code snippet demonstrates how to observe the download progress:
The text was updated successfully, but these errors were encountered: