RangeError - when adding just over 2,400 files

[hashitha]

I am getting RangeError when I add more than 2400 files, Total size of the final zip file would be around 1.3gb. Works fine for anything less than that.

 zip.generateAsync({ type: "blob"  }, function onUpdateCallback(metadata) {
                       console.log(metadata);  //This goes to 100% then RangeError
                    }
                    ).then(function (content) { 
                        saveAs(content,  "test.zip");
                      });

[dduponchel]

To generate a 1.3GB zip file with generateAsync, JSZip currently needs 3 times this amount:

• one time for the original, non compressed content
• one time for the intermediary chunks
• one time for the final zip file

The first one could be mitigated by fixing #290. The second one could be the easiest improvement. The third one is needed by generateAsync.

Could you describe your use case ?
What are these 2400 files ? Files generated client side or real files from a <input type="file" multiple /> ? Do you add them as a Blob, as an ArrayBuffer, as a String, etc ?

[hashitha]

Thanks for the info,

Basically we are writing a medical app that downloads patient CT, Ultrasound, X-Ray, MRI etc.
And some CTs and MRIs can have over 2k images (DICOM files). So when the user clicks download I am using JSZipUtils.getBinaryContent to download the individual images and the combine all those images to one zip file (without compression) using JSZip. This work really well until we hit this limit, then it throws exception at the very end just after onUpdateCallback returns 100%. I am adding them as blob.

We have written all our back-end code in aws lambda, and lambda has a limitation of 500mb disk space so i am unable to compress it from there and serve. Our only other option is to have a server running which I really don't won't to do after all the work we have done to go server-less.

[dduponchel]

Do you have a constraint on the browser ? Using generateInternalStream instead of generateAsync would remove a lot of memory pressure but needs a browser supporting web streams (only chrome right now). Then, with StreamSaver.js, save the stream. The stream API are not compatible, but it should be easy to convert them (disclaimer: I never tested this code):

var writeStream = streamSaver.createWriteStream('output.zip');

zip
.generateInternalStream({type:"uint8array"})
.on('data', function (data, metadata) {
    writeStream.write(data);
})
.on('error', function (e) {
    writeStream.abort(e);
})
.on('end', function () {
    writeStream.close();
});
.resume();

Without streams, I don't see any magical solution. I'm modifying the accumulate method to improve memory usage with blobs (which seem to be moved from the memory under pressure) but the results will depends on the browser implementation.

[hashitha]

I have tried it with the latest chrome and I keep getting "WritableStream is not defined", event using the examples of StreamSaver doesn't work.

I think our only other option is to zip part by part. e.g. every 1,000 images we create one zip file and give to user part1.zip, part2.zip etc. It is of an inconvenience to the user but I don't see any other way at this stage.

[jimmywarting]

I keep getting "WritableStream is not defined"

@hashitha you need to include the web-streams-polyfill to work

[hashitha]

Thanks @jimmywarting , I tried that now getting

StreamSaver.js:122 Something went wrong! TypeError: writeStream.write is not a function

after reading mitm I am now using 127.0.0.1 to test still not working.

This is the code I am using

<script src="lib/StreamSaver.js"></script>
<script src="lib/polyfill.min.js"></script>
<script type="text/javascript">
  var readable = new window.ReadableStream;
</script>

      var writeStream = streamSaver.createWriteStream('output.zip');
      zip.generateInternalStream({ type: "uint8array" })
        .on('data', function (data, metadata) {
          console.log(metadata);
          writeStream.write(data);
        })
        .on('error', function (e) {
          writeStream.abort(e);
        })
        .on('end', function () {
          writeStream.close();
        }).resume(); 

[jimmywarting]

Hmm, they have changed the spec since i wrote this... going to work on updating the docs

looks like you now have to call .getWriter()

So something along this:

var writeStream = streamSaver.createWriteStream('output.zip').getWriter();

[jimmywarting]

Updated the readme/example/dependencies with how you should deal with the new spec
You just do what i wrote above

var writeStream = streamSaver.createWriteStream('output.zip').getWriter();

[hashitha]

I have tried it and it seems to work except its very unstable. Browser tab crashes at least 50% of the time.

[jimmywarting]

Hmm, need to try it myself to figure out what is wrong. Do you have some sample code/files I can try out?

[hashitha]

This is a snippet similar to what I use, only different is that you need to specify the urls, I had to remove the links because they were patient files. You can add around 5 links and run it.

<script src="lib/StreamSaver.js"></script>
<script src="lib/polyfill.min.js"></script>
<script type="text/javascript">
  var readable = new window.ReadableStream;
</script>
<script src="lib/jszip.js"></script>
<script src="lib/jszip-utils.js"></script>
<script src="lib/FileSaver.js"></script>

 var zip = new JSZip();
  var urlList = []; // add the urls here
  var downloadCount = 0;
  var totalCount = urlList.length;
  urlList.forEach(function(url) {
      //download each file
      JSZipUtils.getBinaryContent(url, function(err, data) {
          if (err) {
              console.log(err);
          }
          zip.file(downloadCount + ".dcm", data, {
              binary: true
          });
          downloadCount++;
          if (downloadCount == totalCount) {
              /* old code
                            zip.generateAsync({
                                type: "blob"
                            }, function onUpdateCallback(metadata) {
                                //console.log(metadata);
                            }).then(function(content) {
                                // see FileSaver.js
                                saveAs(content, studyDetails.patientName + ".zip");
                            }); 
              */
              var writeStream = streamSaver.createWriteStream('output.zip').getWriter();
              zip.generateInternalStream({
                      type: "uint8array"
                  })
                  .on('data', function(data, metadata) {
                      //console.log(metadata);
                      writeStream.write(data);
                      /*
                                            if (metadata.percent == 100) {
                                                console.log("END");
                                                writeStream.close();
                                            }
                                            */
                  })
                  .on('error', function(e) {
                      console.log(e);
                      writeStream.abort(e);
                  })
                  .on('end', function() {
                      console.log("END");
                      writeStream.close();
                  }).resume(); 
          }
      });
  });

[jimmywarting]

This is what i played around with: https://jsfiddle.net/Lxwvco3u/
seems to work fine all the time

[jimmywarting]

When i added node stream + node buffer into the fiddle then i manage to get rid of the most memory and i was able to start saving the zip file right away and i could see the progress as i downloaded

https://jsfiddle.net/Lxwvco3u/1/

it would truly help if jszip supported Web streams (or at least the .getReader())

---

The key feature here is that you want to fetch the next binary file when jszip is ready to accept more data, the only way i was able to do that was with .rs._read() cuz that is the only stuff that says "hey i need more data"

Then the only thing you are going to keep in the memory then is the current file you are compressing

• you pipe an ajax to jszip
• jszip compresses it
• jszip flush it to the streamSaver (on to the disk) dos releasing memory
• the data from the ajax is garbage collected
• and then you repeat the process for the rest of the files

Hence why i created #345 #346