Using large file jszip dies silently

[arjan1234]

Using a large ZIP file (7mb compressed, 430mb uncompressed) the library dies silently:

http://www.cultureelerfgoed.nl/geo/RCE_MIP_objecten_punten_GML.zip

(browser: FF 20.0, desktop:Ubuntu 3.2.0-41-generic, x86_64)

[dduponchel]

Good news, I can reproduce it !
Bad news, I don't know how to fix it (yet).

I think this comes from the insane amount of memory used to read the file. I'm already working on the CPU consumption, the memory consumption will be the next one.

If you are only using only one file at a time, my work could help (lazy decompress files).

[arjan1234]

Ok. I am using only one file at a time. Where can I find your work (lazy decompress)...?

[dduponchel]

You can find it on my branch WIP_perfs here : https://github.com/dduponchel/jszip/tree/WIP_perfs (I just pushed my latest commits).
The API is almost the same, the biggest change is the removal of the data attribute. On my index.html, you should check the "Migrating Guide" and the "Performance issues" parts.
As soon as I'm completely satisfied with the code, I'll do a pull request.

[martingraham]

I've put some changes into a branch I've got that "may" address some issues, it uses an arraybuffer if it can to load in the zip rather than a string (saves 50% memory), and after that tries to avoid making substrings or concatenating strings as much as it can (its happy operating over a uint8array view of the original arraybuffer). It also allows you to use a function to grab parts of files as they are inflated (i.e. so you can filter csv files by row/column, maybe write something to grab parts of zipped xml or json files, it'll take you to write those functions though).

https://github.com/martingraham/jszip/

For the OP, trying to fully decompress a 430MB file will take 860MB of string space (2 bytes a char) in a browser and completely bork all browsers, which is why I've been investigating partial decompressions. I'm trying to grab parts of a 75MB zip (290MB uncompressed) as I know decompressing the whole thing isn't going to work.

The infuriating thing is it seems to completely depend on the browser. Chrome is fast as hell, while Opera doesn't work and I can't tell where it fails. IE9 gets there after a fashion after using an insane amount of memory (mainly due to not doing arraybuffers properly when loading with an xhr), Safari works, Firefox works 90% of the time but for some reason will just go at a snails pace the odd time. They all use completely different amounts of memory too.

Trying to profile any of them when grabbing large chunks of big files crashes every one of them, but profiling when I'm taking smaller chunks shows chrome at least is using less memory as I intended, but it looks like opera at least isn't.

• I integrated my updates into the latest versions of the .js files, even though i forked off back in february, and it passes the new test cases

[dduponchel]

I'm glad to see I'm not the only one struggling with optimization and big files :)
Your work seems promising, I'll definitely look into it !

[martingraham]

PS Opera v 15 now works a lot better re memory as its moved to a webkit engine.

[kwroblewski]

I have been developing a very data processing intensive enterprise level application, so this interests me as well. I use jszip as a way of overcoming the 5MB limit on localstorage in HTML5 by dumping my JSON data to a textfile and zipping it with js zip, then saving the base64 of the zip file to local storage instead. Works great!

However, some of this data has several hundred thousand records, so the text file I create gets rather big.

Anyway, given my experience so far, what about the idea of making jszip apply compression asynchronously? I.e. iterate through the data in chunks, using setIntervals() between chunks, then calling a callback function the user passes in for when it's done? Larger files may still take time, but it would never crash the browse and could even provide percent complete feedback as well.

[dduponchel]

@kwroblewski That's an interesting idea but might require too much work : JSZip has a synchronous API and the library handling the DEFLATE algorithm doesn't support that.

To create asynchronous methods when compressing/decompressing data we would need to duplicate or rework the as{Text,Binary,ArrayBuffer,Uint8Array,Buffer}, load and generate mehods. Being asynchronous has clear benefits (not only for the (de)compression) but I don't know if there is a clean way to do it while keeping a synchronous API.

Second point, the compression is handled by a third party library which doesn't support synchronous AND asynchronous (de)compression. Moreover, our library choice is limited by the IE support.

@Stuk do you have an other point of view on the subject ?

PS : sorry for the double notification, I miss-clicked while redacting this message...

[rejas]

does 83aa6b6 somehow relate to this too (or might fix it even)?

[dduponchel]

Yes, this commit will help (just released as v2.2.1). It won't completely fix this bug but I'm working on it, see #121.

[drhewitt]

I am using JSZIp with a Salesforce app but when I load xlsx files larger than 500kb, I get an import error message. How do I overcome this?

[dduponchel]

What is your code and what browser do you use ?

[drhewitt]

Hi David,

I'm using Salesforce and I'm using the app found here:
https://github.com/financialforcedev/apex-zip

I deployed the app to Salesforce and have used Firefix, Crome and IE but
there seems to be a limit on how big the Excel spreadsheet can be (this is
an xlsx spreadsheet)

Not much more than 400kb, if that.

Is this because JSZip has a 1,000,000 character limit?

Look forward to hearing from you

Daniel

On Wed, Jul 22, 2015 at 1:50 AM, David Duponchel notifications@github.com
wrote:

What is your code and what browser do you use ?

—
Reply to this email directly or view it on GitHub
#48 (comment).