Consuming alot of RAM

[mohamedjmustafa]

Iam using JSzip to zip a big file (total about 300MB) , i have noticed a huge consumption of RAM about 6 - 6.5 GB , then the program crashed due to memory leak !!!

also when iam trying to zip a small files (les than 5 MB) , it consumes about 350 - 500 MB of RAM !!!

[dduponchel]

That a lot of memory.
Could you share the code sample using JSZip ? (and the files used if possible).
Also, what is the browser or the version of nodejs you use ?

[mohamedjmustafa]

iam working on a javascript windows store application
and this is the code :

function zipFolder(foldername) {
zip = new JSZip();
//get all folder and subfolders
var options = new Windows.Storage.Search.QueryOptions(Windows.Storage.Search.CommonFileQuery.defaultQuery, ['*']);
options.folderDepth = Windows.Storage.Search.FolderDepth.deep;
Windows.Storage.KnownFolders.documentsLibrary.getFolderAsync("eduBookData").done(
function (eduBookData) {
eduBookData.getFolderAsync(foldername).done(
function (myFolder) {
//get all file ...
myFolder.createFileQueryWithOptions(options).getFilesAsync().then(
function (files) {
fileindex = 1;
files.forEach(
function (file) {
getFileAsUint8Array(file).done(
function (Filecontent) {
spath = file.path.split("");
folderindex = spath.indexOf(myFolder.name) + 1;
var fpath = spath[folderindex]
for (i = folderindex + 1; i < spath.length ; i++) {
fpath += "" + spath[i] //generate the path of each file inside the zip file
}
zip.file(fpath, Filecontent);
//document.getElementById('mylog').innerHTML += fpath + "
";
if (fileindex == files.length) {
content = zip.generate({ compression: 'DEFLATE', type: 'uint8array' });
var buffer = Windows.Security.Cryptography.CryptographicBuffer.createFromByteArray(content);
eduBookData.createFileAsync(myFolder.name + ".zip",
Windows.Storage.CreationCollisionOption.replaceExisting).done(
function (myFile) {
return Windows.Storage.FileIO.writeBufferAsync(myFile, buffer).then(
function () {
new Windows.UI.Popups.MessageDialog("Writing Done! ", "Zip").showAsync().done();
});
});
}
fileindex++;
});
});
});
});
});
}

function getFileAsUint8Array(file) { //convert the file to uint array
return Windows.Storage.FileIO.readBufferAsync(file).then(function (buffer) { //Read the file into a byte array
var fileContents = new Uint8Array(buffer.length);
var dataReader = Windows.Storage.Streams.DataReader.fromBuffer(buffer);
dataReader.readBytes(fileContents);
dataReader.close();
return fileContents;
});
}

[dduponchel]

I tested with nodejs (just to do a quick test, I will investigate further tonight) with a 300MB random file (as Uint8Array and generating the result like you).
I start with 300MB of memory used and finish with 600MB (half for the original file, half for the generated zip file).
I do get a huge memory consumption (about 3GB) when loading the file in memory and converting it to a Uint8Array (don't know why...). Maybe you have the same issue : could you check the same code with the zip.generate() line commented out ?

[mohamedjmustafa]

i have commented it , no RAm consumption ...
the consumption happens when generating the zip file
zip.generate()

[jamshid]

Hi, I'm also interested in using this library to generate a zip file dynamically that is downloaded by the user. Sorry but it's not clear from documentation -- is zip.generate() supposed to be usable (at least in modern non-IE browsers) for large zip files? Maybe http://stuk.github.io/jszip/documentation/howto/write_zip.html could clarify browser memory usage -- i.e. can it stream the contents of the zip as it's generated, or will the zip file always need to be loaded in memory?

[dduponchel]

@mohamedjmustafa I tested in IE11 on Windows 8.1 (I think it's the closest I can go without installing Visual Studio in a VM) and with a 300MB file, I get an Out of Memory message from IE at about 700MB after slowly eating memory. I still don't know why JSZip takes 6GB in your case but even 700MB in this case is too much. Your issue mainly comes from JSZip which holds everything in memory and doesn't support (yet !) zip streaming.

@jamshid the current implementation loads everything in memory so no, JSZip isn't suited (yet) for large zip files (hundreds of MB).

I'm working on async and streams features and this should come "soon" (I'm bad at estimating the remaining time but I'm over 80% completion).

[jamshid]

That's exciting, thanks for your work on this library, async will be awesome.
--Jamshid
On Wednesday, May 21, 2014 4:35 PM, David Duponchel notifications@github.com wrote:

@mohamedjmustafa I tested in IE11 on Windows 8.1 (I think it's the closest I can go without installing Visual Studio in a VM) and with a 300MB file, I get an Out of Memory message from IE at about 700MB after slowly eating memory. I still don't know why JSZip takes 6GB in your case but even 700MB in this case is too much. Your issue mainly comes from JSZip which holds everything in memory and doesn't support (yet !) zip streaming.
@jamshid the current implementation loads everything in memory so no, JSZip isn't suited (yet) for large zip files (hundreds of MB).
I'm working on async and streams features and this should come "soon" (I'm bad at estimating the remaining time but I'm over 80% completion).
—
Reply to this email directly or view it on GitHub.

[mohamedjmustafa]

you get out of memory message because you are using a web browser which allow a limited RAM consumption , while the application have no limit in RAM , it can use whole the free memory until it filled totally , it will give an out of memory message and crashes

note : it also takes too much time to zip a big files

any way thank you for replying to this issue and i wish to try to fix it as soon as you can :)

[davidnormo]

@dduponchel any update on the streams front?

[dduponchel]

@davidnormo Not a lot, unfortunately.

[davidnormo]

That's a pity. I'm in the process of switching to zip.js because of handling larger files. JSZip has been great otherwise!

[amita-trantor]

I am also facing out of memory issue with jszip. Any update on this ?