Getting FILE_ENDED with some routes through code

[richardgroves]

For a given ZIP file (small, ~7KB containing just two files) I can open the file with the OS (macOS 11.1) and the code below has one path that opens and extracts the zip file, the other throws the FILE_ENDED error.

It looks a bit like the issue in #75 as a callstack for the error was very similar.

I'm using the latest npm version of the code - 0.10.11.

I can't share the problem file publicly I'm afraid.

                // Works
		fs.createReadStream(zipName).pipe(unzipper.Extract({ path: '/tmp/unzipper-test' }));

		// Error with FILE_ENDED
		fs.createReadStream(zipName)
		.pipe(unzipper.Parse())
		.on('entry', function (entry) {
			const fileName = entry.path;
			const type = entry.type; // 'Directory' or 'File'
			const size = entry.vars.uncompressedSize; // There is also compressedSize;
			console.log("Entry file " + fileName + " type " + type + " Sz:  " + size);
			entry.autodrain();
	  	});

[richardgroves]

Trying to narrow this down more - the createReadStream route might be working. Having some async issues seeing that I think.

What is definitely is not working is:

let directory = await unzipper.Open.file(zipName)

[dev-brendanprice]

Any updates on this ?

[richardgroves]

@brendanprice2003 Sorry, no. I had to move on to something else and this got left behind. I'll try and dig into the latest code and logs and see if it is still an issue.

[dev-brendanprice]

@brendanprice2003 Sorry, no. I had to move on to something else and this got left behind. I'll try and dig into the latest code and logs and see if it is still an issue.

It still is an issue. I am running the latest version of unzipper yet FILE_ENDED still persists.

I am using unzipper.Extract() directly inside of a response.pipe()

[richardgroves]

@brendanprice2003 Do you have a file that breaks the code that you can share? I was not able to share the ones that broke it for us, and it was only a very small fraction of files that broke.

[dev-brendanprice]

@brendanprice2003 Do you have a file that breaks the code that you can share? I was not able to share the ones that broke it for us, and it was only a very small fraction of files that broke.

The file that breaks the function doesnt make a diffefence. It was simply a case of it working one day then the next its not. All I am using is a zip folder of an Assetto Corsa car.

But yes, I had the above code working just fine one minute then the next it just stopped working for no conceivable reason.

[richardgroves]

Oh - that's very worrying. In the use case I had it was lots of files being sent through the same processing code, and some would fail - I didn't have a good path to retest them, so had assumed it was a problem in the file, not some randomness in the code.

[dev-brendanprice]

Yes, It is worrying. Taking into account that the non-working versions of my application are NOT signed. However when it is tested on my local machine through a compiled app or VScode CLI, it works perfectly fine. So that MAY be a problem but it didn't get in the way when we initially tested it; it worked perfectly.

Here is how I am using the module

for (let item of json) {

      var req = http.get(item.urlpath, async (res) => {

          // Direct the download stream to unzipper function
          res.pipe(
              unzipper.Extract({path: `${gameInstallDir}/content/${item.type}/`})
          );
      });
};

For whatever reason the contents of res.pipe() seized to work randomly.
Please assume that item from json is in working order, as the server only sends back a concatenated string that makes up the download path for the respective items in the json.

If this can't get fixed then I do need an alternative way to unzip a file via the stream as I don't want to be downloading a file to a directory then unzipping, which is then unzipped to another directory.

[dev-brendanprice]

Coming back to this;

I had managed to get this same code working again, by refactoring my server side and how it sends the information back. I had made some changes on the client-side also but I'm sure that isn't the reason why it's working now.

    for (let item of json) {

        // Start file download
        var req = http.get(item.urlpath, (res) => {

            // Direct download stream to unzipper module
            res.pipe(
                unzipper.Extract({
                    path: `${gameInstallDir}/content/${item.type}/`
                })
            );
         }
    };

Some additional information is now being sent by the client to the renderer process (UI). What I would recommend is make sure that the unzipper has the correct and sufficient information. After looking over what the unzipper function is actually receiving I realized that it was wrong.

Thanks.

[abozzi-deloitte]

I get this same exact error but only on large files. For example a file that goes from 90mb to 2 gigs is fine but one that goes from 250mb to 5 gigs will throw this error about half way through.

[dev-brendanprice]

I get this same exact error but only on large files. For example a file that goes from 90mb to 2 gigs is fine but one that goes from 250mb to 5 gigs will throw this error about half way through.

Please review how you are using the file path, for example use static file paths instead of paths that reference a file using ./

I had the same issue, as stated above, however, I cannot remember how exactly I fixed it. The type of files inside of the .zip may also have a play on effect so maybe you could look at adding some sort of call back to log the file type that is currently being extracted.

My guess is probably as good as yours.

Good luck.

[tgbv]

Any updates on this?

[dev-brendanprice]

Any updates on this?

You have probably moved on from this issue by now, but for the sake of future repository viewers;

You should be able to validate what type of files you sending inside of the .zip and breakdown which ones it works with, vice versa. Because of the nature of the problem, I have a feeling that it's not to do with lack of consistency in the module code, rather the data that you are feeding it.

If your problem still persists I suggest looking at other solutions for unzipping archives.
Look, it's Stackoverflow! (Read This Post)

[credcore-dave]

There was an issue with node's zlib integration that was only recently reverted (I'm not sure in which releases): nodejs/node#45268

For us, our unzipper code works fine on node 18, but breaks with FILE_ENDED on node 20. It's very consistent - on certain node versions it always works with any zipfile; on later versions it always breaks. It also seems to be platform dependent.

It's caused a lot of drama and I think there's still no root cause been found besides "something to do with SIMD extensions"