[Question]: Auto format detection

[levicki]

@rikyoz

What it will do if you pass file.exe to it?

For example an NVIDIA driver package which is a self-extracting 7-Zip archive:

https://us.download.nvidia.com/Windows/595.79/595.79-desktop-win10-win11-64bit-international-dch-whql.exe

[rikyoz]

Hi!

What it will do if you pass file.exe to it?

For example an NVIDIA driver package which is a self-extracting 7-Zip archive:

SFX archives are detected as generic Windows PE executables rather than as archives.

Unfortunately, SFX archive detection is more complex than that of other archives, since SFX archives are not a separate archive format.
SFX archives are essentially Windows executables (PE format) that are immediately followed by archive data bytes.
Therefore, detection requires parsing the PE format headers and calculating the size of the entire SFX executable stub before the archive, as it can vary. Once the size of the stub has been found, we need to seek to the end of it and then detect the format of the archive tail data using its signature.

When I wrote the format detection code, I was unaware of all this and could find very little information on the subject, so I gave up.
I will definitely work on implementing SFX format detection, although I would like to refactor the detection code first.
I plan to implement it in the stable v4.1, but probably not in the beta, which I hope to release soon.

[levicki]

Well the thing is that most SFX executables just append the archive to the executable like this NVIDIA driver package for example.

Note the offset 0x0010C400 with ;@Install@!UTF-8! header of the 7-Zip SFX script.

---

- Now note the offset `0x0010CA24` with `7z` archive header concatenated after it.

Now note the last section (.rsrc) raw size (0x00030A00) and raw address (0x000DBA00) — if you add them you get 0x0010C400, sounds familiar?

Meaning you can't just parse the PE section header and be done with it — you'd have to find the end of last section and if file is longer than that assume that the archive starts ~there.

On a side note, while fiddling with all this today I made a stupid mistake of having Format::SevenZip hard-coded and tried to extract file.zip. The error code I got back is 1 (Fail in your enum). Doesn't really tell you much as to what is going on.

[rikyoz]

Meaning you can't just parse the PE section header and be done with it — you'd have to find the end of last section and if file is longer than that assume that the archive starts ~there.

Oh yeah, I didn't want to put much details in my previous comment, but yes, it's not just parsing the PE headers. And I also want to support RAR and ZIP SFX archives.

On a side note, while fiddling with all this today I made a stupid mistake of having Format::SevenZip hard-coded and tried to extract file.zip. The error code I got back is 1 (Fail in your enum). Doesn't really tell you much as to what is going on.

This is strange! I tried to replicate the issue using the same file, but in my case I can extract it without problems (I tested both with the original archive name, and after naming it file.zip):

BitArchiveReader reader{ lib, R"(E:\Downloads\595.79-desktop-win10-win11-64bit-international-dch-whql.exe)", BitFormat::SevenZip };
reader.extractTo(  R"(E:\Downloads\whql)" );

[levicki]

This is strange! I tried to replicate the issue using the same file, but in my case I can extract it without problems (I tested both with the original archive name, and after naming it file.zip):

Perhaps I wasn't clear -- it was an actual ZIP file, not renamed 7-Zip file and format was set in code which initializes extractor:. I will see if I can make a repro.

[rikyoz]

Oh, I see. Sorry, I had completely misunderstood.
The problem is that, when the library provides 7-Zip with the wrong format for the input archive, 7-Zip simply reports E_FAIL S_FALSE (which is equal to 1).
On the bit7z side, this error could mean anything, and it cannot infer what was the actual issue without some extra data, and so it reports the same error back to the user.
Unfortunately, there's not much bit7z can do in this case to improve the error message.
The only way I can think of is to read the file signature after we get the error from 7-Zip, and check if it corresponds to the signature of the format passed by the user.
This could help provide a better error message, with the overhead of having to read the file again, though.

[levicki]

What I would expect is if I pass Format::SevenZip at extractor creation and then pass a file with .zip extension to get a more sensible error message -- perhaps not even try decompression unless the extension matches format requested?

[rikyoz]

There are some problems with performing the check before even calling 7-Zip APIs.

First of all, this breaks the current behavior of the library.
The current behavior is that bit7z assumes that a user passing Format::SevenZip explicitly is saying "I know this is a 7z archive". Refusing to try based on the extension contradicts their explicit intent. Also, files can legitimately have wrong, missing, or non-standard extensions for reasons we cannot foresee, so the library currently allows that, and at most does some checks after the failed open.

Another reason is that you need a mapping between extensions and formats. Currently, bit7z has such mapping only when BIT7Z_AUTO_FORMAT and BIT7Z_DETECT_FROM_EXTENSION are ON. When they are disabled, this mapping is not included in the library to reduce the memory usage. We could do the check only when those options are ON, but then the library behavior would not be consistent. On the other hand, always including the mapping would add some overhead to even users who do not need such a check, as they already know the correct format of their archives.

Finally, not all inputs have paths. Buffer and stream constructors don't have a file name at all, so an extension check simply wouldn't apply there, but an error in the format chosen can happen as well in these cases.

What I'm looking for is a more general approach that improves error messages in all cases.

In fact, I've already worked on this today (f1a04c2): the next version will report more specific errors (e.g., Invalid archive, or wrong format used).

I found out that 7-Zip actually provides a way to retrieve some internal error flags that can be queried to check what was the cause of the opening error. With this approach, there's no need to re-open the file to read the signature, as I previously thought.

Of course, even more detailed error messages could be better (e.g., "Expected 7z archive, got zip"), but for the time being I think the change is already a good improvement.

[levicki]

I found out that 7-Zip actually provides a way to retrieve some internal error flags that can be queried to check what was the cause of the opening error. With this approach, there's no need to re-open the file to read the signature, as I previously thought.

Hey that's great and I think that will be enough (much better than just getting 1), I agree with all your other points anyway.

I for one don't need verbose error messages but I hate when libraries (not just yours) collapse multiple errors behind a single code.

[levicki]

@rikyoz A small update — it turns out 7za.dll can't extract node.js release zip for some reason (I get 0x80040111 which is CLASS_E_CLASSNOTAVAILABLE) when I try extracting it, with 7z.dll it works. Just a heads up in case someone asks about it.

And error 1? It seems to be not being able to load the dll. Why and how its 1 and not proper HRESULT (such as 0x8007007E when its not found or 0x800700C1 when it's bitness mismatch) beats me.

[rikyoz]

A small update — it turns out 7za.dll can't extract node.js release zip for some reason (I get 0x80040111 which is CLASS_E_CLASSNOTAVAILABLE) when I try extracting it, with 7z.dll it works. Just a heads up in case someone asks about it.

This is expected: 7za.dll supports only the 7z format. For zip files, you need the 7z.dll. You can find the features and formats supported by each 7-Zip DLL in the wiki.

And error 1? It seems to be not being able to load the dll. Why and how its 1 and not proper HRESULT (such as 0x8007007E when its not found or 0x800700C1 when it's bitness mismatch) beats me.

Sorry, I'm not sure I follow: in which case do you get error 1?

A DLL-load failure should not produce 1: bit7z reports the GetLastError() value after LoadLibraryW, wrapped into an HRESULT via HRESULT_FROM_WIN32. So a missing DLL would surface as 0x8007007E and a bitness mismatch as 0x800700C1, exactly the codes you mention.

[levicki]

A DLL-load failure should not produce 1

I have to check this more thoroughly and get back to you. I had a situation where I would just get 1 as error and when I put full path to 7z.dll in Program Files\7-Zip it worked but refused to work from same folder of the executable (this could be a C# quirk).

Also I hit an issue where I have "directory not found" after replacing 7za.dll with 7z.dll without making any other changes. Need to debug that one too — I'll let you know.

[levicki]

Ok, the second issue is caused by format detection. I changed it to auto here and now instead of extracting .7z from .exe it tries to extract .rdata section out of it.