MSZIP decompression issues

[HeroponRikiBestest]

A very small number of MsZip MsCAB sets I have seem to suffer from MsZip decompression issues on ST.IO's end, rather than on ST.Serialization's end. I do not know enough about MsZip to be able to fix it, or even diagnose the issue. They give the "had mismatching sizes" debug output, and in all cases I could see, this was an actual issue that caused incorrect extraction rather than superfluous logging.

The cabs with errors are as follows:

AgainstRome1.cab in http://redump.org/disc/103076/
Data.cab in http://redump.org/disc/113648/
SimCitySocieties/Group2.cab in http://redump.org/disc/103079/

(To be clear, for these next 3, I'm referring to the mszip cabinets inside of the uncompressed cabinets. The uncompressed cabinets containing them are fine, and obviously issues with them wouldn't apply to ST.IO anyways.)

Media1.cab in https://dbox.tools/titles/pc/50580FA1/
Media1.cab in https://dbox.tools/titles/pc/4B590FA8/
Media1.cab and Media2.cab in https://dbox.tools/titles/pc/54510881/

[HeroponRikiBestest]

As requested over discord, I have confirmed that MsZip extraction errors still occur with the current release build of ExtractionTool from https://github.com/SabreTools/SabreTools.Serialization/releases/tag/2.2.1 .

It turns out that specifically SimCitySocieties/Group2.cab (RID 103079): , while it does have mszip errors on both the current release version and still as of my PR, it doesn't result in incorrect extraction like the other cabs I mentioned here (which I have all re-confirmed do result in incorrect extraction). It would be nice if this cab was addressed for safety, but I imagine that's not really a priority.

While the output below shows comparisons between 7z and ST output, this was in part just to be able to make these notes easier. I've confirmed that the 7z output is correct by comparing that against the built-in expand tool in Windows XP. If you need me to confirm a third way, let me know what you would prefer.

One of the easiest ones to see correctness with is AgainstRome1.cab, as one of the differing files, collision.bmp3310, is an image, and thus you can easily view that it's broken with ST's output.

(7z/expand left, ST right)

The following log output is all from ExtractionTool 2.2.1 release. My latest PR log output, from what I observed, seems to give the same data block mismatching sizes error numbers, and the same file data output.

Also, to be clear, Media1.cab and Media2.cab for 54510881 are listed seperately because they aren't spanned, and are both standalone cabs. The only spanned cab set in this issue is Data.cab in RID 113648, which is a 3 volume set that spans Data.cab, Data1.cab, and Data2.cab.

AgainstRome1.cab (RID 103076):

$ ./ExtractionTool -d '/home/bestest/mscabs/mszips/GameStar 07-2008 (Germany) (DVD-XL)/AgainstRome1.cab' -o ST
Checking possible path: /home/bestest/mscabs/mszips/GameStar 07-2008 (Germany) (DVD-XL)/AgainstRome1.cab
Attempting to extract all files from /home/bestest/mscabs/mszips/GameStar 07-2008 (Germany) (DVD-XL)/AgainstRome1.cab
Attempting to extract from 'Microsoft Cabinet'

WARNING: LZX and Quantum compression schemes are not supported so some files may be skipped!
Data block 43495 in folder 0 had mismatching sizes. Expected: 32768, Got: 32638

$ diff -r 7z ST
Binary files 7z/collision.bmp3310 and ST/collision.bmp3310 differ
Binary files 7z/team.dat3311 and ST/team.dat3311 differ

Data.cab (RID 113648):
$ ./ExtractionTool -d '/home/bestest/mscabs/mszips/Test Drive Unlimited (Germany) (En,Fr,De,Es,It)/Data.cab' -o ST
Checking possible path: /home/bestest/mscabs/mszips/Test Drive Unlimited (Germany) (En,Fr,De,Es,It)/Data.cab
Attempting to extract all files from /home/bestest/mscabs/mszips/Test Drive Unlimited (Germany) (En,Fr,De,Es,It)/Data.cab
Attempting to extract from 'Microsoft Cabinet'

WARNING: LZX and Quantum compression schemes are not supported so some files may be skipped!
Data block 4043 in folder 0 had mismatching sizes. Expected: 32768, Got: 2679
System.IO.InvalidDataException: source
at SabreTools.Serialization.Wrappers.MicrosoftCabinet.DecompressMSZIPBlock(Int32 folderIndex, Decompressor mszip, Int32 blockIndex, CFDATA block, Byte[] blockData, Boolean includeDebug)
Data block 41 in folder 0 had mismatching sizes. Expected: 32768, Got: 26900
System.IO.InvalidDataException: source
at SabreTools.Serialization.Wrappers.MicrosoftCabinet.DecompressMSZIPBlock(Int32 folderIndex, Decompressor mszip, Int32 blockIndex, CFDATA block, Byte[] blockData, Boolean includeDebug)
Data block 7 in folder 177 had mismatching sizes. Expected: 32768, Got: 32743

$ diff -r 7z ST
Binary files 7z/_86261BAE7A6B4C45A5D691D3320D5993 and ST/_86261BAE7A6B4C45A5D691D3320D5993 differ

Media1.cab (50580FA1):

$ ./ExtractionTool -d Media1.cab -o ST
Checking possible path: /home/bestest/confirmMs/Media1.cab
Attempting to extract all files from /home/bestest/confirmMs/Media1.cab
Attempting to extract from 'Microsoft Cabinet'

WARNING: LZX and Quantum compression schemes are not supported so some files may be skipped!
Data block 8395 in folder 0 had mismatching sizes. Expected: 32768, Got: 32731
Data block 17378 in folder 0 had mismatching sizes. Expected: 32768, Got: 32541
Data block 17383 in folder 0 had mismatching sizes. Expected: 32768, Got: 32679
Data block 17384 in folder 0 had mismatching sizes. Expected: 32768, Got: 32517
Data block 18764 in folder 0 had mismatching sizes. Expected: 32768, Got: 32567
Data block 19473 in folder 0 had mismatching sizes. Expected: 32768, Got: 32567
Data block 19474 in folder 0 had mismatching sizes. Expected: 32768, Got: 32539
Data block 19478 in folder 0 had mismatching sizes. Expected: 32768, Got: 32629
Data block 19479 in folder 0 had mismatching sizes. Expected: 32768, Got: 32581
Data block 21778 in folder 0 had mismatching sizes. Expected: 32768, Got: 32717
Data block 21780 in folder 0 had mismatching sizes. Expected: 32768, Got: 32525
Data block 23479 in folder 0 had mismatching sizes. Expected: 32768, Got: 32742
Data block 23483 in folder 0 had mismatching sizes. Expected: 32768, Got: 32604
Data block 23490 in folder 0 had mismatching sizes. Expected: 32768, Got: 32560

$ diff -r 7z ST
Binary files 7z/Fid_1EF39AC9_9087_2788_3BB7_C99AFDED7810 and ST/Fid_1EF39AC9_9087_2788_3BB7_C99AFDED7810 differ
Binary files 7z/Fid_68FA034E_435D_FA7B_0E5E_ED71FD3561BE and ST/Fid_68FA034E_435D_FA7B_0E5E_ED71FD3561BE differ
Binary files 7z/Fid_72B958AB_ACCA_5A37_7E26_E702D0A21EA3 and ST/Fid_72B958AB_ACCA_5A37_7E26_E702D0A21EA3 differ
Binary files 7z/Fid_7A19A8B3_A2BC_E38B_93F7_14B7E7108784 and ST/Fid_7A19A8B3_A2BC_E38B_93F7_14B7E7108784 differ
Binary files 7z/Fid_857553C9_F9AC_372F_4E19_6F057C11DC30 and ST/Fid_857553C9_F9AC_372F_4E19_6F057C11DC30 differ
Binary files 7z/Fid_9699B2F7_B1CB_DA7C_A45A_7516D1A0AF1A and ST/Fid_9699B2F7_B1CB_DA7C_A45A_7516D1A0AF1A differ
Binary files 7z/Fid_969C07A1_525A_5E6D_23A5_E558941FFD8D and ST/Fid_969C07A1_525A_5E6D_23A5_E558941FFD8D differ
Binary files 7z/Fid_969F0543_1772_FFA8_290C_D1EEDE126760 and ST/Fid_969F0543_1772_FFA8_290C_D1EEDE126760 differ

Media1.cab (4B590FA8):

$ ./ExtractionTool -d Media1.cab -o ST
Checking possible path: /home/bestest/confirmMs/Media1.cab
Attempting to extract all files from /home/bestest/confirmMs/Media1.cab
Attempting to extract from 'Microsoft Cabinet'

WARNING: LZX and Quantum compression schemes are not supported so some files may be skipped!
Data block 1987 in folder 0 had mismatching sizes. Expected: 32768, Got: 32582

$ diff -r 7z ST
Binary files 7z/Fid_CF953DD9_52A1_273B_4A47_9E58C291B984 and ST/Fid_CF953DD9_52A1_273B_4A47_9E58C291B984 differ

Media1.cab (54510881):
$ ./ExtractionTool -d Media1.cab -o ST
Checking possible path: /home/bestest/confirmMs/Media1.cab
Attempting to extract all files from /home/bestest/confirmMs/Media1.cab
Attempting to extract from 'Microsoft Cabinet'

WARNING: LZX and Quantum compression schemes are not supported so some files may be skipped!
Data block 20045 in folder 0 had mismatching sizes. Expected: 32768, Got: 32694
Data block 20474 in folder 0 had mismatching sizes. Expected: 32768, Got: 32653
Data block 20490 in folder 0 had mismatching sizes. Expected: 32768, Got: 32541

$ diff -r 7z ST
Binary files 7z/Fid_97AF7585_CF34_7CEA_4AB2_A29AADED1FD8 and ST/Fid_97AF7585_CF34_7CEA_4AB2_A29AADED1FD8 differ
Binary files 7z/Fid_9AC70FF6_F710_0CFA_F23C_D74EE0903211 and ST/Fid_9AC70FF6_F710_0CFA_F23C_D74EE0903211 differ

Media2.cab (54510881):
$ ./ExtractionTool -d Media2.cab -o ST
Checking possible path: /home/bestest/confirmMs/Media2.cab
Attempting to extract all files from /home/bestest/confirmMs/Media2.cab
Attempting to extract from 'Microsoft Cabinet'

WARNING: LZX and Quantum compression schemes are not supported so some files may be skipped!
Data block 21687 in folder 0 had mismatching sizes. Expected: 32768, Got: 32638
Data block 49596 in folder 0 had mismatching sizes. Expected: 32768, Got: 32585

$ diff -r 7z ST
Binary files 7z/Fid_54FB5C09_1CA4_59A1_4C27_AC491DB66192 and ST/Fid_54FB5C09_1CA4_59A1_4C27_AC491DB66192 differ
Binary files 7z/Fid_BF7B944F_F993_B2F4_A3FC_24CC95A952A9 and ST/Fid_BF7B944F_F993_B2F4_A3FC_24CC95A952A9 differ