Changes to Magic Header v0.8

[pete4abw]

Magic Header Changes

1. Update to Magic Header to shave 4 bytes.

LZMA2 Encoding

By using the LZMA2 encoding for files, it will take only 2 bytes of space instead of 5.
There were two unused bytes between Decompressed File Size and Filter. This was because the space used by Encryption was miscalculated years ago. The Encryption also takes 8 bytes. This is because the Encryption Hash Loops are in fact the first 2 bytes of salt. Only 6 random numbers are attached to the loop bytes/

Reorder Magic Bytes

2. The bytes for MD5 and Encryption at the end of the Magic Header have been moved up to just after the File/Size 8 bytes.

Allow for more choices for MD5 and Encryption

3. The MD5 and Encryption bytes will now accept a value >= 1 for possible future expansion. i.e. Hash 2 could be SHA512, Encryption 2 could be AES256 and so on.

ENCRYPTION IMPORTANT

4. I am considering standardizing the encryption section for two reasons. 1. There is some custom coding used to make AES128 work between blocks that are 16 bytes and those less. The libgcrypt library allows for both situations making the handling of encrypted blocks must easier. It will also allow for additional encryption methods to be seamlessly slipped in.
   HOWEVER this will break all compatibility with encrypted files stored in earlier versions of lrzip or lrzip-next. I'm not sure how many people use encryption anyway.

So anyone with earlier archives should save version 0.7 of lrzip-next or just git down to the last tag for it

Comments? Suggestions?

lrzip-0.8x file format
July 2021
get rid of dead space
encryption is only bytes 6-13 since encryption loops are first
two bytes of salt.
Use lzma2 byte encoding for lc,lp,pb and dictionary size 0-40
Dictionary size will be decoded to fit into the old LZMA Properties
control field and encoded prior to writing the Magic Header.
move up fields for md5sum and encoding.

Byte Content
0-19 Magic

20+ RZip Chunk Data (RCD)
RCD+ Compressed Data Blocks
end-16 or 2 MD5 hash or CRC

Magic Data

0-3 LRZI
4 LRZIP-NEXT Major Version
5 LRZIP-NEXT Minor Version
6-13 Source File Decompressed Size
or If encrypted, Encryption Hash Loops + Salt (2 bytes + 6 bytes)
14 >=1 = md5sum hash is stored at end of the archive, otherwise CRC
(only 0 and 1 used now, but for possible future expansion)
15 >=1 = archive is encrypted (only 1 used now, but for possible future
expansion)
16 Filtering. 0=none, x86, ARM, ARMT, PPC, SPARC, IA64, DELTA (1..7)
high order 5 bits will contain delta offset, 0..31
Filtering is for all compression methods.
17-18 LZMA Properties using LZMA2 encoding which must be converted to LZMA
19 unused

[pete4abw]

Trimmed another byte because LC, LP, and PB are always 3, 0, 2. That encodes to 0x5D, or 93 decimal. No need to store a constant.

0-3 LRZI
4 LRZIP-NEXT Major Version
5 LRZIP-NEXT Minor Version
6-13 Source File Decompressed Size
or If encrypted, Encryption Hash Loops + Salt (2 bytes + 6 bytes)
14 >=1 = md5sum hash is stored at end of the archive, otherwise CRC
(only 0 and 1 used now, but for possible future expansion)
15 >=1 = archive is encrypted (only 1 used now, but for possible future
expansion)
16 Filtering. 0=none, x86, ARM, ARMT, PPC, SPARC, IA64, DELTA (1..7)
high order 5 bits will contain delta offset, 0..31
Filtering is for all compression methods.
17 LZMA Dictionary size Property using LZMA2 encoding which must be converted to LZMA

[pete4abw]

Update again to Magic Header. No impact on earlier files. Now zpaq compression data is stored in magic byte 17. If the compression mode is ZPAQ, then byte 17 will be formatted as follows.

1. High order bit will be set (128, 0x80, 0b10000000). This will be to distinguish it from lzma which can only be 1-40.
2. Bits 7,6, and 5 will contain the compression level, 3, 4, or 5 (0b011, 0b100, 0b101).
3. Bits 4, 3, 2, and 1 will contain the block size 1-11 (0b0001 - 0b1011.

Current versions of lrzip-next from the lzma 21.03beta branch will decode zpaq in byte 17.

Examples, (note zpaq block sizes will depend on available ram).

1. zpaq level 3, block size 4. Encode = 0b1011 0100 = 0xB4, 180.
2. zpaq level 5, block size 9. Encode = 0b1101 1001 = 0xD9, 217
   To Decode
3. Strip away high bit (magic[17] & 0b01111111 (127))
4. Read lower 4 bits for block size (magic[17] & 0b00001111 (15)) = 0b1001 = 9
5. Shift upper 4 bits right by 4 or divide by 16 (magic[17] 0b01010000 >> 4=0b00000101) = 5

Decoding

from lrzip.c get_magic()

 360         if (magic[17] > 0 && magic[17] <= 40)   // lzma dictionary
 361         {
 362                 control->dictSize = LZMA2_DIC_SIZE_FROM_PROP(magic[17]);        // decode dictionary
 363                 control->lzma_properties[0] = LZMA_LC_LP_PB;                    // constant for lc, lp, pb 0x5D
 364                 /* from LzmaDec.c */
 365                 for (i = 0; i < 4; i++)                                         // lzma2 to lzma dictionary expansion
 366                         control->lzma_properties[1 + i] = (Byte)(control->dictSize >> (8 * i));
 367         }
 368         else if (magic[17] && 0b10000000)       // zpaq block and compression level stored
 369         {
 370                 control->zpaq_bs = magic[17] & 0b00001111;      // low order bits are block size
 371                 magic[17] &= 0b01110000;                        // strip high bit
 372                 control->zpaq_level = magic[17] >> 4;           // divide by 16
 373         }

Encoding

from lrzip.c write_magic()

 171         /* save LZMA dictionary size */
 172         if (LZMA_COMPRESS)
 173                 magic[17] = lzma2_prop_from_dic(control->dictSize);
 174         else if (ZPAQ_COMPRESS) {
 175                 /* Save zpaq compression level and block size as one byte */
 176                 /* High order bits = 128 + (16 * Compression Level 3-5)
 177                  * Low order bits = Block Size 1-11
 178                  * 128 necessary to distinguish in decoding LZMA which is 1-40
 179                  * 1CCC BBBB in binary */
 180                 magic[17] = 0b10000000 + (control->zpaq_level << 4) + control->zpaq_bs;
 181                 /* Decoding would be
 182                  * magic byte & 127 (clear high bit)
 183                  * zpaq_bs = magic byte & 0X0F
 184                  * zpaq_level = magic_byte >> 4
 185                  */
 186         }

[demhademha]

How about this:
• lrzip 0.8 and onwards use libgcrypt (for various reasons, including the fact that using libgcrypt would make stuff a lot easier when dealing with encryption)
• create a separate branch for 0.7 (which contains the legacy encryption code
Regards

[pete4abw]

Thank you for your comment, @demhademha

Actually, the master branch already uses libgcrypt. It just does not use the Cipher Text Stealing mode or the SCRYPT key derivation routines which means it is backward compatible with lrzip version 0.6 files and lrzip-next v0.7 files.

I struggle with this because after all this is ALPHA software and there is no guarantee of backward compatibility. What I have done in the past is make static versions of lrzip by checking out a tag and compiling from there and renaming the executable.

Once we get to version 1, and god I hope we do, Whatever it does will be it. I will likely remove all cruft where we try and maintain compatibility backwards. We'll see.

[demhademha]

I agree with you, there is quite a bit of redundant code which could be removed :) - in addition, since lrzip-next is still in alpha, (I don't think there should be any compatibility promises made just yet)
regards

[pete4abw]

Since 2019, I had tried to push some of the early improvements over to lrzip, but development there slowed a lot and they were ignored. As I made more and more changes, our forks diverged past the point of no return. So v0.7 came out with the Filter byte, and now I found dead space in the header to make v0.8. lrzip-next can still read and decode old files (to version 0.4x) and encryption in v0.7 backwards. But I want to improve lrzip-next and as always Alpha software is what it is! There's a big, bold NOT BACKWARD COMPATIBLE README file I wrote way back in 2008 warning about this. ;-)