ZPAQ compression improves with ability to send large blocks to backend

[pete4abw]

ZPAQ backend compressor now accepts blocks that are larger than the given block size. There had been crashes when trying to send anything larger. Now, this appears to be resolved, but at the expense of speed and memory use! See the zpaq_bs_fix branch and try it out!

There was really one major change that took me quite some years to work out! That is the making use of the compressBlock function directly and calling it from the zpaq_compress function. It will send the entire block to compress to the StringBuffer and then try and free up some ram.

1596 extern "C" void zpaq_compress(uchar *c_buf, i64 *c_len, uchar *s_buf, i64 s_len,
1597                 uchar *method, FILE *msgout, bool progress, int thread)
1598 {
1599         i64 total_len = s_len;
1600         int last_pct = 100;
1601         int n; // for StringBuffer
1602
1603         bufRead bufR(s_buf, &s_len, total_len, &last_pct, progress, thread, msgout);
1604         bufWrite bufW(c_buf, c_len);
1605
1606         /* use StringBuffer now and call compressBlock directly with full buffer */
1607         // compress (&bufR, &bufW, (const char *) method, 0, 0, true);
1608         libzpaq::StringBuffer sb(s_len);        // init sb to length of buffer
1609         sb.write(0, s_len);                     // set out ptr to 0
1610         n=bufR.read((char*)sb.data(), s_len);   // read all data to sb
1611         s_buf = (uchar *) realloc(s_buf,1);     // resize s_buf so that it does not take up memory
1612         if (n)
1613                 compressBlock(&sb, &bufW, (const char *) method, NULL, NULL, true);     // call compressBlock directly
1614         else
1615                 libzpaq::error("Serious error: no data to compress!");          // something wrong
1616 }

In addition, since I still can't get progress to print on compression, I added a new message line for ZPAQ to show when a thread returns from compression.

Since the 7.15 SDK of ZPAQ has 5 compression modes and the 5.0 SDK (used by lrzip) has only 3, direct comparisons are difficult. Basically the compression models used in lrzip equate roughly to ZPAQ levels 3,4, and 5 in the current SDK. Compression at the higher levels are better because the rzip preprocessor can has more data to hash prior to sending the blocks of data to the backend. There remains an issue with memory use and a lot of swap space gets used, but that's another problem for another day!

Example:

Source file: usrbin.tar, 1,681,039,360

lrzip/lrzip-next	level	Comp Size	Comp Ratio	Time	Notes
lrzip	1	427,587,904	3.931	01:57.79	Old SDK had only 3 levels. Old level 1 ~ Current SDK Level 3
lrzip-next	1	506,643,242	3.318	00:29.77	A little poorer compression, but so much faster
lrzip-next	1	480,621,893	3.498	00:28.23	with x86 filtering (e8e9)
lrzip	5	325,453,760	5.165	05:45.83
lrzip-next	5	356,681,752	4.713	03:02.18
lrzip-next	5	347,499,487	4.838	03:02.24	with x86 filtering (e8e9)
lrzip	7	318,989,312	5.270	05:30.05
lrzip-next	7	302,899,678	5.550	06:59.67
lrzip-next	7	295,382,029	5.691	06:52.15	With x86 filtering (e8e9)
lrzip	9	282,640,625	5.948	14:20.15
lrzip-next	9	261,507,804	6.428	31:22.67	Really long, but expected

Download, test, and comments welcome.

[pete4abw]

Pushed changes.

[pete4abw]

Branch zpaq_bs_fix has been removed. Merged.