Skip to content

Reduce LZMA dictionary size for small baskets #194

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

dan131riley
Copy link

LZMA by default creates very large hash tables for its dictionaries, e.g., at compression level 4, the hash table is 4Mi 4 byte entries, 16 MiB total. The hash table has to be zeroed before use so it is allocated via calloc(), which means all the pages have to be allocated, mapped and written. ROOT baskets are often much smaller than the default LZMA dictionaries; for small baskets, the large dictionary has very little compression benefit, while zeroing the hash table can be more expensive than the actual compression operation.

Since R__zipLZMA() is actually being used to compress a buffer of known size, not a stream, we can use the size of the buffer to estimate an appropriate size for the dictionary. This PR uses a slightly more advanced part of the LZMA API to set the dictionary size to 1/4 the size of the input buffer, if that is smaller than the default size from the selected preset compression level. In tests with CMS data, this results in less than 1% increase in the output size and (in one test job) a 25% reduction in job total run time, with LZMA compression time reduced by 80% (all of that time that was being spent in memset() zeroing the hash table).

I also tested this with the "Event" test program with Brian's changes from #59. With the same test parameters as Brian ("./Event 4000 6 99 1 1000 2"), I get

ZLIB level-6: 14.4 MB/s
Original LZMA level-6: 2.3 MB/s
Modified LZMA level-6: 3.0 MB/s

With 100 tracks per event (and hence smaller baskets) the improvement is from 2.2 MB/s to 3.9 MB/s.

This change should be fully transparent and backwards compatible.

@bbockelm
Copy link
Contributor

@dan131riley - is there a similar interface for zlib?

@dan131riley
Copy link
Author

@bbockelm

deflateInit2() can be used to set the size of the window and hash table for zlib

http://www.zlib.net/manual.html#Advanced

The default values for zlib give 128KiB for the sliding window and 128KiB for hash table, much more modest than the LZMA sizes. zlib memory initialization doesn't show up in igprof output for cmsRun, so tuning it isn't likely to make a very big difference.

@bbockelm
Copy link
Contributor

(Probably worth at least the test!)

@bbockelm
Copy link
Contributor

Oh - what are the details of the job that improved? Was it simply reading & rewriting the file? Or part of a fixed workflow?

@dan131riley
Copy link
Author

The job was full RECO with 8 threads, reading raw data and writing RECO, AOD, MINIAOD and DQMIO. Details are in Slava's slides from yesterday, but it is a generic enough RECO job that it should show up in any standard RECO workflow writing AOD and MINIAOD. The effect on cmsRun timing was definitely exaggerated by having 8 threads, as the occasional long pauses from LZMA sometimes blocked the other threads too (the job time speedups I quoted were wall clock, not cpu).

@bbockelm
Copy link
Contributor

yeah, that makes sense. There's a reasonable chance that a zlib improvement would show up in the ROOT-based benchmarks but not in CMSSW.

FWIW - Chris and I have gone over various mechanisms to allow other threads to progress while the call to deflate is in progress. Haven't yet figured out anything that doesn't cause deadlocks...

@pcanal
Copy link
Member

pcanal commented Jul 13, 2016

Pushed into the master. Thanks.

@pcanal pcanal closed this Jul 13, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants