JMH benchmark?

[jendap]

The idea in the paper sounds good! Thank you!

Are you still working on this? There are some issues. Many of them actually. The timings are certainly unreliable. The compression ratio is not representative for general compression. But hopefully the numbers are reasonable in comparison with Gorilla :) And that's the most important thing!

It would be much easier to test if there was (well done) JMH benchmark.

[panagiotisl]

Hello @jendap

Thank you for your interest. We do not have the cycles to do this at the moment, even though we would be happy to assist anyone willing to do so, and therefore I will close the issue.

I have repeated the timing tests included in the repository and they match what we report in the paper:

Chimp128: /city_temperature.csv.gz - Bits/value: 22,92, Compression time per block: 22752,85, Decompression time per block: 19032,92
Chimp128: /Stocks-Germany-sample.txt.gz - Bits/value: 13,20, Compression time per block: 16879,55, Decompression time per block: 14923,93
Chimp128: /SSD_HDD_benchmarks.csv.gz - Bits/value: 17,00, Compression time per block: 21949,64, Decompression time per block: 18350,52

Gorilla: /city_temperature.csv.gz - Bits/value: 58,72, Compression time per block: 28443,89, Decompression time per block: 22383,31
Gorilla: /Stocks-Germany-sample.txt.gz - Bits/value: 46,90, Compression time per block: 24918,06, Decompression time per block: 20529,09
Gorilla: /SSD_HDD_benchmarks.csv.gz - Bits/value: 40,25, Compression time per block: 22553,23, Decompression time per block: 18831,44

Moreover, our algorithm is now included in DuckDB and Apache IotDB and they have both verified our claim in the paper, that we offer a significantly improved trade-off between space and speed.

With regards to space, it is evident in our results that the compression ratio we attain depends on the dataset (there are cases in which the ratio is significant as well as cases in which the ratio is small). Please note that, even though the algorithm has been shown to work for non-timeseries, it is primarily built for time series data.

We will be happy to answer any e-mails if you wish to discuss our paper further !

Cheers!

[jendap]

I was just looking at IotDB if it is worth a while even exploring when I saw "state-of-the-art compression algorithm". That's why I looked at your paper :-)

So why am I writing if it is not worth neither of our time?

Because somebody should look at it better. I believe there has to be potential to do much better then all of this (Chimp included).

The timings are wrong. It does not matter how many times you run this code and look at stdout. The most glaringly wrong number in paper for me was lz4 compression time which is more then 10x wrong. Looking at code and running it is far worse. All the timing of generic algorithms in the paper are very wrong. About the gorilla / chimp. To get any timing there you would have to use JMH, use the same underlying bitstream library and use similarly good code.

The compression ratios are not well done either. No reasonable database would create compressor for every 1000 values. Is Chimp better then Gorilla? Maybe. But Gorilla was far worse then lz4 / zstd for any dataset I have thrown at it. If it was not from a big company I'm not sure anybody would even hear about it.

So yeah, we can close this issue. But if any student / researcher is reading this - please do better! There in an opportunity to do much better! ;-)

[panagiotisl]

You opened an issue requesting a JMH benchmark and this is something that we do not have the time to do, so I don't see why the issue should have remained open. We respect your criticism of our work and we are more than willing to defend it, even though I find this platform to be highly inappropriate for this purpose.

InfluxDB uses 1,000 values. The experimental section of the initial version of our paper was focused on this database and we used it's configuration. I believe InfluxDB is a "reasonable database" and has reasons to use this value. I could name a few but this has nothing to do with our research so I guess it's better to seek answers from InfluxDB developers.

Moreover, DuckDB has published a blog post (https://duckdb.org/2022/10/28/lightweight-compression.html) after adopting our algorithm as the default compressor for floating point values where they claim that "To achieve a good compression ratio, blocks of at least 256KB must be used" for general purpose compressors. DuckDB uses shorter blocks. They also claim that "the downside of general purpose compression is that (de)compression is generally expensive". These arguments validate our claims in terms of both execution time and compression ratio. I find DuckDB to be a very "reasonable database" and I was very impressed when reading their commit messages and respective code with the way they tested and improved the implementation of our algorithm.

Is the experimental section of our paper the best it could be? Probably not. But benchmarking is not the focus of our work. Our work is about proposing new ideas in research papers. More importantly, our work is all about doing better than what's already available, so it goes without saying that students / researchers should be encouraged to do better than us. That is why we make our code available. Finally, as we are still actively working in this field and already have some ideas on how to improve our algorithm, we will be very glad to hear any ideas you might have.

[jendap]

For what it's worth I have closed this issue already.

Once again, for the record ... I do believe your Chimp compression may be better than Gorilla. Thank you for that! I'm not sure how much better from the data like this though. In my testing Gorilla was so bad that even 2x compression improvement would be no match for lz4. Although it can indeed be better on different datasets. There's definatelly opportunity to do better. Some researcher should take a look at it.

This is where we should end this conversation as we are clearly both done with Chimp.

There's just one think I have to ask. What do the data from the wizards writing "very reasonable database" with 10k github stars say? Do they really say: Chimp compression is worse then "Uncompressed"?

Version	Taxi	On Time	Lineitem	Notes	Date
DuckDB dev	4.8GB	0.21GB	0.17GB	FSST + Chimp	NOW()
Parquet (Uncompressed)	4.5GB	0.12GB	0.31GB

[panagiotisl]

Did you miss the first line of the table? Let me help you with that:

Version	Taxi	On Time	Lineitem	Notes	Date
DuckDB v0.2.8	15.3GB	1.73GB	0.85GB	Uncompressed	July 2021
DuckDB dev	4.8GB	0.21GB	0.17GB	FSST + Chimp	NOW()
Parquet (Uncompressed)	4.5GB	0.12GB	0.31GB

Clearly, FSST + Chimp is significantly better than Uncompressed. Parquet (Uncompressed) means that no extra compressor -like Snappy, Zstd or LZ4- is used for the data block inside dictionary pages and the data pages of Parquet. However, Parquet uses a number of techniques including Dictionary Compression, Run Length Encoding and Delta Encoding to reduce the size of a csv file. Despite its space efficiency, Parquet is not useful in certain cases, such as the small blocks of DuckDB.

Now, I would not call the people at DuckDB wizards, but they are certainly very good at what they do and we are extremely happy they decided to use our algorithm.