Compression for ledger files

[rajdhandus]

• Are the committed ledger files in compressed format today?
• If not - is it possible to compress them while in storage?
• I/O latency for remote file mounts might benefit if the files are of smaller size

[achamayou]

Are the committed ledger files in compressed format today?

They are not.

If not - is it possible to compress them while in storage?

Sure, but the private section won't benefit from compression of course.

In general, I think we see compression as a storage backend feature, rather than something the framework should provide.

[rajdhandus]

What is special about the private section? Can you elaborate please?

If we use the Compression provided by the storage platform - this will definitely reduce the size of the files while it is at rest. But - it might get decompressed when CCF does a remote read. Which means the transfer of bits would be in an uncompressed format? I was wondering about the I/O latency of the read in a compressed format vs uncompressed format.

[achamayou]

(Good) encryption results in ciphertext that's indistinguishable from random noise. Compression relies on the presence of patterns in the data to achieve a reduction in size, and since the encryption very purposefully emits data that's free from patterns, the output cannot be successfully compressed.

I don't think I can usefully comment on all possible implementations of storage platform compression, or their performance impact. In a very bandwidth-constrained environment, with abundant CPU, compression may improve throughput. Conversely, in a situation with abundant bandwidth but scarce CPU, compression will reduce throughput. The only way to be sure is to run some measurements.