Zip/gZip read a compressed binary file without having previously decompressed it

[chelis10]

Background

I have COBOL binary files compressed in GZIP format, which can be up to 140GB in size, and I'm having difficulties because I can't use Spark's parallel capabilities for decompression.

Question

Is there a way to read the compressed file and have Spark handle the decompression and reading, or do you plan to implement this at some point?

[yruslan]

Hi @chelis10,

Gzip-compressed files are not splittable, so Spark cannot process a single .gz file in parallel. This is not limited to Cobrix as the same applies to other formats like CSV. Spark must first decompress it, and that limits parallelism to one core per file.

That being said adding decompression support to Cobrix might be a good feature, especially for compression algorithms like bzip2 or lz2/snappy which are splittable. But implementing decompression could be challenging, especially for VRL files. We might take a look at it, at least to evaluate the effort. Let's keep this issue open.