igzip compress & decompress API

[BeeBreeze]

Hi, does igzip contain any api for buffer? for example, in zstd, we can use
ZSTD_compress(void* in, int in_len, void* out, int out_len)
and
ZSTD_decompress(void* in, int in_len, void* out, int out_len).
As far as i know, many compression algos have such API, like compress2 in zlib and FSE_compress in FSE. I am sorry to say that in my opinion, the API in igzip is not convenient to use.

[gbtucker]

Thanks for the feedback @BeeBreeze. We have intentionally avoided any dynamic memory allocation inside isa-l library functions. Leaving it up to the user aids in flexibility and performance. We could easily have a function that does something like the following.

isal_simple_compress(*in, in_len, *out, out_len, level)
{
  allocate(stream);
  isal_deflate_stateless_init(stream);
  isal_deflate_stateless(stream);
  free(stream);
}

This function could suffer a lot of overhead especially for small buffers. It is usually better to allocate once and reuse internal context in most cases. If not directly in the external API, would it help to have items in examples/ including wrappers like the above worked out? Perhaps also a three part example that takes reasonable defaults could be helpful. E.g.

isal_create_context(*cxt, level);
isal_compress_buffer(*ctx, *in, ilen, *out, olen);  // can repeat
isal_free_context(ctx);

This would be similar to one of the mid-complexity zstd interfaces. They have 2-3 between "simple" and "advanced."

[BeeBreeze]

Hi @gbtucker, thanks a lot for your kindly reply, you are so nice. I found some wrappers in igzip_rand_test.c and made some modifications. They helped a lot. The mid-complexity interfaces, createCCtx, (de)compressCCtx and freeCCtx, are widely used in our storage/backup product. However, when designing/testing a new compression algorithm, I still think buffer-friendly API is more efficient to be tested, since at this stage, we care more about compression ratio and don't want to learn about any details. Once we observe a good ratio, it is time to adjust the parameters and apply some advanced interfaces in the storage framework.