Intermittent CRC mismatch when reading

[sidy3d]

Hi,

It seems that some npz files are not decode properly. I get a crc mismatch which occurs in the following section of code:

libnpy/src/npz.cpp

Lines 446 to 455 in e76b1da

	compression_method_t cmethod =
	static_cast<compression_method_t>(entry.compression_method);
	if (cmethod == compression_method_t::DEFLATED) {
	uncompressed_bytes = npy_inflate(std::move(uncompressed_bytes));
	}
	std::uint32_t actual_crc32 = npy_crc32(uncompressed_bytes);
	if (actual_crc32 != entry.crc32) {
	throw std::logic_error("CRC mismatch");
	}

Using 7z I've verified that the value reported by entry.crc32 is correct.

Thanks,

[sidy3d]

This seems to happen when multidimensional arrays are involved.

[matajoh]

Thanks for the bug report! I'll look into this.

[matajoh]

Is it possible share the file that produced this behaviour, or alternatively some code so I can generate and test locally?

[sidy3d]

Here is the file that was causing the issue.
data.zip

[matajoh]

Can you send me some more information? Platform, compiler, etc.? I am unable to reproduce this behaviour on any of my normal build environments (the file loads fine for me).

[matajoh]

I wrote some sample code for checking if the library can read and report on an NPY/NPZ file in this branch:

https://github.com/matajoh/libnpy/tree/adding_npy_info

Just so we are running the same code.

[sidy3d]

Thanks for the sample project. Interestingly I am not able to reproduce the issue on the sample project with Windows 11 and the latest version of MSVC.
Give me a couple of days to try to replicate the issue.

[matajoh]

Closing due to inactivity.

[sidy3d]

Sorry for the long delay.

After some more investigation it seems that the error only occurs when reading compressed .npz files. There are no issues reading uncompressed files.

Here is a sample project demonstrating the issue:
https://github.com/sidy3d/libnpy-crc-mismatch

[matajoh]

Thanks, I'll take a look.