Closed
Description
Zarr version
2.16.0
Numcodecs version
0.11.0
Python Version
3.11.4
Operating System
Windows
Installation
py -m pip install -U zarr
Description
Since zarr v2.15.0, the tifffile.ZarrTiffStore fails to read any chunks when wrapping the store in zarr.LRUStoreCache. The chunks returned by LRUStoreCache are always zeroed.
I don't understand yet what change in v2.15 causes the failure. It may be related to ZarrTiffStore not returning all chunk keys from the keys method because there may be many million keys and it would take significant resources to parse the TIFF file(s) for all keys upfront. Instead, the store relies on the __contains__ method.
The following patch fixes the issue for me:
diff --git a/zarr/storage.py b/zarr/storage.py
index 4f7b9905..c3260638 100644
--- a/zarr/storage.py
+++ b/zarr/storage.py
@@ -2436,7 +2436,7 @@ class LRUStoreCache(Store):
with self._mutex:
if self._contains_cache is None:
self._contains_cache = set(self._keys())
- return key in self._contains_cache
+ return key in self._contains_cache or key in self._store
def clear(self):
self._store.clear()Steps to reproduce
import numpy
import zarr
import tifffile
data = numpy.random.rand(8, 8)
tifffile.imwrite('test.tif', data)
store = tifffile.imread('test.tif', aszarr=True)
try:
# without cache
zarray = zarr.open(store, mode='r')
numpy.testing.assert_array_equal(zarray, data)
# with cache
cache = zarr.LRUStoreCache(store, max_size=2**10)
zarray = zarr.open(cache, mode='r')
numpy.testing.assert_array_equal(zarray, data)
finally:
store.close()Additional output
Traceback (most recent call last):
File "D:\Dev\Python\tifffile\test_zarr_lrucache.py", line 16, in <module>
numpy.testing.assert_array_equal(zarray, data)
File "X:\Python311\Lib\site-packages\numpy\testing\_private\utils.py", line 920, in assert_array_equal
assert_array_compare(operator.__eq__, x, y, err_msg=err_msg,
File "X:\Python311\Lib\contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "X:\Python311\Lib\site-packages\numpy\testing\_private\utils.py", line 797, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Arrays are not equal
Mismatched elements: 64 / 64 (100%)
Max absolute difference: 0.99263945
Max relative difference: 1.
x: array([[0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 0.],...
y: array([[0.992639, 0.580277, 0.454157, 0.055261, 0.664422, 0.82762 ,
0.144393, 0.240842],
[0.241279, 0.190615, 0.37115 , 0.078737, 0.706201, 0.593853,...