PermissionError when writing to larger zarr array

[JoeWise]

Code to reproduce

points = zarr.open('data/points.zarr', mode='w', shape=(10000, 20000, 3), dtype=np.uint8, chunks=(10000, 1, 3))

for col in range(0, 20000):
    for row in range(0, 10000):
        points[row, col] = 50, 100, 150

Problem description

When I run the above code snippet I get a PermissionError, even when running as administrator. This doesn't happen for smaller zarr arrays e.g. shape=(100, 200, 3)

Traceback (most recent call last): File "<stdin>", line 3, in <module> File "C:\UserPath\AppData\Roaming\Python\Python38\site-packages\zarr\core.py", line 1115, in __setitem__ self.set_basic_selection(selection, value, fields=fields) File "C:\UserPath\AppData\Roaming\Python\Python38\site-packages\zarr\core.py", line 1210, in set_basic_selection return self._set_basic_selection_nd(selection, value, fields=fields) File "C:\UserPath\AppData\Roaming\Python\Python38\site-packages\zarr\core.py", line 1501, in _set_basic_selection_nd self._set_selection(indexer, value, fields=fields) File "C:\UserPath\AppData\Roaming\Python\Python38\site-packages\zarr\core.py", line 1550, in _set_selection self._chunk_setitem(chunk_coords, chunk_selection, chunk_value, fields=fields) File "C:\UserPath\AppData\Roaming\Python\Python38\site-packages\zarr\core.py", line 1664, in _chunk_setitem self._chunk_setitem_nosync(chunk_coords, chunk_selection, value, File "C:\UserPath\AppData\Roaming\Python\Python38\site-packages\zarr\core.py", line 1729, in _chunk_setitem_nosync self.chunk_store[ckey] = cdata File "C:\UserPath\AppData\Roaming\Python\Python38\site-packages\zarr\storage.py", line 825, in __setitem__ replace(temp_path, file_path) PermissionError: [WinError 5] Access is denied: 'C:\\UserPath\\DataPath\\data\\points.zarr\\0.0.0.fca098bcac1b4eb6a62733dc4f2d6966.partial' -> 'C:\\UserPath\\DataPath\\data\\points.zarr\\0.0.0'

Version and installation information

• Value of zarr.__version__ == 2.4.0
• Value of numcodecs.__version__ == 0.6.4
• Version of Python interpreter == Python 3.8.3
• Operating system (Linux/Windows/Mac) == Windows
• Zarr was installed using pip

[ericgyounkin]

I am also encountering this in zarr 2.6. I believe what is happening is the antivirus on my workstation (mandatory) prevents the os.replace from writing the tempfile to the array file. Makes it an intermittent issue, which is always fun.

I believe there is a simple fix for this. I've included it below. I've tested it and it resolves the issue for me.

storage.py at line 860

Replace

os.replace(temp_path, file_path)

with

attempts = 0
while attempts < 10:
    try:
        os.replace(temp_path, file_path)
        break
    except PermissionError as e:
        time.sleep(0.1)
        attempts += 1
if attempts == 10:
    raise e

I found a 0.1 seconds to be a pretty good time to wait, vast majority of replace commands succeed on first or second attempt using this delay time.

Is this suitable for a pull request?

[dstansby]

Looks like this was fixed by #698? If not, please do open a new issue!