FileNotFoundError when used across ray workers

[kostrykin]

I have a function which I run across multiple processes using the ray framework. This function temporarily allocates a huge portion of memory. To prevent memory errors due to too many of these allocations occurring at the same time, I want to use ilock.

The structure of my code is the following:

@ray.remote
def _do_work(in):
    res1 = run_long_computations1(in)
    with ilock.ILock('huge allocation'):
        res2 = huge_allocation(res1)
        res3 = run_fast_computation(res2)
        del res2
    return run_long_computations2(res3)

Most of the time it works perfectly, but occasionally, a FileNotFoundError occurs which roots in ilock:

  File "~/test.py", line 7, in _do_work
    del res2
  File "~/.anaconda3/envs/myenv/lib/python3.7/site-packages/ilock/__init__.py", line 59, in __exit__
    os.unlink(self._filepath)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ilock-9b91444836a91e7d27f6ca3479d5f0b728c7861f6622709ae0b4e2a38734655b.lock

Why is that? How to fix it?

[tkratky]

We have the same issue, our application runs in ~10 processes. We use ILock for locking of API calls only.

[Kacao9x]

I have exact same issue while attempting to run 3 test programs in parallel. iLock was used to allow one process to make hardware API call at a time for every 2s

[kostrykin]

FTR: My final solution was to ditch ilock and use posix_ipc instead which works great

[haf]

Same in Dask in k8s:

   File "/opt/conda/lib/python3.9/site-packages/ilock/__init__.py", line 59, in __exit__
     os.unlink(self._filepath)
 FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ilock-85f3d558db120199accde070c0ff02514efb69af24c043f1f4f4d9d378cf81e9.lock'

[cr-perry]

I experienced this issue and here is my analysis of the situation.

In short, when two ILock objects are created with the same unique name, they'll use the same file as the locking entity (passed to portalocker). They create the file (if it doesn't exist) using open(path, 'w') upon ILock.__enter__, and they call os.unlink(path) upon ILock.__exit__.

However consider the following scenario:

process1: ILock.__enter__  # file is created, lock acquired
process2: ILock.__enter__  # file already exists, lock pending
process1: does its thing under the lock
process1: ILock.__exit__   # file is unlinked, lock released
process2: does its thing under the lock
process2: ILock.__exit__   # Error: cannot unlink, file does not exist

On the surface, it could be that this can be fixed by silently allowing unlink to fail; or perhaps, by recreating the file as necessary after the lock has been acquired. I am not sure though if portalocker would behave nicely in this case.

Perhaps the easiest workaround is to simply NEVER delete the file (get rid of os.unlink altogether).

[endlisnis]

I also experience this problem. Is anyone maintaining this project?

[Kacao9x]

@endlisnis for your curiosity. As this project hadn't been maintained for a while, I ended switching to use filelock. This library preserves the legacy of ilock. Additionally it comes with extensive use cases and better community support

[endlisnis]

If this project isn't maintained, and there is such a glaring error, can we get it de-listed from the pip database?

BTW, I tried "filelock" and it seems to be working, thanks.

[microfarad-de]

Please do not delist since this would break existing projects already using iLock.

I have implemented a simple workaround in https://github.com/microfarad-de/nastia-server/blob/master/opt/nastia-server/lib/serial-cli.py line 104.

Thanks