bpo-38377: multiprocessing.SemLock requires working /dev/shm on Linux.

[mcepl]

Skip test_get_event_loop_new_process on systems which don’t provide it
(e.g., building environments of some Linux distributions).

https://bugs.python.org/issue38377

Automerge-Triggered-By: @pitrou

[vstinner]

cc @pablogsal @pitrou @applio

[pitrou]

+1 from me

[miss-islington]

@mcepl: Status check is done, and it's a failure ❌ .

[pitrou]

@mcepl You should have pulled from the updated PR instead of force-pushing :-(

[pitrou]

Please reapply the lost changes now.

[mcepl]

Please reapply the lost changes now.

What lost changes? I have rebased my branch on the top of the current master and add one more commit. What did I loose? The change is still the same.

I am sorry, what I am missing here?

[pitrou]

Yes, you rebased and force-pushed while I had pushed another commit, so you lost it.

[pitrou]

See 8fd3d0c

[tiran]

Did you consider to implement auto-detection of SHM interface instead of an env var? Something like

def has_shm():
    if sys.platform == 'linux':
   	     with open("/proc/mounts") as f:
             return "/dev/shm" in f.read()
     else:
         ...

[tiran]

nit-pick, you should also check sys.flags.ignore_environment here.

[mcepl]

Did you consider to implement auto-detection of SHM interface instead of an env var? Something like

@tiran Isn’t this exact opposite of what Victor asked for in #19073 (comment) ?

[tiran]

Did you consider to implement auto-detection of SHM interface instead of an env var? Something like

@tiran Isn’t this exact opposite of what Victor asked for in #19073 (comment) ?

Ah, I didn't see the comments because they were not on BPO and GH hides coments of closed discussions.

[vstinner]

@mcepl:

@tiran Isn’t this exact opposite of what Victor asked for in #19073 (comment) ?

My first remark was:

I'm not sure if /dev/shm should be hardcoded here.

@tiran:

Did you consider to implement auto-detection of SHM interface instead of an env var?

That would avoid hardcoding and so it sounds like a reasonable approach.

I also wrote:

I dislike that test_asyncio becomes tidely coupled to multiprocessing internals. Is there a function in multiprocessing somewhere to check if /dev/shm is required and usable?

If code is written, I suggest to add a private function to multiprocessing.util. See for example _cleanup_tests() there which is used by multiprocessing tests but also by concurrent.futures tests.

---

To come back to https://bugs.python.org/issue38377 it seems like the problem is that creating _multiprocessing.SemLock() raises an OSError. There are other tests which are skipped if SemLock doesn't work. Example with test_concurrent_futures:

# Skip tests if sem_open implementation is broken.
support.import_module('multiprocessing.synchronize')

Maybe multiprocessing.synchronize should raises an exception on import on Linux if /dev/shm, rather than trying to put logic far from the code?

The simplest check is to create a SemLock() object and destroy it. But maybe a different approach which doesn't avoid creating a lock can be found?

Another simple check is to raise an ImportError on Linux if /dev/shm/ directory doesn't exist.

/dev/shm path is already hardcoded in Lib/multiprocessing/heap.py: Arena._dir_candidates.

[vstinner]

I wrote a different approach: PR #20944 attempts to create a lock and convert OSError to unittest.SkipTest exception.

[mcepl]

Closing this in preference to #20944.