[Core] Long Running Test test_many_drivers.py failed after 2 days with (Failed at ProfileEvent)

[rkooo567]

What is the problem?

test_many_drivers.py long running test failed after 2 days. @simon-mo mentioned he saw the same issue from his PR.

Ray version and other system information (Python version, TensorFlow version, OS):
Nightly Wheel

Reproduction (REQUIRED)

Not clear how to reproduce the error. Happened after 2 days of long running tests. It doesn't happen commonly.

If we cannot run your script, we cannot fix your issue.

• I have verified my script runs in a clean environment and reproduces the issue.
• I have verified the issue also occurs with the latest wheels.

[simon-mo]

The issue seems to be during worker shutdown, there's a race condition between import thread and core worker. The core worker is already shutdown but the import thread was trying to get core worker to fetch_and_run_function. Because profiling the fetch_and_run_function requires a valid core worker instance, but the core worker is already shutdown, therefore we see the segfault.

[kfstorm]

FYI, we are aware of this issue. Instead of fixing it, we disabled profiling in our internal repo as a workaround.

[mehrdadn]

FYI this may also related to #9074 on Windows.

I do not know enough to tell, but both of them look like race conditions during shutdown related to workers and I/O.

[rkooo567]

It is happening because cpp core worker is Gc'ed before Python threads are cleaned (like import thread). When the thread tries to access the GC'ed core worker, it happens. It should be the lower priority because it only happens in the termination.

[mehrdadn]

I reproduce this consistently on master (78dfed2) on Ubuntu with

(cd python && pytest -q -s -v ray/tests/test_queue.py)

[rkooo567]

#9346 will be temporarily fix the issue.