[runtime env] Make plugin setup process that has not been refactor run in threads. #22588
Conversation
Catch-Bull
requested review from
architkulkarni,
edoakes,
SongGuyang and
fyrestone
|
|
||
|
|
||
| def test_plugin_hang(ray_start_regular): | ||
| @ray.remote(num_cpus=0.1) |
There was a problem hiding this comment.
before this PR, this case will be failed.
There was a problem hiding this comment.
Nice! Thanks for adding this!
|
Thanks @Catch-Bull. Could you please also leave a comment for why we're doing this and a TODO to remove it once refactored? |
|
We only make the The problem is that running sync logic in an async method blocks the asyncio loop. This PR makes the setup process (sync logic) in threads before they are fully async. |
| # Currently create method is still a sync process, to avoid blocking | ||
| # the loop, need to run this function in another thread. | ||
| # TODO(Catch-Bull): Refactor method create into an async process, and | ||
| # make this method running in current loop. |
There was a problem hiding this comment.
Thanks @Catch-Bull. Could you please also leave a comment for why we're doing this and a TODO to remove it once refactored?
@edoakes I add annotations and TODO, and copy those to conda.py, py_modules.py, and working_dir.py
|
|
hmmm, It seems flaky:
|
According to https://flaky-tests.ray.io/ |
It will raise Import error, I have no idea why this happens on windows but, I found that after I did not call RuntimeEnvPlugin in remote_function, the test basically passed every time, change in here: 37698a6 |
I recently realized that during a runtime_env creation process, a plugin/manager that is very slow to setup may block the creation of other runtime_env, so I make plugin/manager setup run in threads.
The refactor of
PipManageris about to be completed, so I ignore it in this PR.Related issue number
Checks
scripts/format.shto lint the changes in this PR.