[runtiime env] Use coroutine to create runtime envs in runtime_env_agent

[Catch-Bull]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

• current:
  • Different CreateRuntimeEnv request will trigger different thread for runtime env setup.
  • The setup steps of runtime_env is synchronous. (such as downloading packages, pip install, conda env prepare...)
• proposal:
  • Every setup steps of runtime envs will be executed in different coroutine with the same event loop(Thread).
• reason:
  • The setup of runtime_env is an IO-heavy operation, which is very suitable for coroutines.
  • Coroutines have less overhead than threads, It can carry more concurrent runtime env setups.
  • We hope to use some in-process singletons(such as DownloadManager which can limit download peak ) to manage the whole runtime_env's creation to achieve some optimization purposes. Using coroutines can avoid using mutex lock and make it easier to implement
• action:
  • step 1:
    • Change the steps of runtime env setup to use coroutines, but their implementations are not modified, we just modify def xxx to async def xxx
    • Use asyncio.Task to trigger runtime_env setup. The existing method is run_setup_with_logger.
  • step 2: use coroutine to implement every setup step of different runtime_env modules, like:
    • pip runtime env
      • create virtualenv
      • pip install
      • check ray version
    • conda runtime env
      • create conda env
    • package download for py_modules and working_dir
      ...

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[SongGuyang]

I believe this is a issue from history. In the first version, runtime envs are installed in workers(setup_worker.py). So all the methods of runtime env setup are synchronous. It made sense at that time. When we moved the logic from workers to agent, we used run_in_executor creating multiple threads to achieve concurrent setup tasks. But I think it is a temporary walk-around implementation. I agree to use coroutines because it is a recommended way in Python.
@edoakes @architkulkarni What do you think?

[edoakes]

@SongGuyang @Catch-Bull coroutines sounds good (we're actually using asyncio already, just doing it with the thread executor). My only concern is if all of the plugins/options are able to gracefully support asyncio.

[architkulkarni]

This sounds good to me as well. I think you're right about the historical reasons @SongGuyang.

Regarding @edoakes's concern, I'm trying to imagine a case where a plugin's setup step wouldn't be able to support asyncio, but I'm struggling to think of one. Any synchronous routine can still be run with asyncio with no changes right? It just won't get any optimization from the async but that's fine.

[SongGuyang]

If any step of plugins/options didn't support asyncio, I think we can use run_in_executor to make it executed in a short thread. @fyrestone @Catch-Bull What do you think?

[Catch-Bull]

@SongGuyang @Catch-Bull coroutines sounds good (we're actually using asyncio already, just doing it with the thread executor). My only concern is if all of the plugins/options are able to gracefully support asyncio.

We use coroutine to implement pip install, virtualenv setup, java env setup (download jar package), prepare working_dir etc and we've been using them for over a year. So we have confidence about these parts. Although we have not used conda, after viewing the implementation of conda setup, this part seems ok. If there are some plugins in the future that cannot be created by coroutines, we can use asyncio.to_thread(if ray still supports python3.6, we can simply implement a similar) to work around this.

About implementation, we need to pay attention to compatibility with py3.6, compared to py3.7, py3.6 lacks support for asyncio. However, we do not think it is a big problem when we actually implement it in our internal.

cc @architkulkarni @SongGuyang

[Catch-Bull]

This sounds good to me as well. I think you're right about the historical reasons @SongGuyang.

Regarding @edoakes's concern, I'm trying to imagine a case where a plugin's setup step wouldn't be able to support asyncio, but I'm struggling to think of one. Any synchronous routine can still be run with asyncio with no changes right? It just won't get any optimization from the async but that's fine.

Yes. We're going to do this part in two steps:

1. Step 1, we just change def to async def, This will not bring any optimization.
2. Step 2, after finished step 1, We can do the following work in parallel:
   • use coroutine to implement pip install $SOME_PAKCAGE
   • use coroutine to implement conda setup
   • use coroutine to implement java_env creation
     ...

[fyrestone]

@SongGuyang @Catch-Bull coroutines sounds good (we're actually using asyncio already, just doing it with the thread executor). My only concern is if all of the plugins/options are able to gracefully support asyncio.

We use coroutine to implement pip install, virtualenv setup, java env setup (download jar package), prepare working_dir etc and we've been using them for over a year. So we have confidence about these parts. Although we have not used conda, after viewing the implementation of conda setup, this part seems ok. If there are some plugins in the future that cannot be created by coroutines, we can use asyncio.to_thread(if ray still supports python3.6, we can simply implement a similar) to work around this.

About implementation, we need to pay attention to compatibility with py3.6, compared to py3.7, py3.6 lacks support for asyncio. However, we do not think it is a big problem when we actually implement it in our internal.

cc @architkulkarni @SongGuyang

Some asyncio APIs are different between Python 3.6 & 3.7. But, it's OK to write the code to compatible both of them.

[fyrestone]

This sounds good to me as well. I think you're right about the historical reasons @SongGuyang.
Regarding @edoakes's concern, I'm trying to imagine a case where a plugin's setup step wouldn't be able to support asyncio, but I'm struggling to think of one. Any synchronous routine can still be run with asyncio with no changes right? It just won't get any optimization from the async but that's fine.

Yes. We're going to do this part in two steps:

1. Step 1, we just change def to async def, This will not bring any optimization.

2. Step 2, after finished step 1, We can do the following work in parallel:

   • use coroutine to implement pip install $SOME_PAKCAGE
   • use coroutine to implement conda setup
   • use coroutine to implement java_env creation
     ...

Yes, the first step just change runtime env manager's API to async, e.g. delete_uri, setup, ... The implementation of API use asyncio.to_thread or run_in_executor to put the sync logic to async.

Then, we can refactor all the runtime env implementations to async step by step.

• pip
• conda
• ...

Finally, we can run some logic concurrently. e.g. py_modules, working_dir and pip can run concurrently.

[edoakes]

Sounds good to me.