Make the remote inference engine runnable in jupyter notebooks.

[taenin]

Asyncio is an invasive dependency. If code using an asyncio loop (such as a Jupyter Notebook) runs code that creates its own asyncio loop, asyncio will throw a RuntimeError.

We get around this issue by creating any asyncio loops in new threads which we block on. Asyncio requires that only one asyncio loop can exist per thread, and this approach preserves that invariant.

Towards OPE-320

[linear]

OPE-320 FR: Update `lema.infer` to support batch inference

1. Read prompts from files
2. Write responses to files

[wizeng23]

Any potential downsides or gotchas to this approach you can think of? Is the thread creation overhead time minimal I assume?

[taenin]

Any potential downsides or gotchas to this approach you can think of? Is the thread creation overhead time minimal I assume?

There are definitely tradeoffs.

• As you mentioned, there is overhead for creating a thread. So long as the running job is somewhat lengthy (as it should be in our case), then this concern is minimal.
• The real drawback is in the scenario where an upstream caller is calling us asynchronously. As our async code is running in a thread, ALL of our async jobs will only run when their current thread is active. This means if there is more than one thread that is not blocking, our async jobs will see significant slow down.

[nikg4]

is it "in a new thread", or "in a new Python process" given that multiprocessing is used?
Looks like it's actually the former (a thread pool) https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.dummy
Thsi page has this recommendation: Users should generally prefer to use concurrent.futures.ThreadPoolExecutor, which has a simpler interface that was designed around threads from the start, and which returns concurrent.futures.Future instances that are compatible with many other libraries, including asyncio.

Would it make sense to switch to ThreadPoolExecutor here ?

[taenin]

Sure, I don't see a problem with using ThreadPoolExecutor here. Updated

[taenin]

To address your original question: my intent here is to run the event loop in a separate thread.

[taenin]

See a plethora of examples here on github with similar approaches: https://github.com/search?q=asyncio.set_event_loop+thread&type=code

This is typically done for code that is executing in an environment with an asyncio loop already running (Jupyter Notebook, IPython, Discord bots).