Executing code in thread or process pools: run_in_executor example

[aaraney]

BPO	41127
Nosy	@aaraney

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-06-26.15:24:56.727>
labels = ['3.8', 'docs']
title = 'Executing code in thread or process pools: run_in_executor example'
updated_at = <Date 2020-06-26.15:24:56.727>
user = 'https://github.com/aaraney'

bugs.python.org fields:

activity = <Date 2020-06-26.15:24:56.727>
actor = 'aaraney'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2020-06-26.15:24:56.727>
creator = 'aaraney'
dependencies = []
files = []
hgrepos = []
issue_num = 41127
keywords = []
message_count = 1.0
messages = ['372428']
nosy_count = 2.0
nosy_names = ['docs@python', 'aaraney']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue41127'
versions = ['Python 3.8']

[aaraney]

I found an issue with the concurrent.futures.ProcessPoolExecuter() example (#3) in the asyncio event loops documentation. The call to asyncio.run(main()) should be guarded by `__name__=="__main__":`, as it sits now a RuntimeError is thrown.

https://docs.python.org/3/library/asyncio-eventloop.html#executing-code-in-thread-or-process-pools

[slateny]

Not getting errors on my end for python 3.8 and 3.10, could you reconfirm the error?

[AlexWaygood]

@slateny FYI I believe, due to some oddities of the github migration, some "mannequin" users won't necessarily receive notifications on issues unless you actually @ them, even if they show up as having previously participated in the thread. (You can safely assume that non-mannequin users are still subscribed to all the issues they were "nosy" to on BPO.)

[slateny]

Not getting errors on my end for python 3.8 and 3.10, could you reconfirm the error?

@aaraney ^ I know it's been a little while, but if you have time could you take another look?

[aaraney]

Certainly, thanks for pinging me, @slateny. Admittedly, the original bug report is not very descriptive, so Ill try to clarify the issue.

In the python 3.8 concurrent futures documentation the provided executor example (see below) throws a RuntimeError if you copy and paste the example into a file and run it. It appears the documentation for 3.9 and 3.10 also have this issue. This is because the asyncio.run(main()) statement at the bottom of the example is not guarded by if __name__ == "__main__":.

import asyncio
import concurrent.futures

def blocking_io():
    # File operations (such as logging) can block the
    # event loop: run them in a thread pool.
    with open('/dev/urandom', 'rb') as f:
        return f.read(100)

def cpu_bound():
    # CPU-bound operations will block the event loop:
    # in general it is preferable to run them in a
    # process pool.
    return sum(i * i for i in range(10 ** 7))

async def main():
    loop = asyncio.get_running_loop()

    ## Options:

    # 1. Run in the default loop's executor:
    result = await loop.run_in_executor(
        None, blocking_io)
    print('default thread pool', result)

    # 2. Run in a custom thread pool:
    with concurrent.futures.ThreadPoolExecutor() as pool:
        result = await loop.run_in_executor(
            pool, blocking_io)
        print('custom thread pool', result)

    # 3. Run in a custom process pool:
    with concurrent.futures.ProcessPoolExecutor() as pool:
        result = await loop.run_in_executor(
            pool, cpu_bound)
        print('custom process pool', result)

asyncio.run(main())

The example should be updated to the following:

import asyncio
import concurrent.futures

def blocking_io():
    # File operations (such as logging) can block the
    # event loop: run them in a thread pool.
    with open('/dev/urandom', 'rb') as f:
        return f.read(100)

def cpu_bound():
    # CPU-bound operations will block the event loop:
    # in general it is preferable to run them in a
    # process pool.
    return sum(i * i for i in range(10 ** 7))

async def main():
    loop = asyncio.get_running_loop()

    ## Options:

    # 1. Run in the default loop's executor:
    result = await loop.run_in_executor(
        None, blocking_io)
    print('default thread pool', result)

    # 2. Run in a custom thread pool:
    with concurrent.futures.ThreadPoolExecutor() as pool:
        result = await loop.run_in_executor(
            pool, blocking_io)
        print('custom thread pool', result)

    # 3. Run in a custom process pool:
    with concurrent.futures.ProcessPoolExecutor() as pool:
        result = await loop.run_in_executor(
            pool, cpu_bound)
        print('custom process pool', result)

if __name__ == "__main__":
    asyncio.run(main())

[slateny]

Thanks for the clarification - I tried the code and it seems to run for me without error. Are you running this from another file or is this the only file?

[aaraney]

Im running it from a standalone script. So, not from the IDLE.

[slateny]

Hmm interesting, I ran it on ubuntu 20 w/ python 3.8 and 3.10 and no errors for me - maybe a version/platform thing?