Executing code in thread or process pools: run_in_executor example

[aaraney]

BPO	41127
Nosy	@aaraney

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-06-26.15:24:56.727>
labels = ['3.8', 'docs']
title = 'Executing code in thread or process pools: run_in_executor example'
updated_at = <Date 2020-06-26.15:24:56.727>
user = 'https://github.com/aaraney'

bugs.python.org fields:

activity = <Date 2020-06-26.15:24:56.727>
actor = 'aaraney'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2020-06-26.15:24:56.727>
creator = 'aaraney'
dependencies = []
files = []
hgrepos = []
issue_num = 41127
keywords = []
message_count = 1.0
messages = ['372428']
nosy_count = 2.0
nosy_names = ['docs@python', 'aaraney']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue41127'
versions = ['Python 3.8']

[aaraney]

I found an issue with the concurrent.futures.ProcessPoolExecuter() example (#3) in the asyncio event loops documentation. The call to asyncio.run(main()) should be guarded by `__name__=="__main__":`, as it sits now a RuntimeError is thrown.

https://docs.python.org/3/library/asyncio-eventloop.html#executing-code-in-thread-or-process-pools

[slateny]

Not getting errors on my end for python 3.8 and 3.10, could you reconfirm the error?

[AlexWaygood]

@slateny FYI I believe, due to some oddities of the github migration, some "mannequin" users won't necessarily receive notifications on issues unless you actually @ them, even if they show up as having previously participated in the thread. (You can safely assume that non-mannequin users are still subscribed to all the issues they were "nosy" to on BPO.)

[slateny]

Not getting errors on my end for python 3.8 and 3.10, could you reconfirm the error?

@aaraney ^ I know it's been a little while, but if you have time could you take another look?

[aaraney]

Certainly, thanks for pinging me, @slateny. Admittedly, the original bug report is not very descriptive, so Ill try to clarify the issue.

In the python 3.8 concurrent futures documentation the provided executor example (see below) throws a RuntimeError if you copy and paste the example into a file and run it. It appears the documentation for 3.9 and 3.10 also have this issue. This is because the asyncio.run(main()) statement at the bottom of the example is not guarded by if __name__ == "__main__":.

import asyncio
import concurrent.futures

def blocking_io():
    # File operations (such as logging) can block the
    # event loop: run them in a thread pool.
    with open('/dev/urandom', 'rb') as f:
        return f.read(100)

def cpu_bound():
    # CPU-bound operations will block the event loop:
    # in general it is preferable to run them in a
    # process pool.
    return sum(i * i for i in range(10 ** 7))

async def main():
    loop = asyncio.get_running_loop()

    ## Options:

    # 1. Run in the default loop's executor:
    result = await loop.run_in_executor(
        None, blocking_io)
    print('default thread pool', result)

    # 2. Run in a custom thread pool:
    with concurrent.futures.ThreadPoolExecutor() as pool:
        result = await loop.run_in_executor(
            pool, blocking_io)
        print('custom thread pool', result)

    # 3. Run in a custom process pool:
    with concurrent.futures.ProcessPoolExecutor() as pool:
        result = await loop.run_in_executor(
            pool, cpu_bound)
        print('custom process pool', result)

asyncio.run(main())

The example should be updated to the following:

import asyncio
import concurrent.futures

def blocking_io():
    # File operations (such as logging) can block the
    # event loop: run them in a thread pool.
    with open('/dev/urandom', 'rb') as f:
        return f.read(100)

def cpu_bound():
    # CPU-bound operations will block the event loop:
    # in general it is preferable to run them in a
    # process pool.
    return sum(i * i for i in range(10 ** 7))

async def main():
    loop = asyncio.get_running_loop()

    ## Options:

    # 1. Run in the default loop's executor:
    result = await loop.run_in_executor(
        None, blocking_io)
    print('default thread pool', result)

    # 2. Run in a custom thread pool:
    with concurrent.futures.ThreadPoolExecutor() as pool:
        result = await loop.run_in_executor(
            pool, blocking_io)
        print('custom thread pool', result)

    # 3. Run in a custom process pool:
    with concurrent.futures.ProcessPoolExecutor() as pool:
        result = await loop.run_in_executor(
            pool, cpu_bound)
        print('custom process pool', result)

if __name__ == "__main__":
    asyncio.run(main())

[slateny]

Thanks for the clarification - I tried the code and it seems to run for me without error. Are you running this from another file or is this the only file?

[aaraney]

Im running it from a standalone script. So, not from the IDLE.

[slateny]

Hmm interesting, I ran it on ubuntu 20 w/ python 3.8 and 3.10 and no errors for me - maybe a version/platform thing?

[aaraney]

Yeah, that is pretty strange. I'm AFK, but from memory I reproduced it on 3.8.10 sourced from miniconda on OSX. I'll see if I can reproduce it in a container tomorrow and put together a repository and share it here if I can.

[aaraney]

So, I was not able to reproduce the issue in a docker container. I used the following base images in attempt to be more complete: python:3.8-slim-buster, python:3.8.10-slim-buster, and continuumio/miniconda3:4.11.0. However, I am beginning to think this is a platform issue not a python version issue. More specifically, something to do with the spawn semantics used vs fork exec. This behavior was changed in 3.8 for OSX.

Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725.

Instead of droning on about why I think that is the case, I threw together a repo that uses github actions and a macos 11 runner. Below ive included the gh action yaml for convenience. I was able to reproduce the issue using gh actions. Please find the failing job here.

name: Run Unit Tests

on: push

jobs:
  cpython_85299:

    runs-on: macos-11

    steps:
    - uses: actions/checkout@v2
    - name: Set up Python 3.8
      uses: actions/setup-python@v2
      with:
        python-version: '3.8'
    - name: Reproduce Runtime Error
      run: |
        python3 main.py

[slateny]

Hmm, if it's a platform issue then do you think that the docs still need updating? If the file's the only one being run, then if __name__ == "__main__": seems equivalent to leaving it out

[aaraney]

Yeah I think that is a fair point, @slateny. Ive not seen it stated in python docs that examples are runnable as their own script, but I think that is kind of implicit to an example snippet. I would prefer that if __name__ == "__main__": is included where examples are expected to be run in a standalone context.

[slateny]

The third example uses ProcessPoolExecutor, defined here, which then seems to get the start method, which as you mentioned uses spawn since 3.8. A bit down below, there is a note about protecting the program's entry point via if __name__ == '__main__' if using the spawn method.

Taking all that into account, I think the doc change should be instead to add a note/link to the warning for macOS, advising the entrypoint guard as needed, instead of adding it directly into the example as there might be confusion on why the guard's there when it may not be necessary.

[gvanrossum]

The problem seems to be with how the OP is running the code, not with the example. Nevertheless it is fine to add a __main__ check to the example, since that is customary for all main programs. No notes about platforms though.