Skip to content

Async redis in multiprocessing situation hangs while attempting connection in the child process #3141

Open
@dgpetrie

Description

@dgpetrie

Version: redis.asyncio ver. 5.0.1

Platform: Python 3.8.10 on Ubuntu 20.04.6 LTS

Description: The following steps are taken to reproduce this problem:

  1. In the parent process: create an async ConnectionPool, create an async redis client, connect to Redis and perform a command.
  2. Using concurrent.futures.ProcessPoolExecutor, fork a child process and in that child process: create an async ConnectionPool, create an async redis client, attempting to connect to Redis and perform a command will hang.

My speculation is that there is some contamination of the socket or connection created in the parent process by Python asyncio or redis.asyncio that causes this problem in the child process. However, I do not know either python package well enough to debug this much further.

Included link is a file which can be run from pytest or in the command line that reproduces the problem. I am sorry that I could not reduce the code to anything much smaller. Though this is greatly reduces from the original code. Changing the variable test_redis_in_parent from True to False will make this problem go away. The difference being that the command/connection is not made in the parent process before the child is forked off.

Link provided as python attachments are not allowed:
https://raw.githubusercontent.com/py-vcon/py-vcon/acba392bfd2f966db7820e8a9595c8c0ad5df41d/py_vcon_server/tests/test_redis_multiprocess.py

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions