Stderr backtrace caused by `redis.client.Redis.__del__` on process exit.

[byron-lambda]

Version: 5.0.1

Platform: Python 3.8.18 on Ubuntu 20.04

Description: Backtrace to stderr on process exit due to redis.client.Redis.__del__()

Here is the trace:

Exception ignored in: <function Redis.__del__ at 0x7fcf29e50ca0>
Traceback (most recent call last):
  File "/venv/lib/python3.8/site-packages/redis/client.py", line 491, in __del__
  File "/venv/lib/python3.8/site-packages/redis/client.py", line 506, in close
  File "/venv/lib/python3.8/site-packages/redis/connection.py", line 1155, in disconnect
  File "/venv/lib/python3.8/site-packages/redis/connection.py", line 1062, in _checkpid
AttributeError: 'NoneType' object has no attribute 'getpid'

Occurs due to unloaded globals being referenced in methods called in destructors on process exit. See warning in Python docs here

This can be fixed in a couple of ways:

1. Add an exception handler to redis.client.Redis.__del__() as it appears has been done in many other destructors (

   redis-py/redis/connection.py

   Line 226 in d3a3ada

   def __del__(self):

   for example)
2. Add references to required globals to the redis.connection.ConnectionPool class and call via the references. The globals that require references in are os.getpid and chain (from itertools) afaict.

NOTE: It is quite hard to come up with a small piece of code to reproduce this issue as it is caused by a race in the unload order at exit. I'll try to come up with something reliable if req'd.

NOTE: The issue is triggered by

redis-py/redis/client.py

Line 506 in d3a3ada

self.connection_pool.disconnect()

which was recently introduced in 012f7cf

[diogosilva30]

Can confirm this behaviour with version 5.0.1 with Django. I get this error when the development server restarts

[Alucido]

Can confirm this behaviour with version 5.0.8, Python 3.9, when running pytest test suite opening a Redis Client in a subclass of the beam.DoFn class, as part of its setup method. The error is reproduced when the test suite ends and the client is closed.

[noirbee]

I've opened #3397 after encountering this issue following a ddtrace update (probably due to the module loading / patching shenanigans it performs for libraries it wants to monkey-patch).

Thanks @byron-lambda for opening this issue in the first place, and pointing me in the right direction !

[AlexPetul]

See the same behavior with python==3.12.9 and redis==5.2.0.
I ended up subclassing redis.Redis and overriding __del__ method. Might be not the best solution, but it works:

import json
from uuid import UUID

import redis

from users.caching.base import UserMetadataCache


class Redis(redis.Redis):
    def __del__(self):
        """Release the connection.

        This is an overriden method, that bypasses the exception raise,
        when the `os` module was unloaded before the client's __del__ method was called.
        This can happen on process exit (for example Django hot reload).
        """
        try:
            self.close()
        except AttributeError as exc:
            if exc.name != "getpid":
                raise


class UserMetadataRedisCache(UserMetadataCache):

    def __init__(self, host: str, port: int = 6379) -> None:
        self.client = Redis(host=host, port=port, db=0)

    def set(self, user_id: UUID, data: dict) -> None:
        self.client.set(str(user_id), json.dumps(data))

    def delete(self, user_id: UUID) -> None:
        self.client.delete(str(user_id))

[Lyndon-prequel]

I ran into this issue too.  To make it easier for others to figure out if they're affected, I created a common reliability enumeration (CRE). You can preview the rule here or run it against your logs here.