RemoteConfigSubscriber thread has no asyncio event loop.

[rmorshea]

Summary of problem

We send logs to a collection service using PyZMQ. Ideally we'd be able to use the async bindings for this library but we're running into an issue where ddtrace is producing debug logs in a thread that does not have an asyncio event loop. Those logs are getting picked up by the PyZMQ's async log handler where it complains because there's no event loop for it to submit tasks to.

File ".../lib/python3.10/site-packages/ddtrace/internal/logger.py", line 147, in handle
  super(DDLogger, self).handle(record)
File ".../lib/python3.10/logging/__init__.py", line 1634, in handle
  self.callHandlers(record)
File ".../lib/python3.10/logging/__init__.py", line 1696, in callHandlers
  hdlr.handle(record)
File ".../lib/python3.10/logging/__init__.py", line 968, in handle
  self.emit(record)
File ".../lib/python3.10/site-packages/zmq/log/handlers.py", line 186, in emit
  self.socket.send_multipart([btopic, bmsg])
File ".../lib/python3.10/site-packages/zmq/_future.py", line 321, in send_multipart
  return self._add_send_event('send_multipart', msg=msg_parts, kwargs=kwargs)
File ".../lib/python3.10/site-packages/zmq/_future.py", line 509, in _add_send_event
  f = future or self._Future()
File ".../lib/python3.10/asyncio/events.py", line 656, in get_event_loop
  raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'ddtrace.internal.remoteconfig._subscribers:RemoteConfigSubscriber'.

Thankfully, I don't think it would be a great deal of effort to modify PeriodicThread to operate using async primitives instead of thread-based in a backwards compatible way. Here's how I think it could work:

class PeriodicThread(threading.Thread):

    def __init__(self, *args, **kwargs):
        # same init logic as exists now...
        self.quit = forksafe.ResetObject(asyncio.Event)

    def run(self):
        asyncio.run(self._run_async())

    def _run_async(self):
        while not (await self.quit.wait(self.interval)):
            self._target()
        if self._on_shutdown is not None:
            self._on_shutdown()

One could argue this is a problem with pyzmq since it would be ideal if we could manually specify what loop it should be sending its async tasks to. With that said, doing so would require a fair bit of care to ensure the internals of pyzmq are thread-safe. That is, a bunch of calls inside pyzmq which interact with the event loop would suddenly need to use loop.call_soon_threadsafe. Adding and testing that behavior to pyzmq seems pretty burdensome so I've chosen to post this issue here.

Which version of dd-trace-py are you using?

2.5.0

Which version of pip are you using?

Using Poetry 1.8.0

Which libraries and their versions are you using?

The library related to this problem is pyzmq.

How can we reproduce your problem?

• Turn on DEBUG logs
• Add a zmq.log.PUBHandler logging handler instance that uses a zmq.asyncio.Context.
• Use the ddtrac.tracer()

What is the result that you get?

See traceback above

What is the result that you expected?

No errors.

[rmorshea]

Closing in favor of: zeromq/pyzmq#1967