RuntimeError: can't start new thread

[rvanlaar]

Describe the bug: We're getting errors "RuntimeError: can't start new thread".
The only change we've introduced is turning Elastic on.
Since than we're getting these errors. When running long enough, this
exhausts the thread pool on the server.

To Reproduce

1. Use APM with Django

Expected behavior: To not run out of threads.

Environment (please complete the following information)

• OS: Ubuntu 16.04.5 LTS
• Python version: 3.5.2
• Framework and version [e.g. Django 2.1]: Django 1.11.20
• APM Server version: elastic cloud
• Agent version: elastic-apm 4.2.1

Additional context

RuntimeError: can't start new thread
File "threading.py", line 1180, in run
self.function(*self.args, **self.kwargs)
File "elasticapm/transport/base.py", line 124, in flush
self._start_flush_timer()
File "elasticapm/transport/base.py", line 160, in _start_flush_timer
self._flush_timer.start()
File "threading.py", line 844, in start
_start_new_thread(self._bootstrap, ())

We're running with celery and sentry.

• Agent config options

  Click to expand

  if env('ELASTIC_APM'):
  ELASTIC_APM = {
      # Set required service name. Allowed characters:
      # a-z, A-Z, 0-9, -, _, and space
      'SERVICE_NAME': 'CLIENT PRODUCTION',
      # Use if APM Server requires a token
      'SECRET_TOKEN': env('ELASTIC_APM_SECRET_TOKEN'),
      # Set custom APM Server URL (default: http://localhost:8200)
      'SERVER_URL': env('ELASTIC_APM_SERVER_URL)
  }
  
  INSTALLED_APPS += ('elasticapm.contrib.django',)
  # The Elastic APM tracing middleware needs to be first.
  MIDDLEWARE_CLASSES = (
      'elasticapm.contrib.django.middleware.TracingMiddleware',
      ) + MIDDLEWARE_CLASSES
  

[beniwohli]

Hi @rvanlaar

Thanks for the report. This is pretty odd, but it touches on a larger problem: we use the threading.Timer class that comes with the standard library. Timer is a one-shot timer, as in, it only runs the given function once after the timeout, then ends the thread. To make the timer repeat, we start a new thread in the timer function. This is not great, because we're creating and destroying threads all the time, and as your bug report suggests, we can also run into scenarios where thread creation runs off.

To remedy this, I opened (and already merged) #452. It works slightly differently to Timer, in that it re-runs the given function in the same thread, which should waste a bit less resources, as well as hopefully fix your issue.

[rvanlaar]

Thanks for the reply and information. Is there an idea about when a new release hits pypi?

…

On April 9, 2019 1:58:26 PM GMT+02:00, Benjamin Wohlwend ***@***.***> wrote: Hi @rvanlaar Thanks for the report. This is pretty odd, but it touches on a larger problem: we use the `threading.Timer` class that comes with the standard library. `Timer` is a one-shot timer, as in, it only runs the given function once after the timeout, then ends the thread. To make the timer repeat, we start a new thread in the timer function. This is not great, because we're creating and destroying threads all the time, and as your bug report suggests, we can also run into scenarios where thread creation runs off. To remedy this, I opened (and already merged) #452. It works slightly differently to `Timer`, in that it re-runs the given function in the same thread, which should waste a bit less resources, as well as hopefully fix your issue. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #450 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[beniwohli]

I plan to do a release tomorrow :)

[rvanlaar]

Everything seems fine. The errors related to not starting threads aren't showing up in our logging.

[beniwohli]

@rvanlaar that's great news :) thanks for checking! I'll close this issue for now

[Kalanamith]

Issue still existst

/site-packages/elasticapm/metrics/base_metrics.py:104: in _start_collect_timer
    self._collect_timer.start()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <IntervalTimer(eapm metrics collect timer, initial daemon)>

    def start(self):
        """Start the thread's activity.
    
        It must be called at most once per thread object. It arranges for the
        object's run() method to be invoked in a separate thread of control.
    
        This method will raise a RuntimeError if called more than once on the
        same thread object.
    
        """
        if not self._initialized:
            raise RuntimeError("thread.__init__() not called")
    
        if self._started.is_set():
            raise RuntimeError("threads can only be started once")
        with _active_limbo_lock:
            _limbo[self] = self
        try:
>           _start_new_thread(self._bootstrap, ())
E           RuntimeError: can't start new thread