Add Trace.wrap decorator #30

elijahandrews · 2016-07-29T21:18:23Z

Say we have a function:

def f():
    meaning_of_life = 40 + 2
    return meaning_of_life

Right now, if we want to add tracing to the whole function, we'd do something like this:

def f():
    with tracer.trace('meaning_of_life'):
        meaning_of_life = 40 + 2
        return meaning_of_life

This is annoying because we have to indent all of code in the function, which makes diffs difficult to read and obscures git history. It also means we need an extra level of indentation which we may be able to avoid. Finally, if a function is long, it may not be obvious how much of the function the context handler applies to.

It would be cool if we could instead add a span around an entire function using a decorator:

@tracer.wrap()  # span name defaults to function name
def f():
    meaning_of_life = 40 + 2
    return meaning_of_life

This PR adds this functionality.

clutchski · 2016-07-30T11:11:07Z

tests/test_span.py

+    exception_occurred = False
+    try:
+        f()
+    except:


i think these usally are better using assert_raises or

try: whatever() except Exception: pass else: assert 0, "oops"

That's way nicer. Forgot about assert_raises and that you can throw else on a try block! 💯

clutchski · 2016-07-30T11:13:17Z

I omitted this for the exact reason you mentioned: you can't set tags, metadata, etc (which can be important) and I hate decorators because they make pdb difficult, are really inflexible, etc. But I suspect that eventually customer would ask for this.

If we do this, I think @tracer.trace() is better than @Span(tracer).

I guess we could always do

@tracer.trace()
def whatever();
     # do something
     span = tracer.current_span()
     span.set_tag(1, 2)

but that is a bit indirect. thoughts @elijahandrews @LotharSee?

elijahandrews · 2016-08-01T00:59:17Z

Thanks for the feedback. I think that we should have the decorator option, but only encourage its use when you want to instrument an entire function, and you don't want to set tags or metrics.

As soon as you want to instrument a smaller part of the function, or you want to add tags or metrics, we should encourage people to use one of the other, more flexible notations.

I think what will actually end up happening is that a developer's first pass on tracing a system will be to throw @tracer.trace() on top of a bunch of functions, so they can get a rough idea of where they're spending most of their time. After that, they'll probably want to add some child spans that give more specific information about some of the slower functions. So in the end, they'll up with code that looks like this:

@tracer.trace()
def some_quick_function():
    # No need to add extra spans since we barely spend any time in here

@tracer.trace()
def some_slow_function():
     # The bulk of the work is done in here, so we need finer tracing
    with @tracer.trace() as awesome_span:
        # do something awesome
    with @tracer.trace() as different_span:
        # do something different
        different_span.set_tag(1, 2)

clutchski · 2016-08-02T22:48:24Z

ddtrace/tracer.py

        if sampler is not None:
            self.sampler = sampler

+    def trace_function(self, name=None, service=None, resource=None, span_type=None):


instead of trace_function what about wrap?

@tracer.wrap('foo-bar') def abcdef(self): ...

elijahandrews · 2016-08-03T17:57:46Z

ddtrace/tracer.py

+    def wrap(self, name=None, service=None, resource=None, span_type=None):
+        def wrap_decorator(func):
+            if name is None:
+                span_name = '{}.{}'.format(func.__module__, func.__name__)


It's worth discussing whether we want to include the module name as well as the function name here.

Including the module name is good because we can tell exactly which function is being traced.

Including the module name is bad because we may get really long span names, and our interface doesn't display these well at this time.

I'm learning towards including the module name.

elijahandrews · 2016-08-03T19:16:11Z

ddtrace/tracer.py

+
+        def wrap_decorator(func):
+            if name is None:
+                span_name = '{}.{}'.format(func.__module__, func.__name__)


(comment got buried so reposting):

It's worth discussing whether we want to include the module name as well as the function name here.

Including the module name is good because we can tell exactly which function is being traced.

Including the module name is bad because we may get really long span names, and our interface doesn't display these well at this time.

I'm learning towards including the module name.

i really want to nudge people towards short spans. datalayer.metrics.query.spidly._parse_spidly will make me slightly insane. i guess its' ok though.

clutchski · 2016-08-03T19:34:53Z

ok go ahead.

## Description https://datadoghq.atlassian.net/browse/PROF-13112 This is an attempt to address the following crash. There seems to be a case (that I wasn't able to reproduce in a Docker image, but maybe my "code environment" didn't match the customer's exactly) where using `uvloop` results in a crash caused by `PeriodicThread_start` after `uvloop` tries to restart Threads after a fork. ``` #0 0x00007f9a7acdbefa cfree #1 0x00007f9a7accc6b5 pthread_create #2 0x00007f9a7a63aaa5 std::thread::_M_start_thread #3 0x00007f9a7a639d18 PeriodicThread_start #4 0x00007f9a2e71d565 __pyx_f_6uvloop_4loop_9UVProcess__after_fork (uvloop/loop.c:120214:3) #5 0x00007f9a2e6369a8 __pyx_f_6uvloop_4loop___get_fork_handler (uvloop/loop.c:163075:24) #6 0x00007f9a7ad17073 __fork #7 0x00007f9a2e732d62 uv__spawn_and_init_child_fork (src/unix/process.c:831:10) #8 0x00007f9a2e732d62 uv__spawn_and_init_child (src/unix/process.c:919:9) #9 0x00007f9a2e732d62 uv_spawn (src/unix/process.c:1013:18) #10 0x00007f9a2e71fb87 __pyx_f_6uvloop_4loop_9UVProcess__init (uvloop/loop.c:119056:19) #11 0x00007f9a2e711bf7 __pyx_f_6uvloop_4loop_18UVProcessTransport_new (uvloop/loop.c:126866:16) #12 0x00007f9a2e712aa7 __pyx_gb_6uvloop_4loop_4Loop_116generator16 (uvloop/loop.c:54030:28) #13 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #14 0x00007f9a2e699f8a __Pyx_Coroutine_AmSend (uvloop/loop.c:196492:18) #15 0x00007f9a2e69a052 __Pyx_Coroutine_Yield_From_Coroutine (uvloop/loop.c:197380:14) #16 0x00007f9a2e69b0e5 __Pyx_Coroutine_Yield_From (uvloop/loop.c:197408:16) #17 0x00007f9a2e69b0e5 __pyx_gb_6uvloop_4loop_4Loop_122generator18 (uvloop/loop.c:55002:15) #18 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #19 0x00007f9a2e69bb86 __Pyx_Generator_Next (uvloop/loop.c:196581:18) #20 0x00007f9a2e6398eb __Pyx_PyObject_Call (uvloop/loop.c:191431:15) #21 0x00007f9a2e6398eb __Pyx_PyObject_FastCallDict (uvloop/loop.c:191552:16) #22 0x00007f9a2e715a69 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66873:27) #23 0x00007f9a2e71996b __pyx_f_6uvloop_4loop_4Loop__on_idle (uvloop/loop.c:17975:25) #24 0x00007f9a2e713e52 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66927:24) #25 0x00007f9a2e715c88 __pyx_f_6uvloop_4loop_cb_idle_callback (uvloop/loop.c:87335:19) #26 0x00007f9a2e731311 uv__run_idle (unix/loop-watcher.c:68:1) #27 0x00007f9a2e72e647 uv_run (src/unix/core.c:439:5) #28 0x00007f9a2e64fdb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (uvloop/loop.c:18458:23) #29 0x00007f9a2e6b7e50 __pyx_f_6uvloop_4loop_4Loop__run (uvloop/loop.c:18876:18) #30 0x00007f9a2e6c8cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (uvloop/loop.c:31528:18) #31 0x00007f9a2e6c8cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (uvloop/loop.c:31331:13) #32 0x00007f9a7b065c25 PyObject_VectorcallMethod #33 0x00007f9a2e6ccd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (uvloop/loop.c:33768:23) #34 0x00007f9a2e6ce591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (uvloop/loop.c:33318:13) #35 0x00007f9a7b039358 PyObject_Vectorcall ``` ## Fix The `_after_fork` boolean field marks that this thread object is in a "post-fork zombie state." When the flag is set to true, Thread methods (e.g. `join`) become no-ops because the threads do not exist anymore so we should not try to do something with them. By checking that same flag, we can tell that we are trying to start a Thread that doesn't really exist and so we shouldn't try to do it.

## Description https://datadoghq.atlassian.net/browse/PROF-13112 This is an attempt to address the following crash. There seems to be a case (that I wasn't able to reproduce in a Docker image, but maybe my "code environment" didn't match the customer's exactly) where using `uvloop` results in a crash caused by `PeriodicThread_start` after `uvloop` tries to restart Threads after a fork. ``` #0 0x00007f9a7acdbefa cfree #1 0x00007f9a7accc6b5 pthread_create #2 0x00007f9a7a63aaa5 std::thread::_M_start_thread #3 0x00007f9a7a639d18 PeriodicThread_start #4 0x00007f9a2e71d565 __pyx_f_6uvloop_4loop_9UVProcess__after_fork (uvloop/loop.c:120214:3) #5 0x00007f9a2e6369a8 __pyx_f_6uvloop_4loop___get_fork_handler (uvloop/loop.c:163075:24) #6 0x00007f9a7ad17073 __fork #7 0x00007f9a2e732d62 uv__spawn_and_init_child_fork (src/unix/process.c:831:10) #8 0x00007f9a2e732d62 uv__spawn_and_init_child (src/unix/process.c:919:9) #9 0x00007f9a2e732d62 uv_spawn (src/unix/process.c:1013:18) #10 0x00007f9a2e71fb87 __pyx_f_6uvloop_4loop_9UVProcess__init (uvloop/loop.c:119056:19) #11 0x00007f9a2e711bf7 __pyx_f_6uvloop_4loop_18UVProcessTransport_new (uvloop/loop.c:126866:16) #12 0x00007f9a2e712aa7 __pyx_gb_6uvloop_4loop_4Loop_116generator16 (uvloop/loop.c:54030:28) #13 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #14 0x00007f9a2e699f8a __Pyx_Coroutine_AmSend (uvloop/loop.c:196492:18) #15 0x00007f9a2e69a052 __Pyx_Coroutine_Yield_From_Coroutine (uvloop/loop.c:197380:14) #16 0x00007f9a2e69b0e5 __Pyx_Coroutine_Yield_From (uvloop/loop.c:197408:16) #17 0x00007f9a2e69b0e5 __pyx_gb_6uvloop_4loop_4Loop_122generator18 (uvloop/loop.c:55002:15) #18 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #19 0x00007f9a2e69bb86 __Pyx_Generator_Next (uvloop/loop.c:196581:18) #20 0x00007f9a2e6398eb __Pyx_PyObject_Call (uvloop/loop.c:191431:15) #21 0x00007f9a2e6398eb __Pyx_PyObject_FastCallDict (uvloop/loop.c:191552:16) #22 0x00007f9a2e715a69 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66873:27) #23 0x00007f9a2e71996b __pyx_f_6uvloop_4loop_4Loop__on_idle (uvloop/loop.c:17975:25) #24 0x00007f9a2e713e52 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66927:24) #25 0x00007f9a2e715c88 __pyx_f_6uvloop_4loop_cb_idle_callback (uvloop/loop.c:87335:19) #26 0x00007f9a2e731311 uv__run_idle (unix/loop-watcher.c:68:1) #27 0x00007f9a2e72e647 uv_run (src/unix/core.c:439:5) #28 0x00007f9a2e64fdb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (uvloop/loop.c:18458:23) #29 0x00007f9a2e6b7e50 __pyx_f_6uvloop_4loop_4Loop__run (uvloop/loop.c:18876:18) #30 0x00007f9a2e6c8cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (uvloop/loop.c:31528:18) #31 0x00007f9a2e6c8cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (uvloop/loop.c:31331:13) #32 0x00007f9a7b065c25 PyObject_VectorcallMethod #33 0x00007f9a2e6ccd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (uvloop/loop.c:33768:23) #34 0x00007f9a2e6ce591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (uvloop/loop.c:33318:13) #35 0x00007f9a7b039358 PyObject_Vectorcall ``` ## Fix The `_after_fork` boolean field marks that this thread object is in a "post-fork zombie state." When the flag is set to true, Thread methods (e.g. `join`) become no-ops because the threads do not exist anymore so we should not try to do something with them. By checking that same flag, we can tell that we are trying to start a Thread that doesn't really exist and so we shouldn't try to do it. (cherry picked from commit 4c69fdd)

…15858) Backport 4c69fdd from #15798 to 4.1. ## Description https://datadoghq.atlassian.net/browse/PROF-13112 This is an attempt to address the following crash. There seems to be a case (that I wasn't able to reproduce in a Docker image, but maybe my "code environment" didn't match the customer's exactly) where using `uvloop` results in a crash caused by `PeriodicThread_start` after `uvloop` tries to restart Threads after a fork. ``` #0 0x00007f9a7acdbefa cfree #1 0x00007f9a7accc6b5 pthread_create #2 0x00007f9a7a63aaa5 std::thread::_M_start_thread #3 0x00007f9a7a639d18 PeriodicThread_start #4 0x00007f9a2e71d565 __pyx_f_6uvloop_4loop_9UVProcess__after_fork (uvloop/loop.c:120214:3) #5 0x00007f9a2e6369a8 __pyx_f_6uvloop_4loop___get_fork_handler (uvloop/loop.c:163075:24) #6 0x00007f9a7ad17073 __fork #7 0x00007f9a2e732d62 uv__spawn_and_init_child_fork (src/unix/process.c:831:10) #8 0x00007f9a2e732d62 uv__spawn_and_init_child (src/unix/process.c:919:9) #9 0x00007f9a2e732d62 uv_spawn (src/unix/process.c:1013:18) #10 0x00007f9a2e71fb87 __pyx_f_6uvloop_4loop_9UVProcess__init (uvloop/loop.c:119056:19) #11 0x00007f9a2e711bf7 __pyx_f_6uvloop_4loop_18UVProcessTransport_new (uvloop/loop.c:126866:16) #12 0x00007f9a2e712aa7 __pyx_gb_6uvloop_4loop_4Loop_116generator16 (uvloop/loop.c:54030:28) #13 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #14 0x00007f9a2e699f8a __Pyx_Coroutine_AmSend (uvloop/loop.c:196492:18) #15 0x00007f9a2e69a052 __Pyx_Coroutine_Yield_From_Coroutine (uvloop/loop.c:197380:14) #16 0x00007f9a2e69b0e5 __Pyx_Coroutine_Yield_From (uvloop/loop.c:197408:16) #17 0x00007f9a2e69b0e5 __pyx_gb_6uvloop_4loop_4Loop_122generator18 (uvloop/loop.c:55002:15) #18 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #19 0x00007f9a2e69bb86 __Pyx_Generator_Next (uvloop/loop.c:196581:18) #20 0x00007f9a2e6398eb __Pyx_PyObject_Call (uvloop/loop.c:191431:15) #21 0x00007f9a2e6398eb __Pyx_PyObject_FastCallDict (uvloop/loop.c:191552:16) #22 0x00007f9a2e715a69 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66873:27) #23 0x00007f9a2e71996b __pyx_f_6uvloop_4loop_4Loop__on_idle (uvloop/loop.c:17975:25) #24 0x00007f9a2e713e52 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66927:24) #25 0x00007f9a2e715c88 __pyx_f_6uvloop_4loop_cb_idle_callback (uvloop/loop.c:87335:19) #26 0x00007f9a2e731311 uv__run_idle (unix/loop-watcher.c:68:1) #27 0x00007f9a2e72e647 uv_run (src/unix/core.c:439:5) #28 0x00007f9a2e64fdb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (uvloop/loop.c:18458:23) #29 0x00007f9a2e6b7e50 __pyx_f_6uvloop_4loop_4Loop__run (uvloop/loop.c:18876:18) #30 0x00007f9a2e6c8cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (uvloop/loop.c:31528:18) #31 0x00007f9a2e6c8cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (uvloop/loop.c:31331:13) #32 0x00007f9a7b065c25 PyObject_VectorcallMethod #33 0x00007f9a2e6ccd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (uvloop/loop.c:33768:23) #34 0x00007f9a2e6ce591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (uvloop/loop.c:33318:13) #35 0x00007f9a7b039358 PyObject_Vectorcall ``` ## Fix The `_after_fork` boolean field marks that this thread object is in a "post-fork zombie state." When the flag is set to true, Thread methods (e.g. `join`) become no-ops because the threads do not exist anymore so we should not try to do something with them. By checking that same flag, we can tell that we are trying to start a Thread that doesn't really exist and so we shouldn't try to do it. Co-authored-by: Thomas Kowalski <thomas.kowalski@datadoghq.com>

## Description https://datadoghq.atlassian.net/browse/PROF-13112 This is an attempt to address the following crash. There seems to be a case (that I wasn't able to reproduce in a Docker image, but maybe my "code environment" didn't match the customer's exactly) where using `uvloop` results in a crash caused by `PeriodicThread_start` after `uvloop` tries to restart Threads after a fork. ``` #0 0x00007f9a7acdbefa cfree #1 0x00007f9a7accc6b5 pthread_create #2 0x00007f9a7a63aaa5 std::thread::_M_start_thread #3 0x00007f9a7a639d18 PeriodicThread_start #4 0x00007f9a2e71d565 __pyx_f_6uvloop_4loop_9UVProcess__after_fork (uvloop/loop.c:120214:3) #5 0x00007f9a2e6369a8 __pyx_f_6uvloop_4loop___get_fork_handler (uvloop/loop.c:163075:24) #6 0x00007f9a7ad17073 __fork #7 0x00007f9a2e732d62 uv__spawn_and_init_child_fork (src/unix/process.c:831:10) #8 0x00007f9a2e732d62 uv__spawn_and_init_child (src/unix/process.c:919:9) #9 0x00007f9a2e732d62 uv_spawn (src/unix/process.c:1013:18) #10 0x00007f9a2e71fb87 __pyx_f_6uvloop_4loop_9UVProcess__init (uvloop/loop.c:119056:19) #11 0x00007f9a2e711bf7 __pyx_f_6uvloop_4loop_18UVProcessTransport_new (uvloop/loop.c:126866:16) #12 0x00007f9a2e712aa7 __pyx_gb_6uvloop_4loop_4Loop_116generator16 (uvloop/loop.c:54030:28) #13 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #14 0x00007f9a2e699f8a __Pyx_Coroutine_AmSend (uvloop/loop.c:196492:18) #15 0x00007f9a2e69a052 __Pyx_Coroutine_Yield_From_Coroutine (uvloop/loop.c:197380:14) #16 0x00007f9a2e69b0e5 __Pyx_Coroutine_Yield_From (uvloop/loop.c:197408:16) #17 0x00007f9a2e69b0e5 __pyx_gb_6uvloop_4loop_4Loop_122generator18 (uvloop/loop.c:55002:15) #18 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #19 0x00007f9a2e69bb86 __Pyx_Generator_Next (uvloop/loop.c:196581:18) #20 0x00007f9a2e6398eb __Pyx_PyObject_Call (uvloop/loop.c:191431:15) #21 0x00007f9a2e6398eb __Pyx_PyObject_FastCallDict (uvloop/loop.c:191552:16) #22 0x00007f9a2e715a69 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66873:27) #23 0x00007f9a2e71996b __pyx_f_6uvloop_4loop_4Loop__on_idle (uvloop/loop.c:17975:25) #24 0x00007f9a2e713e52 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66927:24) #25 0x00007f9a2e715c88 __pyx_f_6uvloop_4loop_cb_idle_callback (uvloop/loop.c:87335:19) #26 0x00007f9a2e731311 uv__run_idle (unix/loop-watcher.c:68:1) #27 0x00007f9a2e72e647 uv_run (src/unix/core.c:439:5) #28 0x00007f9a2e64fdb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (uvloop/loop.c:18458:23) #29 0x00007f9a2e6b7e50 __pyx_f_6uvloop_4loop_4Loop__run (uvloop/loop.c:18876:18) #30 0x00007f9a2e6c8cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (uvloop/loop.c:31528:18) #31 0x00007f9a2e6c8cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (uvloop/loop.c:31331:13) #32 0x00007f9a7b065c25 PyObject_VectorcallMethod #33 0x00007f9a2e6ccd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (uvloop/loop.c:33768:23) #34 0x00007f9a2e6ce591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (uvloop/loop.c:33318:13) #35 0x00007f9a7b039358 PyObject_Vectorcall ``` ## Fix The `_after_fork` boolean field marks that this thread object is in a "post-fork zombie state." When the flag is set to true, Thread methods (e.g. `join`) become no-ops because the threads do not exist anymore so we should not try to do something with them. By checking that same flag, we can tell that we are trying to start a Thread that doesn't really exist and so we shouldn't try to do it. (cherry picked from commit 4c69fdd)

…15861) Backport 4c69fdd from #15798 to 4.0. ## Description https://datadoghq.atlassian.net/browse/PROF-13112 This is an attempt to address the following crash. There seems to be a case (that I wasn't able to reproduce in a Docker image, but maybe my "code environment" didn't match the customer's exactly) where using `uvloop` results in a crash caused by `PeriodicThread_start` after `uvloop` tries to restart Threads after a fork. ``` #0 0x00007f9a7acdbefa cfree #1 0x00007f9a7accc6b5 pthread_create #2 0x00007f9a7a63aaa5 std::thread::_M_start_thread #3 0x00007f9a7a639d18 PeriodicThread_start #4 0x00007f9a2e71d565 __pyx_f_6uvloop_4loop_9UVProcess__after_fork (uvloop/loop.c:120214:3) #5 0x00007f9a2e6369a8 __pyx_f_6uvloop_4loop___get_fork_handler (uvloop/loop.c:163075:24) #6 0x00007f9a7ad17073 __fork #7 0x00007f9a2e732d62 uv__spawn_and_init_child_fork (src/unix/process.c:831:10) #8 0x00007f9a2e732d62 uv__spawn_and_init_child (src/unix/process.c:919:9) #9 0x00007f9a2e732d62 uv_spawn (src/unix/process.c:1013:18) #10 0x00007f9a2e71fb87 __pyx_f_6uvloop_4loop_9UVProcess__init (uvloop/loop.c:119056:19) #11 0x00007f9a2e711bf7 __pyx_f_6uvloop_4loop_18UVProcessTransport_new (uvloop/loop.c:126866:16) #12 0x00007f9a2e712aa7 __pyx_gb_6uvloop_4loop_4Loop_116generator16 (uvloop/loop.c:54030:28) #13 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #14 0x00007f9a2e699f8a __Pyx_Coroutine_AmSend (uvloop/loop.c:196492:18) #15 0x00007f9a2e69a052 __Pyx_Coroutine_Yield_From_Coroutine (uvloop/loop.c:197380:14) #16 0x00007f9a2e69b0e5 __Pyx_Coroutine_Yield_From (uvloop/loop.c:197408:16) #17 0x00007f9a2e69b0e5 __pyx_gb_6uvloop_4loop_4Loop_122generator18 (uvloop/loop.c:55002:15) #18 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #19 0x00007f9a2e69bb86 __Pyx_Generator_Next (uvloop/loop.c:196581:18) #20 0x00007f9a2e6398eb __Pyx_PyObject_Call (uvloop/loop.c:191431:15) #21 0x00007f9a2e6398eb __Pyx_PyObject_FastCallDict (uvloop/loop.c:191552:16) #22 0x00007f9a2e715a69 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66873:27) #23 0x00007f9a2e71996b __pyx_f_6uvloop_4loop_4Loop__on_idle (uvloop/loop.c:17975:25) #24 0x00007f9a2e713e52 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66927:24) #25 0x00007f9a2e715c88 __pyx_f_6uvloop_4loop_cb_idle_callback (uvloop/loop.c:87335:19) #26 0x00007f9a2e731311 uv__run_idle (unix/loop-watcher.c:68:1) #27 0x00007f9a2e72e647 uv_run (src/unix/core.c:439:5) #28 0x00007f9a2e64fdb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (uvloop/loop.c:18458:23) #29 0x00007f9a2e6b7e50 __pyx_f_6uvloop_4loop_4Loop__run (uvloop/loop.c:18876:18) #30 0x00007f9a2e6c8cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (uvloop/loop.c:31528:18) #31 0x00007f9a2e6c8cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (uvloop/loop.c:31331:13) #32 0x00007f9a7b065c25 PyObject_VectorcallMethod #33 0x00007f9a2e6ccd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (uvloop/loop.c:33768:23) #34 0x00007f9a2e6ce591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (uvloop/loop.c:33318:13) #35 0x00007f9a7b039358 PyObject_Vectorcall ``` ## Fix The `_after_fork` boolean field marks that this thread object is in a "post-fork zombie state." When the flag is set to true, Thread methods (e.g. `join`) become no-ops because the threads do not exist anymore so we should not try to do something with them. By checking that same flag, we can tell that we are trying to start a Thread that doesn't really exist and so we shouldn't try to do it. Co-authored-by: Thomas Kowalski <thomas.kowalski@datadoghq.com>

## Description https://datadoghq.atlassian.net/browse/PROF-13112 This is an attempt to address the following crash. There seems to be a case (that I wasn't able to reproduce in a Docker image, but maybe my "code environment" didn't match the customer's exactly) where using `uvloop` results in a crash caused by `PeriodicThread_start` after `uvloop` tries to restart Threads after a fork. ``` #0 0x00007f9a7acdbefa cfree DataDog#1 0x00007f9a7accc6b5 pthread_create DataDog#2 0x00007f9a7a63aaa5 std::thread::_M_start_thread DataDog#3 0x00007f9a7a639d18 PeriodicThread_start DataDog#4 0x00007f9a2e71d565 __pyx_f_6uvloop_4loop_9UVProcess__after_fork (uvloop/loop.c:120214:3) DataDog#5 0x00007f9a2e6369a8 __pyx_f_6uvloop_4loop___get_fork_handler (uvloop/loop.c:163075:24) DataDog#6 0x00007f9a7ad17073 __fork DataDog#7 0x00007f9a2e732d62 uv__spawn_and_init_child_fork (src/unix/process.c:831:10) DataDog#8 0x00007f9a2e732d62 uv__spawn_and_init_child (src/unix/process.c:919:9) DataDog#9 0x00007f9a2e732d62 uv_spawn (src/unix/process.c:1013:18) DataDog#10 0x00007f9a2e71fb87 __pyx_f_6uvloop_4loop_9UVProcess__init (uvloop/loop.c:119056:19) DataDog#11 0x00007f9a2e711bf7 __pyx_f_6uvloop_4loop_18UVProcessTransport_new (uvloop/loop.c:126866:16) DataDog#12 0x00007f9a2e712aa7 __pyx_gb_6uvloop_4loop_4Loop_116generator16 (uvloop/loop.c:54030:28) DataDog#13 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) DataDog#14 0x00007f9a2e699f8a __Pyx_Coroutine_AmSend (uvloop/loop.c:196492:18) DataDog#15 0x00007f9a2e69a052 __Pyx_Coroutine_Yield_From_Coroutine (uvloop/loop.c:197380:14) DataDog#16 0x00007f9a2e69b0e5 __Pyx_Coroutine_Yield_From (uvloop/loop.c:197408:16) DataDog#17 0x00007f9a2e69b0e5 __pyx_gb_6uvloop_4loop_4Loop_122generator18 (uvloop/loop.c:55002:15) DataDog#18 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) DataDog#19 0x00007f9a2e69bb86 __Pyx_Generator_Next (uvloop/loop.c:196581:18) DataDog#20 0x00007f9a2e6398eb __Pyx_PyObject_Call (uvloop/loop.c:191431:15) DataDog#21 0x00007f9a2e6398eb __Pyx_PyObject_FastCallDict (uvloop/loop.c:191552:16) DataDog#22 0x00007f9a2e715a69 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66873:27) DataDog#23 0x00007f9a2e71996b __pyx_f_6uvloop_4loop_4Loop__on_idle (uvloop/loop.c:17975:25) DataDog#24 0x00007f9a2e713e52 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66927:24) DataDog#25 0x00007f9a2e715c88 __pyx_f_6uvloop_4loop_cb_idle_callback (uvloop/loop.c:87335:19) DataDog#26 0x00007f9a2e731311 uv__run_idle (unix/loop-watcher.c:68:1) DataDog#27 0x00007f9a2e72e647 uv_run (src/unix/core.c:439:5) DataDog#28 0x00007f9a2e64fdb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (uvloop/loop.c:18458:23) DataDog#29 0x00007f9a2e6b7e50 __pyx_f_6uvloop_4loop_4Loop__run (uvloop/loop.c:18876:18) DataDog#30 0x00007f9a2e6c8cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (uvloop/loop.c:31528:18) DataDog#31 0x00007f9a2e6c8cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (uvloop/loop.c:31331:13) DataDog#32 0x00007f9a7b065c25 PyObject_VectorcallMethod DataDog#33 0x00007f9a2e6ccd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (uvloop/loop.c:33768:23) DataDog#34 0x00007f9a2e6ce591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (uvloop/loop.c:33318:13) DataDog#35 0x00007f9a7b039358 PyObject_Vectorcall ``` ## Fix The `_after_fork` boolean field marks that this thread object is in a "post-fork zombie state." When the flag is set to true, Thread methods (e.g. `join`) become no-ops because the threads do not exist anymore so we should not try to do something with them. By checking that same flag, we can tell that we are trying to start a Thread that doesn't really exist and so we shouldn't try to do it.

First pass on make Span usable as a function decorator

9bf2df8

clutchski reviewed Jul 30, 2016
View reviewed changes

elijahandrews added 2 commits August 1, 2016 10:54

Clean up test_decorator_exception by using assert_raises

c9c2bef

Add tracer.trace_function

845eddc

elijahandrews force-pushed the elijah/decorator branch from c8d99b4 to 845eddc Compare August 2, 2016 22:02

clutchski reviewed Aug 2, 2016
View reviewed changes

elijahandrews added 5 commits August 3, 2016 13:23

Remove decorator from Span

9564ccb

Tracer.trace_function -> Tracer.wrap, add more wrap tests

f30c4e0

Remove unused import from span

6fa5c74

Remove unused import from span tests

78968d9

Merge branch 'master' into elijah/decorator

089021d

elijahandrews changed the title ~~[WIP] Trace with function decorator~~ Add Trace.wrap decorator Aug 3, 2016

Include module name in default span name when wrapping functions

c73e1b3

elijahandrews reviewed Aug 3, 2016
View reviewed changes

elijahandrews added 4 commits August 3, 2016 13:58

Fix typo in test_tracer_wrap_span_nesting comment

44d4f72

Add docs for tracer.wrap

8faed3b

Add test for setting tags within wrapped function

ff0fbef

Make sure that we can pass args to function wrapped with trace.wrap

a582a89

elijahandrews reviewed Aug 3, 2016
View reviewed changes

elijahandrews merged commit 7a462f2 into master Aug 3, 2016

elijahandrews deleted the elijah/decorator branch August 3, 2016 20:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Trace.wrap decorator #30

Add Trace.wrap decorator #30

Uh oh!

elijahandrews commented Jul 29, 2016 •

edited

Loading

Uh oh!

clutchski Jul 30, 2016

Uh oh!

elijahandrews Aug 1, 2016

Uh oh!

clutchski commented Jul 30, 2016 •

edited

Loading

Uh oh!

elijahandrews commented Aug 1, 2016

Uh oh!

clutchski Aug 2, 2016 •

edited

Loading

Uh oh!

elijahandrews Aug 2, 2016

Uh oh!

elijahandrews Aug 3, 2016

Uh oh!

elijahandrews Aug 3, 2016

Uh oh!

clutchski Aug 3, 2016

Uh oh!

clutchski commented Aug 3, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add Trace.wrap decorator #30

Add Trace.wrap decorator #30

Uh oh!

Conversation

elijahandrews commented Jul 29, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clutchski Jul 30, 2016

Choose a reason for hiding this comment

Uh oh!

elijahandrews Aug 1, 2016

Choose a reason for hiding this comment

Uh oh!

clutchski commented Jul 30, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elijahandrews commented Aug 1, 2016

Uh oh!

clutchski Aug 2, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elijahandrews Aug 2, 2016

Choose a reason for hiding this comment

Uh oh!

elijahandrews Aug 3, 2016

Choose a reason for hiding this comment

Uh oh!

elijahandrews Aug 3, 2016

Choose a reason for hiding this comment

Uh oh!

clutchski Aug 3, 2016

Choose a reason for hiding this comment

Uh oh!

clutchski commented Aug 3, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

elijahandrews commented Jul 29, 2016 •

edited

Loading

clutchski commented Jul 30, 2016 •

edited

Loading

clutchski Aug 2, 2016 •

edited

Loading