CI Tests beta feature: Is it possible to log errors/failures from beforeHooks

[declancarrollcb]

I've followed the setup instructions for integrating Datadog tracing into our Jest Circus tests. Most of the time it's working fine, however if there is an error/exception in the "before" hooks of a suite, there is no error sent to Datadog.

This gives a very inaccurate impression of what happened from the Dashboard view. Has anyone else encountered this?

[rochdev]

cc @juan-fernandez

[juan-fernandez]

hi @declancarrollcb ! Thanks for reaching out. This is a great observation. We're indeed missing visibility over what happens in before / after kind of hooks. We map tests to traces with a test span as parent, but we haven't figured out exactly what to do with hooks that are associated to a whole suite (or any set of tests). It's on our roadmap to work on a solution, but we don't have a fixed timeline for this. What would you find sensible in this case?

[declancarrollcb]

Hey Juan,
Thanks for the reply. In my opinion for it would make sense to include other types of spans related to testing. For instance most testing frameworks support some manner of "setup/teardown".

In the Datadog tracing tags I can see a test suite attribute, so it would make sense to try wrap a suite span also. It could be included as an optional column on the main test dashboard though. It would also be a valuable metric to track the longest individual test times, and the aggregated longest test suite times, that would be particularly useful for people who use Jest as an E2E UI framework where a single test suite is broken up into multiple steps for a test.

[juan-fernandez]

For instance most testing frameworks support some manner of "setup/teardown".

Indeed. We need to find a solution for every test framework and language.

It would also be a valuable metric to track the longest individual test times, and the aggregated longest test suite times,

Great that you bring this up, because we've received this same feedback already. An option we're considering is to have a trace associated to the whole "test run", which would include test suites as children spans (together with their setup / teardown hooks) and tests (together with their hooks too) as children of those test suite spans.

We can keep this discussion open and I'll add updates as we advance towards a solution 😄

Thanks again!