Multiple tag values: how to get Cypress or Playwright test tags into Test Visibility

[cgutierrezpa]

Hello community,

So this question is oriented at enriching the metadata for the test runs that are recorded into Datadog with the Test Visibility feature. We are currently migrating from Cypress to Playwright, and to get more visibility on what's going on with the testing in our pipelines, we want to add more tags and metadata to each test run. One of the things that we want to do is to add the tags that a test already has, i.e. both in Cypress and Playwright we can basically tags: ["@smoke", "@third-party"] and so on, and then we can filter tests including those tags (or run them by tag). However, these tags are not natively sent to Datadog from what I've seen so far

Ideally, we would like to push an array of these tag values to some names tag in Datadog, but from what I've researched so far in the docs, it seems like a tag is basically a key-value pair, so for the same key, we can't have multiple values (as we would like in this case, something like TestTag: ["@smoke", "@third-party"]. Some of the alternatives that I could think of were:

• Concatenating the string values with some separator like ;
  • The problem is that this increases the cardinality of the values greatly, because in case of having 3 tag values, there can be this many combinations (in case we sort them alphabetically first), and I don't know if this affects the pricing of the injected data
    • @tag-1
    • @tag-2
    • @tag-3
    • @tag-1;@tag-2
    • @tag-1;@tag-3
    • @tag-2;@tag-3
    • @tag-1;@tag-2;@tag-3
• Defining each tag value as a key, having true as value, i.e. TestTag.@smoke: true or TestTag.@third-party: true
  • Not much problem with this rather than it's ugly and makes the facets not very intuitive

Has anyone come up with a better approach? Any help is very much appreciated 😄

[cgutierrezpa]

Probably @juan-fernandez can shed some light in this topic? 🙏

[juan-fernandez]

hey @cgutierrezpa! Thanks for opening the discussion.
I'll try to go point by point.

One of the things that we want to do is to add the tags that a test already has, i.e. both in Cypress and Playwright we can basically tags: ["@smoke", "@third-party"] and so on, and then we can filter tests including those tags (or run them by tag). However, these tags are not natively sent to Datadog from what I've seen so far

Could you tell me exactly how do you set those per test tags in your code? Is it by using playwright's tags? https://playwright.dev/docs/test-annotations#tag-tests

test('test login page', {
  tag: '@fast',
}, async ({ page }) => {
  // ...
});

test('test full report @slow', async ({ page }) => {
  // ...
});

This is something that we'd like to support natively so any piece of information is welcome.

Ideally, we would like to push an array of these tag values to some names tag in Datadog, but from what I've researched so far in the docs, it seems like a tag is basically a key-value pair, so for the same key, we can't have multiple values (as we would like in this case, something like TestTag: ["@smoke", "@third-party"].

Arrays are indeed not supported for arbitrary tags. Custom tags are supported by playwright annotations: https://docs.datadoghq.com/tests/setup/javascript/?tab=playwright#adding-custom-tags-to-tests, but only strings are supported. You could do something like:

test('user profile', async ({ page }) => {
  test.info().annotations.push({
    type: 'DD_TAGS[test.memory.array]', // DD_TAGS is mandatory and case sensitive
    description: JSON.stringify(["@smoke", "@third-party"]),
  });

But the experience wouldn't be great, as you'd have to use string query params to do search, like:

@test.memory.array:*@smoke*

I think this would be essentially the same as your suggested approach with ;.

and I don't know if this affects the pricing of the injected data

It does not 😄

Defining each tag value as a key, having true as value, i.e. TestTag.@smoke: true or TestTag.@third-party: true

This is probably the approach that makes filtering the easiest, though I agree having a long list of facets is not the greatest.

To sum up:

1. I'd like to know how you use playwright tags so we support them natively.
2. We (DataDog) need to come up with a better way to have array-like facets.
3. Until the two above are tackled, either of the options you describe works, and I don't think there's any better alternative.

[cgutierrezpa]

This might be the problem. Sadly, the only place where you can call tracer.scope().active() is within the test function (that is, the it/test block). Sadly, jest makes it hard to instrument a single "test function" including beforeEach and afterEach hooks.

Do you think that moving everything to a separate function like we did with the utils example could also affect to the "null span" issue? I moved the function within the it/test block and still doesn't work :(

# run locally
DD_API_KEY=$MY_API_KEY DD_CIVISIBILITY_AGENTLESS_ENABLED=1 NODE_OPTIONS='-r dd-trace/ci/init' DD_ENV=local DD_SERVICE=my-local-tests yarn nx run-many --target=test --configuration=integration-all

This is actually what I tried but didn't work. Will try more configuration combinations

[juan-fernandez]

Do you think that moving everything to a separate function like we did with the utils example could also affect to the "null span" issue? I moved the function within the it/test block and still doesn't work :(

It should not 🤔 . Could you share what the function looks like? And where in the test you're calling it.

This is actually what I tried but didn't work. Will try more configuration combinations

You can pass DD_TRACE_DEBUG=1 env var. This should tell you if there's any connection problem (which is the most likely)

[juan-fernandez]

hey @cgutierrezpa did you have any luck with DD_TRACE_DEBUG ?

[cgutierrezpa]

Hey @juan-fernandez! Sorry for the delayed response. So here are the conclusions that I came up with:

• The environment variables were not working when running the tests via yarn nx run command, neither adding them before the command as in ENV_VAR=value yarn nx run test nor defining them in a .env file. I don't really know the root cause, as we have multiple commands defined in Nx and we are able to make these environment variables work (regardless of the variable definition mentioned before), but in this case it didn't work, that's why I couldn't see any debug output in the console
• Running it with ENV_VAR=value node <path_to_jest> ... is what worked. Every environment variable was correctly set up, but the problem was the command
• The reason why we weren't seeing the tags in the CI tests was that the tests we performed were done with the addTags() function added to the beforeEach hook. We changed it to the it/test blocks and now we are seeing the data correctly. It's a bit annoying but it is what is it 😄
• Using tracer.addTags works as fine as tracer.setTag. Much more convenient tho
• As mentioned in one comment above, we went ahead with the JSON.stringify([tagsArray]) solution, as we are able to easily query the data with @TestTag:(*\"tag-1\"* OR *\"tag-2\"*). For the facets you can have predefined values so is not that bad

Thank you so much for the help @juan-fernandez! Really appreciated

[juan-fernandez]

thanks for the extra context!