Exclude 'unknown' scenarios from html report

[oellul]

🤔 What's the problem you're trying to solve?

The HTML report currently includes all specified scenarios, even those that aren't being run. This, in my opinion, makes the report less intuitive, as there can potentially be a lot to visually filter through just to see the scenarios that were run. Also I don't see any reason to include scenarios that are purposely tagged out of the run.

✨ What's your proposed solution?

Probably the simplest solution would be to have the "unknown" scenario status filter disabled by default.
IMO a more preferable solution would be to completely omit them from the report, as I just don't see the point of them being there.

⛏ Have you considered any alternatives or workarounds?

Yes, an easy workaround is to hide the scenarios just by deselecting the "unknown" status filter, but just having them displayed by default may be counter-intuitive especially to those within an organisation not particularly well versed in Cucumber and only interested in seeing Pass/Fail scenarios.

Additional context

To reproduce, simply tag a scenario/feature with @wip, and run cucumber with not @wip in the tag expression.

[mattwynne]

For extra context, you're using cucumber-ruby, right @oellul?

This formatter is cross-platform, and I'm not clear yet whether this happens for all flavours of Cucumber.

[mattwynne]

I wonder if it would be best to work this into the UX as a switch that comes alive if there are scenarios with the unknown result status:

• We default to not showing scenarios with unknown status in the report
• If unknown scenarios exist, offer a toggle to hide / show them.
• We tell the user the number of unknown scenarios that are not being displayed

e.g.

---

15/50 scenarios were not tested and are not currently displayed.

• Display all scenarios

---

[mattwynne]

For context, it's possible that (at least in the future) we could stream messages to something displaying the HTML report components while the tests are running and so the unknown status could be something that changes dynamically. I don't think we need to consider this scenario at the moment, but it's preferable to also not make decisions that will make it more difficult in the future.

[davidjgoss]

This formatter is cross-platform, and I'm not clear yet whether this happens for all flavours of Cucumber.

cucumber-js does the same FWIW - if you filter some scenarios out via tags or whatever, they get unknown status.

Tend to agree with @oellul that it's not useful to include them in the same format as the scenarios that did run. Maybe a separate list of "scenarios that didn't run because of your filtering" would work, but I think just omitting them would also be an improvement for now, because the unknown status is confusing.

Also, I'd note this is not to be conflated with the skipped status you'd get from doing return 'skipped' from a hook or step - they should stay in.

[oellul]

For extra context, you're using cucumber-ruby, right @oellul?

Yes, that's right.

I agree with both of these ideas - to offer a toggle to hide/show 'unknown' scenarios (the default being to hide them) and having them show in a separate list seems like an ideal solution.

Also, I'd note this is not to be conflated with the skipped status you'd get from doing return 'skipped' from a hook or step - they should stay in.

Changing the title to avoid any confusion...

[davidjgoss]

For context, it's possible that (at least in the future) we could stream messages to something displaying the HTML report components while the tests are running and so the unknown status could be something that changes dynamically. I don't think we need to consider this scenario at the moment, but it's preferable to also not make decisions that will make it more difficult in the future.

That's a good point! I think unknown as the status for these is basically a fallback in either the React components because we don't find a status. I think we could reliably infer which scenarios are "omitted" by checking if there is a testCase for the relevant pickle id, since test cases are made from only the filtered pickles. Does that make sense?

[davidh87]

To add to this, the report is a little confusing when first viewed if unfamiliar. We have a large set of feature files that we run in different groupings, so get reports like the attached. This is with cucumber-java.

Real behaviour: 1860 tests matched tags, and 100% of those tests passed.
Observed behaviour: 3711 tests 'executed', and 50% of tests passed.

It's quite unclear about what the report is showing, especially if you're not deeply familiar with it. Having the report include tests that didn't run because they didn't match tags is a useful thing, but it's misleading to say these tests executed and then include in the success percentages.

[gogonutsss]

Hello, everyone. To give more emphasis to the issue, I am comparing this html report being created locally and also the report created and published to cucumber.io.
To make it clearer, I have a feature file of 10 scenarios and just trigger to run 1 of them. The scenario passes. Then, on the reports part, the local html report claims a 10% passed rate while the published one 100%.

Definitely the information presented to the online report is what consumers of the report expect to see.
As discussed here, in general, it is really confusing that we get info back about all the not executed scenarios of the feature file(s) of the scenarios that were actually requested for execution. I assume this is by design, but again it seems even more confusing as atm the terms used to refer to the not executed is not consistent between the 2 reports, unknown vs ignored respectively.
In short, wouldn't it be better one narative approach to be selected and both reports to be normalized to it (up to design and template)?

[vamsi1821]

@davidjgoss, @gogonutsss - Thanks for the info on issue. We are facing the same issue where unknown filter was creating impact on pass % in the html report. Is there any workaround for this issue?

[davidjgoss]

@vamsi1821 no workaround that I'm aware of unfortunately, although we should be able to get the fix released pretty soon.

[davidjgoss]

A bit more to do on this, I’ll get to it soon.