New html formatter produces very large report files

[ben-nc2]

The new html formatter produces very large report files. Please consider reducing the size or providing the old html formatter as an option to switch back to for 6.x.x

We have been using cucumber 4 for a number of years and recently made the changes to first upgrade to 5.7.0 and then on to 6.8.1. Our tests are running fine and although the new html formatter is much easier to read and use for debugging of test failures for our build (which currently runs 423 scenarios that invoke selenium for ui testing).

Our 423 scenarios produce an html report file of 314MB which when we are storing this test artifact uses up a lot of space fast.

[ben-nc2]

Currently this is a blocker to use moving to 6.x as we need the test reports with screenshots for helping debug build failures but don't want to have to handle the large report size. Any suggestions for a workaround also welcome.

[aslakhellesoy]

Hi @ben-nc2 - we're aware that the large size can be a problem.

How big were your HTML reports prior to upgrading to v6?

Screenshot sizes should be identical between v5 and v6, so the increase in size might come from elsewhere. I think it might have to do with the large number of duplicated stepDefinition messages on Cucumber-JVM, as well as the inclusion of more JavaScript.

Some things we can try to minimize the size:

More space-efficient message format

Some options:

• Use binary representation and use a more compact protobuf library
• Keep using NDJSON, but don't include messages.js + the protobuf library in the bundle
• Use msgpack

More space efficient JavaScript

• Tree shaking
• Minification

Fewer messages

• Cucumber-JVM outputs all the step definition messages once for every single scenario. It only needs to be outputted once.

[ben-nc2]

@aslakhellesoy running with 5.7.0 html reports were 7MB in size where all tests have passed. The same tests running with 6.8.1 create html reports of 310MB in size. If it would be helpful to share both I can check with my employers if the information is sensitive or not.

[aslakhellesoy]

Hi @ben-nc2 having some example reports would be very helpful! If you're allowed to share, you can email them to aslak hellesoy smartbear com (add dots and @)

[ben-nc2]

I hope the example reports I emailed help you find ways to reduce the overall size of the report files.

I'm very willing to test any alpha/beta releases against the same scenarios to provide a direct comparison if that is at all helpful.

[mpkorstanje]

I can think of a few options:

a. Fix Cucumber JVM to only emit step definition once.
b. Externalize attachments when creating the html report
c. Use a different encoding then Base64 to reduce the overhead and make zipping more efficient
d. Filter out messages that are not used by the html report.

None of these are ideal.

a. Is a significant amount of work.
b. Removes the SPA nature.
c. Assumes the attachments are the problem

However after running a suite of ~550 scenarios without any screenshots:

$ ls -l --block-size=M cucumber.ndjson
142M nov  5 22:48 cucumber.ndjson
$ cat cucumber.ndjson | cut -d \" -f 2 | sort | uniq -c 
   8934 attachment
    112 gherkinDocument
  10584 hook
      1 meta
    469 pickle
    112 source
 380583 stepDefinition
    441 testCase
    441 testCaseFinished
    441 testCaseStarted
      1 testRunFinished
      1 testRunStarted
  11399 testStepFinished
  11399 testStepStarted
$ sed '/${"stepDefinition/d' cucumber.ndjson > stripped.ndjson
$ ls -l --block-size=M stripped.ndjson
-rw-rw-r-- 1 mpkorstanje mpkorstanje  15M nov  5 22:31 stripped.ndjson

So originally the file was 142M and after removing stepDefinition messages only 15M remains. So that removes about 90%. So the attachments are not the problem.

While d. requires that the MessagesToHtmlWriter knows the implementation details of cucumber/react it would be a pretty significant optimization overall at this point in time. If the html report does use the step definitions in the future then we have the same problem back though.

@aslakhellesoy for the short term could do something like cucumber/common#1236. And it would be even better if react could output a list of messages it consumes because then the filter could be generated in all implementations.

[mpkorstanje]

@ben-nc2 if you don't mind building snapshots you could add the following to the HtmlFormatter in cucumber-jvm.

    private void write(Envelope event) {
        if (event.hasStepDefinition() || event.hasHook()) {
            return;
        }