Possible Incompatibility with SimpleCov 0.18+

[PragTob]

Hi there,

SimpleCov maintainer here.

To the best of my knowledge you're using resultset.json to do the code coverage reporting to code climate. Due to the introduction of branch coverage the format of this file will change slightly in the next release (beta release just released) - namely there are now top level "lines" and "branches" keys.

However, the file format might be changed further in the future due to problems with the standard branch format (we might even switch to YML format or something else). See simplecov-ruby/simplecov#801 for some information.

I don't want to break code climate test reporting, however:

a.) we need these changes to work correctly
b.) resultset.json was never intended or communicated as an integration point - it's merely the way in which we do merging results from different processes

As a result of this, I can't guarantee stability for resultset.json, furthermore as it pretty much just dumps and merges the standard coverage data it also doesn't support a lot of features of simplecov (off the top of my hat):

• ignoring coverage through nocov :nocov: is ignored with simplecov #246
• filtering out/ignoring files (all the different ways in which simplecov allows you to do this)
• tracking files/adding files that weren't loaded
• I'm not 100% sure if we write the file again after the last process finished writing, might be related to SimpleCov merged reports are not processed correctly #385

My suggested approach would be to write a gem/formatter that writes a file in a format that you control and can continue working. (I know this is against the general "no integration needed" idea of this project) I'm happy to work with you to ensure that the APIs you need are stable. I also still have hopes (:crossed_fingers:) to release a 1.0 version soon-ish (couple of months) and to make another pass through the code to make sure that public vs. private APIs in the code are accurately documented. Happy to provide you with more public APIs if you need them.

Cheers and happy new year,
Tobi

[efueger]

Thanks for the heads up @PragTob ! Pulling in a dev colleague to take a look.

[ale7714]

Hi @PragTob thank you for reaching out. We would really like to keep this project without requiring integration. Have you considered for simplecov to support a json output of the results? Other test coverage tools offer different formats for the outputs of the result. We would be happy on helping with writing that.

[PragTob]

Hi @ale7714 !

That's what I thought (although I think it's an integration either way) - you mean without adjusting the user simplecov code in any way, correct?

We have different output formats, supported through formatters (https://github.com/colszowka/simplecov/blob/master/doc/alternate-formatters.md, one of them is even a JSON formatter but it looks like it's unmaintained as of now: https://github.com/vicentllongo/simplecov-json ) :) Having a JSON output for simplecov is an idea, but if I understand your application correctly you'd rather that the user doesn't have to set anything - in other words you'd want the JSON report to be written on every run - correct?
Do you have an example of a JSON output from another coverage tool that you like?

I see how a JSON report could be useful, I just somewhat worry about the format of it (and then the complexity for you to parse it). Ruby now has 4 coverage criteria (lines, branch, method, oneshot lines) and the vision for SimpleCov is to allow all of them to be run individually or together. I guess a structure like:

{"file_name": {"coverage_method": data}}

could work. But one of the problems for your current way of integrating is that this would mean that you also need to be able to process all n types of coverage data we support.

Cheers,
Tobi

[ale7714]

Hi @PragTob,

in other words you'd want the JSON report to be written on every run - correct?
Yes that's correct.

XCov does something like this and it works out great. Handling the parsing on the test-reporter is fairly straightforward once we know how the template looks like. We have some tools that do lines and branch coverage and we're able to successfully parse the information we need once we get know how the format looks like.

I'll reiterate that we're happy to contribute to get the implementation of the JSON output up and running. Although I would defer to you on the template of the format. In general lines the information, currently provided in the resultset is enough for us but if we were able to add additional information link no-cov that would be awesome. Once we have a new template, on our side we will put together a new parser for it.

[PragTob]

Hi @ale7714,

it's nice that you offer help for the JSON formatter and I recognize that. I'd have to think up a suitable format and what to do, right now that's not my focus though.

The thing that irks me a bit here though is that in the end that we'd create and write a JSON report on every run while that's only necessary for the minority of runs that run on CI for a code climate customer. I made a quick check with 3 other online tools that provide code coverage and all of them integrate with us through a ruby gem (which imo is the right way to do it).
Feels odd right now that we'd have to do something to make the integration with Code Climate working not the other way around.

Does Code Climate still have an environment variable like "CODE_CLIMATE_TOKEN" or something that could be checked to only write the JSON report (unless configured in other ways) when that is set?

[ale7714]

Hi @PragTob,

Thank you for your response. I appreciate you’re taking the time to discuss this with us. I understand that other tools might do things differently. A couple of years ago, we took the direction of building a test coverage reporter tool that could parse output from different coverage tools with the main goal of having a consistent way to setup coverage reporting with Code Climate across different languages.

Before writing the tool, we performed some research and we were able to identify that most coverage tools offer a way to output a file (in JSON, XML, or similar) with the test coverage data that we would be able to parse.

Does Code Climate still have an environment variable like "CODE_CLIMATE_TOKEN" or something that could be checked to only write the JSON report (unless configured in other ways) when that is set?

Yes, users are setting CC_TEST_REPORTER_ID on their environment.

We would love to setup a time to chat with you to discuss how can we partner and support you on this. Please, let us know your availability over email at hello@codeclimate.com or a DM us in Twitter. Looking forward to hearing back from you.

[PragTob]

Thanks I'll get in touch although I thought communication here was mostly fine.

And forgot to update here, the version with the incompatible file format was released 2 days ago so this is now a current issue.

[maximeg]

Hi there,

Simplecov 0.18.0 is released and we could not use it as it leads to an error with cc-test-reporter:

#!/bin/bash -eo pipefail
./cc-test-reporter format-coverage -t simplecov -o coverage/codeclimate.json
./cc-test-reporter upload-coverage -i coverage/codeclimate.json

Error: json: cannot unmarshal object into Go struct field input.coverage of type []formatters.NullInt
Usage:
  cc-test-reporter format-coverage [coverage file] [flags]

Flags:
      --add-prefix string   add this prefix to file paths
  -t, --input-type string   type of input source to use [clover, cobertura, coverage.py, excoveralls, gcov, gocov, jacoco, lcov, simplecov, xccov]
  -o, --output string       output path (default "coverage/codeclimate.json")
  -p, --prefix string       the root directory where the coverage analysis was performed (default "/home/circleci/project")

Global Flags:
  -d, --debug   run in debug mode

Exited with code exit status 255

[efueger]

Hi @maximeg - Emily here with Code Climate Support. Thanks for the report! We're working on this now.

• for now, the workaround is to use SimpleCov 0.17.x

If you have any other questions in the meantime, feel free to reach out to hello@codeclimate.com and we'll be happy to help.