Files showing reduced coverage with no reason

[elenajdanova]

Describe a bug

I was introducing minor text changes in my code, additionally updating the tests, to make them pass accordingly.
I didn't add any tests, I didn't delete any tests.
Output is giving me drastically reduced coverage for several files, one of them wasn't touched at all in mentioned PR. Can somebody explain a bit more on the algorithm comparing coverage and what's going on?

Expected behavior

• I would expect the coverage not to change if I do not remove or add tests and logic
• I would expect not to see untouched in PR files mentioned in reduced test coverage

Details

• OS, where your action is running: mac

  

  

[ArtiomTr]

Hello @elenajdanova 👋,

Jest provides information about file coverage. It could be an issue if coverage information is extracted incorrectly. To verify this hypothesis, you can run npx jest --coverage or yarn jest --coverage and see, if the numbers in the local report and Github comment differ.

Jest checks each line of code and calculates, how many times it was executed. With this information, you can easily calculate coverage information.

There are too many reasons, why your code coverage differs so dramatically. I can't tell for sure, without seeing the actual source code. However, I can guess why coverage of untouched file has changed. Consider this example:
File sum.ts:

export const sum = (a: number, b: number) => a + b;

File logic.ts:

import { sum } from './sum';

export const fancyLogic = (userInput: number) => {
    return sum(userInput, 5);
}

Imagine you have tests for the fancyLogic function. Because it calls function sum always, jest will assume that the sum function is also covered with tests, despite it doesn't have any dedicated tests. However, in some PR, you decide to change the logic.ts file:

import { subtract } from './subtract';

export const fancyLogic = (userInput: number) => {
    return subtract(userInput, 5);
}

Because the sum function doesn't have any dedicated tests, it becomes uncovered, as it is never being called. (and function subtract becomes covered). As a result, by changing only one line of code, you get this report (not a real report, just an example):

St.	File	Lines
🟢	logic.ts	100%
🔴	sum.ts	0% (-100% 🔻)
🟢	subtract.ts	100% (+100% 🔼 )

And that's not bad - if you have a small amount of code/tests, you will see these reports often. However, as the codebase will grow, these reports will be more stable and meaningful 😄

Hope my explanation helped you. I will close the issue, as this behaviour doesn't look like a bug. However, feel free to ping me or reopen the issue.

[cristianrgreco]

@ArtiomTr thanks for creating this action!

I've come across this issue because I had a similar observation. There is another unexpected behaviour, that if someone adds a large, untested file, the coverage report won't change. This is because Jest by default will not pick it up unless it goes through the tests.

You can however configure Jest to collect coverage from wherever you'd like, the files don't have to be tested, e.g:

  collectCoverageFrom: [
    'src/**/*.ts'
  ]

And the result is what you'd expect. The original issue is addressed, and so is the issue of people committing large untested files, only to find out they're untested later once someone adds the first test.

I'd suggest making this collectCoverageFrom a default behaviour, or maybe documenting it. Let me know if I've misunderstood something or if I can help, thanks again!

If you did make this change, I'd just suggest removing the word 'covered' here:

[ArtiomTr]

Hello @cristianrgreco 👋,

The main goal of this action is to report coverage, collected by jest. I try to avoid any modifications to the original jest behaviour, so the report looks exactly the same, as you see in your console locally.

Another reason collectCoverageFrom would not be a default behaviour is that there is no way to determine which files are project sources confidentially. For instance, some people save their source code into src/**/*.ts and have scripts/**/*.ts files, which don't need any tests. Some people use this action in monorepos, so collectCoverageFrom must have the value packages/*/src/**/*.ts.

So yeah, I think the best way to address this issue would be to document this pitfall in the documentation.