Invalid code coverage report using sfdx force:source:deploy and force:mdapi:deploy

[bzacharski]

Summary

Tested on version 7.155.0 (salesforce/salesforcedx:7.155.0-full), 154 and 153
I wanted to use code coverage from deploy commands to feed sonarqube (json format) and gitlab (cobertura format) with code coverage information, but it is failing. I've analyzed reports from deploy commands and compared them to reports from force:apex:test:run command.

1. For json format it is different between deploy and test:run commands
2. It looks like reports from deploy commands are invalid. For example I have a class x on destination org and by running it using test:run command I received report that class has 13 total lines and 11 is covered. Which is correct, but both json and cobertura from deploy commands have entries for 15 lines of which 2 are not covered (0). I've checked for sure that I am deploying the same class. It is reporting lines that are empty, have only comments, etc. And it looks that data from deploy commands are invalid for all files. Class form example (I replaced x, y, z, v, w, s, t):

/**************************************************************************************
 *  @Class            : x
 *  @Author           : y
 *  @Version History  : 1.0
 *  @Date             : 
 *  @Description      : Handler class for  z.
****************************************************************************************/
public with sharing class x extends TriggerHandler {
    
    private static final string RE_OBJ_API = 'v';
    
    /* Class constructor which invokes parent constructor using standard super function */
    public x(){
        super(RE_OBJ_API );     
    } 
    /* To get the API name of the executing trigger context */
    public override String getName(){
        return RE_OBJ_API ;
    }
    public override  void beforeInsert(List<SObject> newItems){}
    
    /* To invoke before update trigger logic */
    public override void beforeUpdate(List<SObject> newItems, Map<Id, SObject> newItemsMap, Map<Id, SObject> oldItemsMap) {}
    
    /* To invoke before delete trigger logic */
    public override void beforeDelete(Map<Id, SObject> oldItemsMap) {
     w.s(oldItemsMap);
    }
    
    /* To invoke after update trigger logic */
    public override void afterUpdate(List<SObject> newItems, Map<Id, SObject> newItemsMap, Map<Id, SObject> oldItemsMap) {}
    
    /* To invoke after delete trigger logic */
    public override void afterDelete(Map<Id, SObject> oldItemsMap) {
        w.t(oldItemsMap);
    }
    
    /* To invoke after undelete trigger logic */
    public  override void afterUndelete(Map<Id, SObject> oldItemsMap) {}
    
    /* To invoke after insert trigger logic */
    public override void afterInsert(List<SObject> newItems, Map<Id, SObject> newItemsMap) {}

}

test:run entry for given class:

{
    "id": "01p3N000009V0efQAC",
    "name": "x",
    "totalLines": 13,
    "lines": {
      "13": 1,
      "14": 1,
      "17": 0,
      "18": 0,
      "20": 1,
      "23": 1,
      "26": 1,
      "27": 1,
      "31": 1,
      "34": 1,
      "35": 1,
      "39": 1,
      "42": 1
    },
    "totalCovered": 11,
    "coveredPercent": 85
  }

deploy json entry for class:

"src/x": {
        "fnMap": {},
        "branchMap": {},
        "path": "src/x",
        "f": {},
        "b": {},
        "s": {
            "17": 0,
            "18": 0,
            "19": 1,
            "20": 1,
            "21": 1,
            "22": 1,
            "23": 1,
            "24": 1,
            "25": 1,
            "26": 1,
            "27": 1,
            "28": 1,
            "29": 1,
            "30": 1,
            "31": 1
        },
        "statementMap": {
            "17": {
                "start": {
                    "line": 17,
                    "column": 0
                },
                "end": {
                    "line": 17,
                    "column": 0
                }
            },
            "18": {
                "start": {
                    "line": 18,
                    "column": 0
                },
                "end": {
                    "line": 18,
                    "column": 0
                }
            },
            "19": {
                "start": {
                    "line": 19,
                    "column": 0
                },
                "end": {
                    "line": 19,
                    "column": 0
                }
            },
            "20": {
                "start": {
                    "line": 20,
                    "column": 0
                },
                "end": {
                    "line": 20,
                    "column": 0
                }
            },
            "21": {
                "start": {
                    "line": 21,
                    "column": 0
                },
                "end": {
                    "line": 21,
                    "column": 0
                }
            },
            "22": {
                "start": {
                    "line": 22,
                    "column": 0
                },
                "end": {
                    "line": 22,
                    "column": 0
                }
            },
            "23": {
                "start": {
                    "line": 23,
                    "column": 0
                },
                "end": {
                    "line": 23,
                    "column": 0
                }
            },
            "24": {
                "start": {
                    "line": 24,
                    "column": 0
                },
                "end": {
                    "line": 24,
                    "column": 0
                }
            },
            "25": {
                "start": {
                    "line": 25,
                    "column": 0
                },
                "end": {
                    "line": 25,
                    "column": 0
                }
            },
            "26": {
                "start": {
                    "line": 26,
                    "column": 0
                },
                "end": {
                    "line": 26,
                    "column": 0
                }
            },
            "27": {
                "start": {
                    "line": 27,
                    "column": 0
                },
                "end": {
                    "line": 27,
                    "column": 0
                }
            },
            "28": {
                "start": {
                    "line": 28,
                    "column": 0
                },
                "end": {
                    "line": 28,
                    "column": 0
                }
            },
            "29": {
                "start": {
                    "line": 29,
                    "column": 0
                },
                "end": {
                    "line": 29,
                    "column": 0
                }
            },
            "30": {
                "start": {
                    "line": 30,
                    "column": 0
                },
                "end": {
                    "line": 30,
                    "column": 0
                }
            },
            "31": {
                "start": {
                    "line": 31,
                    "column": 0
                },
                "end": {
                    "line": 31,
                    "column": 0
                }
            }
        }
    }

deploy cobertura entry for class:

<class name="x" filename="src/x" line-rate="0.8665999999999999" branch-rate="1">
          <methods>
          </methods>
          <lines>
            <line number="17" hits="0" branch="false"/>
            <line number="18" hits="0" branch="false"/>
            <line number="19" hits="1" branch="false"/>
            <line number="20" hits="1" branch="false"/>
            <line number="21" hits="1" branch="false"/>
            <line number="22" hits="1" branch="false"/>
            <line number="23" hits="1" branch="false"/>
            <line number="24" hits="1" branch="false"/>
            <line number="25" hits="1" branch="false"/>
            <line number="26" hits="1" branch="false"/>
            <line number="27" hits="1" branch="false"/>
            <line number="28" hits="1" branch="false"/>
            <line number="29" hits="1" branch="false"/>
            <line number="30" hits="1" branch="false"/>
            <line number="31" hits="1" branch="false"/>
          </lines>
        </class>

Steps To Reproduce:

Create class similar to example, deploy to org, run test:run and gather json coverage report, deploy the same class using deploy command (mdapi or source) and gather json and/or cobertura coverage reports

Expected result

Deploy commands produce valid reports. Json reports for tesr:run and deploy commands have similar/identical structure.

Actual result

Reports from deploy commands are invalid. Json reports have different structure.

System Information

{
	"cliVersion": "sfdx-cli/7.155.0",
	"architecture": "linux-x[64](https://code.roche.com/dws/DWS/-/jobs/9233716#L64)",
	"nodeVersion": "node-v16.13.2",
	"pluginVersions": [
		"@oclif/plugin-autocomplete 0.3.0 (core)",
		"@oclif/plugin-commands 1.3.0 (core)",
		"@oclif/plugin-help 3.3.1 (core)",
		"@oclif/plugin-not-found 1.2.6 (core)",
		"@oclif/plugin-plugins 1.10.11 (core)",
		"@oclif/plugin-update 1.5.0 (core)",
		"@oclif/plugin-warn-if-update-available 1.7.3 (core)",
		"@oclif/plugin-which 1.0.4 (core)",
		"@salesforce/sfdx-plugin-lwc-test 0.1.7 (core)",
		"alias 2.0.1 (core)",
		"apex 0.13.0 (core)",
		"auth 2.1.0 (core)",
		"community 2.0.0 (core)",
		"config 1.4.12 (core)",
		"custom-metadata 2.0.0 (core)",
		"data 2.0.3 (core)",
		"generator 2.0.1 (core)",
		"info 2.0.1 (core)",
		"limits 2.0.1 (core)",
		"org 1.13.2 (core)",
		"salesforce-alm 54.5.0 (core)",
		"schema 2.1.1 (core)",
		"sfdx-cli 7.155.0 (core)",
		"signups 1.1.2 (core)",
		"source 1.10.2 (core)",
		"telemetry 2.0.0 (core)",
		"templates 54.8.0 (core)",
		"trust 2.0.1 (core)",
		"user 2.0.2 (core)"
	],
	"osVersion": "Linux 5.4.0-109-generic"
}

Additional information

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[bzacharski]

Test level was always RunLocalTests

[WillieRuemmele]

Hi @bzacharski could you please provide the exact commands and arguments you running?

[bzacharski]

@WillieRuemmele
for deployment it will be for example:
sfdx force:source:deploy --json --loglevel ERROR --targetusername myAlias --checkonly --wait 1440 --testlevel RunLocalTests --verbose --sourcepath source --resultsdir tests/apex/ --coverageformatters cobertura,json --junit
for apex test:
sfdx force:apex:test:run -c -r json,cobertura -l RunLocalTests -w 60 -d ./testoutput -u myAlias

[bzacharski]

I've just checked it on 7.156.1 and the issue still exist. @WillieRuemmele were you able to check / reproduce it?

[mshanemc]

I dug into this with the person who added all the fancy new coverage stuff to deploy commands and I'm attempting to summarize:

1. deploy and test actually do slightly different things when they hit the server, and report back different information, which is why they won't agree
2. deploy results give us back only the uncovered lines and the number of lines...so we are 100% cheating and saying "that which is not uncovered must be covered" which, like you saw is different from apex:test:run results with respect to how blank lines and comments are treated.
3. We have a longer term plan to do this the right way (changing how the server-side stuff works) but those types of changes

• have to be done by somebody else
• have to be done as part of the major release cycle
• are pretty complicated

I know it sucks, but I'm going to close this and call it "working as designed" with full sympathy that the design isn't great.

[nvuillam]

Meanwhile, here is a workaround for those who need to check code coverage while calling force:source:deploy :)

https://nicolas.vuillamy.fr/test-apex-tests-org-wide-coverage-while-simulating-salesforce-deployments-bb738130480f

[mcarvin8]

I dug into this with the person who added all the fancy new coverage stuff to deploy commands and I'm attempting to summarize:

1. deploy and test actually do slightly different things when they hit the server, and report back different information, which is why they won't agree
2. deploy results give us back only the uncovered lines and the number of lines...so we are 100% cheating and saying "that which is not uncovered must be covered" which, like you saw is different from apex:test:run results with respect to how blank lines and comments are treated.
3. We have a longer term plan to do this the right way (changing how the server-side stuff works) but those types of changes

• have to be done by somebody else
• have to be done as part of the major release cycle
• are pretty complicated

I know it sucks, but I'm going to close this and call it "working as designed" with full sympathy that the design isn't great.

@mshanemc - Is there any updates to the above in regards to fixing how the deploy commands (sf project deploy at this point) report back code coverage, specifically "covered" lines? @xL3o and I have exhausted workarounds with "covered" lines and fed that to SonarQube for coverage. The "uncovered" lines are correctly reported by the deploy command as stated previously and I'm currently using that alone to feed SonarQube via my plugin. However, SonarQube requires "covered" and "uncovered" lines to calculate Coverage %. So my plugin can correctly set "uncovered" lines in SonarQube but cannot set "covered" lines to calculate coverage %.

My plugin just converts the sf project deploy code coverage JSON output into an XML that SonarQube will accept. We have figured out the CLI's JSON output will print extra lines as "covered" (ex: line 100 as "covered" if the apex class is only 95 lines of code) and lines. So there's no reliable way to get the "covered" line count accurate for the code coverage %, even after I basically mapped out-of-range "covered" lines to unused lines in-range.

Basically, I can't update my plugin to report "covered" lines until the Salesforce CLI is updated to correctly return "covered" lines in the coverage JSON output.

Ref: mcarvin8/apex-code-coverage-transformer#2

[mshanemc]

Definitely read the VSCode issue. WI is W-16286380 when you open those magical Support Cases that will get the Apex team's attention !