A way to get report data in detail

[werney]

The report method only returns the total coverage amount. However, sometimes we need this information in detail. For example, I need to know the function-by-function coverage of a file, and I can't do this using the report method. I know it's possible to do this using the html report method, but having the value inside a variable would make it much easier for a more detailed analysis of where the coverage is bad.

So if the report method returned a dictionary, for example, with the names of the functions and their coverage, it would be very useful.

[nedbat]

Take a look at the json report: it has all of the data in a machine-readable format.

[werney]

I used the json report and the results were the same as the conventional report. Just a summary of the file, it does not have a report by functions like html does. However, extracting this information from an HTML file is quite laborious.

[nedbat]

Ah sorry, you are right: the per-function and per-class information is not yet in the other reports. That won't be hard to add.

[werney]

Can you give me some guidance on how I can do this?

[nedbat]

You could look at the commit that added them to the HTML report. There's a lot of extra stuff happening in that commit, but that's the meat of it.

[devdanzin]

Also the commit that added regions helps to understand the approach: 68c5f93

If you're not planning to, I can try to add regions to other kinds of report.

[devdanzin]

I've implemented a simple way to output region stats in the basic report, but would like to get some input about the report design before going further. Basically, we can opt to print stats about files, classes and functions in a single report, or we can do it like the html version and report only on files, or classes, or functions in each report. What's the preferred way?

Here's what I have so far:

> coverage report --functions --classes
Name                         Stmts   Miss Branch BrPart  Cover
--------------------------------------------------------------
c.py                            11      3      0      0    73%
c.py: class A                    1      1      0      0     0%
c.py: class B                    1      1      0      0     0%
c.py: class B.Inner              1      1      0      0     0%
c.py: function A.test            1      1      0      0     0%
c.py: function B.Inner._in       1      1      0      0     0%
c.py: function B.test            1      1      0      0     0%
c.py: function c                 1      0      0      0   100%
d.py                             3      0      0      0   100%
--------------------------------------------------------------
TOTAL                           14      3      0      0    79%

[nedbat]

Thanks for checking in. My gut reaction is that it will be too much information for a plain text report, but I'd like to hear from others. I like the interleaved lines of files/classes/functions, it kind of provides a drill-down right in one place. Now I wonder if that's a better way to organize the HTML report.

If I had to choose a place to start, it would be the JSON report, because it's not space constrained, and it gives access to the data for other programmatic uses.

[devdanzin]

Thanks, I'll try to get it working for the JSON report first then. It should be simple to add functions and classes dictionaries to each reported_file and just add the data by class/function name. I hope to have a PoC working soon.

FWIW I like the current layout of the HTML report, maybe I'm biased because I stared at it for too long, but having a page for each kind of region plus one for files is simple and works for me. It might make sense to add yet another page with interleaved lines, for a quick glimpse of the coverage situation, but retaining the ability to look at only functions or files at a time.

We could also add a --no-files option to the text (and JSON) report, so you can get a report consisting only of functions, or classes, matching the focused views available in the HTML report.

[devdanzin]

Here's what a JSON report looks like with regions enabled by calling coverage json --functions --classes --pretty-print:

{
    "meta": {
        "format": 2,
        "version": "7.5.4a0.dev1",
        "timestamp": "2024-06-13T06:49:42.776728",
        "branch_coverage": true,
        "show_contexts": false
    },
    "files": {
        "c.py": {
            "executed_lines": [
                1,
                2,
                5,
                6,
                9,
                10,
                13,
                14
            ],
            "summary": {
                "covered_lines": 8,
                "num_statements": 11,
                "percent_covered": 72.72727272727273,
                "percent_covered_display": "73",
                "missing_lines": 3,
                "excluded_lines": 0,
                "num_branches": 0,
                "num_partial_branches": 0,
                "covered_branches": 0,
                "missing_branches": 0
            },
            "missing_lines": [
                3,
                7,
                11
            ],
            "excluded_lines": [],
            "executed_branches": [],
            "missing_branches": [],
            "class": {
                "A": {
                    "executed_lines": [],
                    "summary": {
                        "covered_lines": 0,
                        "num_statements": 1,
                        "percent_covered": 0.0,
                        "percent_covered_display": "0",
                        "missing_lines": 1,
                        "excluded_lines": 0,
                        "num_branches": 0,
                        "num_partial_branches": 0,
                        "covered_branches": 0,
                        "missing_branches": 0
                    },
                    "missing_lines": [
                        3
                    ],
                    "excluded_lines": [],
                    "executed_branches": [],
                    "missing_branches": []
                },
                "B": {
                    "executed_lines": [],
                    "summary": {
                        "covered_lines": 0,
                        "num_statements": 1,
                        "percent_covered": 0.0,
                        "percent_covered_display": "0",
                        "missing_lines": 1,
                        "excluded_lines": 0,
                        "num_branches": 0,
                        "num_partial_branches": 0,
                        "covered_branches": 0,
                        "missing_branches": 0
                    },
                    "missing_lines": [
                        7
                    ],
                    "excluded_lines": [],
                    "executed_branches": [],
                    "missing_branches": []
                },
                "B.Inner": {
                    "executed_lines": [],
                    "summary": {
                        "covered_lines": 0,
                        "num_statements": 1,
                        "percent_covered": 0.0,
                        "percent_covered_display": "0",
                        "missing_lines": 1,
                        "excluded_lines": 0,
                        "num_branches": 0,
                        "num_partial_branches": 0,
                        "covered_branches": 0,
                        "missing_branches": 0
                    },
                    "missing_lines": [
                        11
                    ],
                    "excluded_lines": [],
                    "executed_branches": [],
                    "missing_branches": []
                }
            },
            "function": {
                "A.test": {
                    "executed_lines": [],
                    "summary": {
                        "covered_lines": 0,
                        "num_statements": 1,
                        "percent_covered": 0.0,
                        "percent_covered_display": "0",
                        "missing_lines": 1,
                        "excluded_lines": 0,
                        "num_branches": 0,
                        "num_partial_branches": 0,
                        "covered_branches": 0,
                        "missing_branches": 0
                    },
                    "missing_lines": [
                        3
                    ],
                    "excluded_lines": [],
                    "executed_branches": [],
                    "missing_branches": []
                },
                "B.test": {
                    "executed_lines": [],
                    "summary": {
                        "covered_lines": 0,
                        "num_statements": 1,
                        "percent_covered": 0.0,
                        "percent_covered_display": "0",
                        "missing_lines": 1,
                        "excluded_lines": 0,
                        "num_branches": 0,
                        "num_partial_branches": 0,
                        "covered_branches": 0,
                        "missing_branches": 0
                    },
                    "missing_lines": [
                        7
                    ],
                    "excluded_lines": [],
                    "executed_branches": [],
                    "missing_branches": []
                },
                "B.Inner._in": {
                    "executed_lines": [],
                    "summary": {
                        "covered_lines": 0,
                        "num_statements": 1,
                        "percent_covered": 0.0,
                        "percent_covered_display": "0",
                        "missing_lines": 1,
                        "excluded_lines": 0,
                        "num_branches": 0,
                        "num_partial_branches": 0,
                        "covered_branches": 0,
                        "missing_branches": 0
                    },
                    "missing_lines": [
                        11
                    ],
                    "excluded_lines": [],
                    "executed_branches": [],
                    "missing_branches": []
                },
                "c": {
                    "executed_lines": [
                        14
                    ],
                    "summary": {
                        "covered_lines": 1,
                        "num_statements": 1,
                        "percent_covered": 100.0,
                        "percent_covered_display": "100",
                        "missing_lines": 0,
                        "excluded_lines": 0,
                        "num_branches": 0,
                        "num_partial_branches": 0,
                        "covered_branches": 0,
                        "missing_branches": 0
                    },
                    "missing_lines": [],
                    "excluded_lines": [],
                    "executed_branches": [],
                    "missing_branches": []
                }
            }
        },
        "d.py": {
            "executed_lines": [
                1,
                2,
                3
            ],
            "summary": {
                "covered_lines": 3,
                "num_statements": 3,
                "percent_covered": 100.0,
                "percent_covered_display": "100",
                "missing_lines": 0,
                "excluded_lines": 0,
                "num_branches": 0,
                "num_partial_branches": 0,
                "covered_branches": 0,
                "missing_branches": 0
            },
            "missing_lines": [],
            "excluded_lines": [],
            "executed_branches": [],
            "missing_branches": []
        }
    },
    "totals": {
        "covered_lines": 11,
        "num_statements": 14,
        "percent_covered": 78.57142857142857,
        "percent_covered_display": "79",
        "missing_lines": 3,
        "excluded_lines": 0,
        "num_branches": 0,
        "num_partial_branches": 0,
        "covered_branches": 0,
        "missing_branches": 0
    }
}

We might want to add something to meta to indicate what kind of region is included in the report, what do you think?

[nedbat]

I don't see a reason to add to meta, since you can see the region types based on the actual content. Though, we have "branch_coverage": true in the meta which I guess is technically redundant.

I'm wondering about the need for --functions and --classes. We don't have that for the HTML report, but maybe we should?

[devdanzin]

Maybe we don't need --functions and --classes for the JSON, HTML and XML reports, as they are able to include/present all the information on regions in a clear way. I think it'd be good to have those options (or a single --regions one) for the text report, so users can keep the current output format or choose to get a more detailed view.

I'm happy to implement it any way you prefer, and also to iterate on different designs. Should I create a couple of PRs with different designs so you can test them and maybe have others opine? Or is it better to wait and decide on a design before submitting a PR?