Skip to content

Consider doing validation of data accuracy and caveat until then... #468

Open
@JustinCappos

Description

@JustinCappos

At a glance, it seems like many of the insights are quite inaccurate (see #467, for instance). I'd suggest that the metrics be checked against human verified analyses of projects to see if they seem to be correlated and meaningful. For downstream tools like baseline, it looks there is an LF effort to get people to believe that the provided score has meaning. If so, the team providing it has a responsibility to make that score as accurate as is practical and to be forthright about any shortcomings whenever discussing it.

So, perhaps doing and publishing a study where you look at the accuracy using some outside, verified metrics for projects would help to validate your scoring.

In the meantime, I'd recommend adding caveats to insights and whatever downstream things are using it that discuss the limitations of your analysis.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingneeds-triageA new issue that hasn't been reviewed yet

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions