Skip to content

Improve coverage stability display #217

@Liam-DeVoe

Description

@Liam-DeVoe

Follow-ups from #5. See also #216 (comment).

  • Add a documentation page for "stability":

We should also report coverage stability fraction, including a rating stable (100%), unstable (85%--100%), or serious problem (<85%"); and explain the difference between stability (=coverage) and flakiness (=outcome). Stability is mostly an efficiency thing; flakiness means your test is broken.

  • Consider reporting the stability of the covering corpus, in addition to the rolling corpus. We'll need to consider the best way to display this once we have two numbers instead of just one. Simplest proposal: a single checkmark when both metrics are fully stable, and covering / rolling (eg 30% / 92% or check / 73%) otherwise. Discoverability of which is which via tooltips.
    • also unclear why/when we would keep an entry in the corpus once we know it's flaky.
  • Mark low stability numbers as bad, in some way. Color is the easiest way (yellow below 98?%, red below 85%), but I want to save colors for important information like failures and the test status, and avoid spread-out color in the table. We might try coloring the text anyway though; or try adding a new "unstable" pill next to the test status, making the status area into "test features you should be aware of", where test status is one but might be alongside others.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions