-
Notifications
You must be signed in to change notification settings - Fork 5
Open
Labels
frontendui and/or uxui and/or ux
Description
Follow-ups from #5. See also #216 (comment).
- Add a documentation page for "stability":
We should also report coverage stability fraction, including a rating stable (100%), unstable (85%--100%), or serious problem (<85%"); and explain the difference between stability (=coverage) and flakiness (=outcome). Stability is mostly an efficiency thing; flakiness means your test is broken.
- Consider reporting the stability of the covering corpus, in addition to the rolling corpus. We'll need to consider the best way to display this once we have two numbers instead of just one. Simplest proposal: a single checkmark when both metrics are fully stable, and
covering / rolling(eg30% / 92%orcheck / 73%) otherwise. Discoverability of which is which via tooltips.- also unclear why/when we would keep an entry in the corpus once we know it's flaky.
- Mark low stability numbers as bad, in some way. Color is the easiest way (yellow below 98?%, red below 85%), but I want to save colors for important information like failures and the test status, and avoid spread-out color in the table. We might try coloring the text anyway though; or try adding a new "unstable" pill next to the test status, making the status area into "test features you should be aware of", where test status is one but might be alongside others.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
frontendui and/or uxui and/or ux