-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
action, conftest: initial xfail support #95
Conversation
Signed-off-by: William Woodruff <william@trailofbits.com>
Signed-off-by: William Woodruff <william@trailofbits.com>
Signed-off-by: William Woodruff <william@trailofbits.com>
Signed-off-by: William Woodruff <william@trailofbits.com>
Example of strict xfail failing (since the tests actually pass):
|
This reverts commit 25ffed8.
CC @steiza: I didn't change anything with this PR, but I think we'll eventually want to replace the |
Yeah, on one hand we don't want to have to add a set of flags for every possible subset of functionality a client library supports, and supporting On the other hand, when a client library updates to a new release of conformance testing, they'll have to manually go through any new tests that are failing, to determine if the failure indicates a bug or is using a feature their library doesn't support. Hopefully they are careful in their analysis!
After thinking through it, I'm okay with replacing |
Signed-off-by: William Woodruff <william@trailofbits.com>
Yeah, I think rewriting it to use |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM for use with sigstore-rs.
I'll do a follow-up with |
This adds an
xfail
input that configures a whitespace-separated list of test names to mark asxfail
. Thesexfail
tests are "strict", meaning that an unexpected pass with them causes a test failure, rather than a silent pass.Some thoughts:
fnmatch
syntax here, e.g.test_signature_verify::*
?Closes #87.
Closes #94.