Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve unexpected success handling #108

Merged
merged 2 commits into from
Oct 13, 2023

Conversation

jku
Copy link
Member

@jku jku commented Oct 13, 2023

Summary / Release note

  • Improved handling for the "unexpected success" case: print details like the command arguments
  • command arguments are now shown in a more useful form in error messages

An unexpected success now looks like this:

_____________________________ test_verify_rejects_invalid_inclusion_proof ______________________________

client = <test.client.SigstoreClient object at 0x7f842a9066d0>
make_materials_by_type = <function make_materials_by_type.<locals>._make_materials_by_type at 0x7f842a90c0e0>

    def test_verify_rejects_invalid_inclusion_proof(
        client: SigstoreClient, make_materials_by_type: _MakeMaterialsByType
    ) -> None:
        """
        Check that the client rejects a bundle with an old inclusion proof
        """
    
        materials: BundleMaterials
        input_path, materials = make_materials_by_type("a.txt", BundleMaterials)
        materials.bundle = Path("a.txt.invalid_inclusion_proof.sigstore")
    
>       with client.raises():

/home/jkukkonen/src/sigstore-conformance/test/test_bundle.py:142: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3.11/contextlib.py:144: in __exit__
    next(self.gen)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test.client.SigstoreClient object at 0x7f842a9066d0>

    @contextmanager
    def raises(self):
        try:
            yield
        except ClientFail:
            pass
        else:
            assert self.completed_process
            msg = _CLIENT_ERROR_MSG.format(
                exitcode=self.completed_process.returncode,
                command=" ".join(map(str, self.completed_process.args)),
                stdout=self.completed_process.stdout,
                stderr=self.completed_process.stderr,
            )
>           raise ClientUnexpectedSuccess(msg)
E           test.client.ClientUnexpectedSuccess: 
E           Command: /home/jkukkonen/src/sigstore-conformance/sigstore-python-conformance verify-bundle --bundle a.txt.invalid_inclusion_proof.sigstore --certificate-identity https://github.com/sigstore-conformance/extremely-dangerous-public-oidc-beacon/.github/workflows/extremely-dangerous-oidc-beacon.yml@refs/heads/main --certificate-oidc-issuer https://token.actions.githubusercontent.com a.txt
E           Exit code: 0
E           
E           !!! STDOUT !!!
E           ==============
E           
E           OK: a.txt
E           
E           
E           !!! STDERR !!!
E           ==============
E           
E           None

/home/jkukkonen/src/sigstore-conformance/test/client.py:158: ClientUnexpectedSuccess

It would be even nicer if the custom raises() wasn't shown in the stack trace but I think this is an improvement:

  • I can now see what fails
  • I can pretty much copy paste the command to reproduce it outside the test suite (as long as I'm in the same directory with the assets)

I removed the "CLIENT FAILURE" line from output (the exception name should be very clear already: ClientFail or ClientUnexpectedSuccess).

Fixes #107

This allows managing the "unexpected success" case, and printing
the same details as in "unexpected failure" case.

Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
Try to print a usable shell command:
 * use str() so that we don't get "PosixPath()" in the args list
 * join the arguments into a single string

This means that I can now copy paste the command and reproduce the
problem by just being in the same directory with assets.

Signed-off-by: Jussi Kukkonen <jkukkonen@google.com>
Copy link
Member

@woodruffw woodruffw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks!

@woodruffw woodruffw merged commit 0f4e883 into sigstore:main Oct 13, 2023
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

unexpected success should lead to printing args
2 participants