Tests are failing

[di]

          I think I've identified what's going on, although I'm not sure what the right next step is to correct it.

I'm using the 2.0.0rc1 of sigstore-python:

steiza/sigstore-conformance$ sigstore --version
sigstore 2.0.0rc1

I ran just the bundle verification (with signing skipped - so we don't need a valid identity-token) to get the full command that was failing:

steiza/sigstore-conformance$ pytest test/test_bundle.py --entrypoint /Users/steiza/code/steiza/sigstore-conformance/sigstore-python-conformance --identity-token asdf --skip-signing
...
E               subprocess.CalledProcessError: Command '['/Users/steiza/code/steiza/sigstore-conformance/sigstore-python-conformance', 'verify-bundle', '--bundle', PosixPath('a.txt.sigstore'), '--certificate-identity', 'https://github.com/sigstore-conformance/extremely-dangerous-public-oidc-beacon/.github/workflows/extremely-dangerous-oidc-beacon.yml@refs/heads/main', '--certificate-oidc-issuer', 'https://token.actions.githubusercontent.com', PosixPath('a.txt')]' returned non-zero exit status 1.

I ran just that command locally:

steiza/sigstore-conformance$ /Users/steiza/code/steiza/sigstore-conformance/sigstore-python-conformance verify-bundle --bundle test/assets/a.txt.sigstore --certificate-identity 'https://github.com/sigstore-conformance/extremely-dangerous-public-oidc-beacon/.github/workflows/extremely-dangerous-oidc-beacon.yml@refs/heads/main' --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' --offline test/assets/a.txt
An issue occurred while parsing the verification materials.

The provided verification materials are malformed and may have been
modified maliciously.

Additional context:

expected checkpoint in inclusion proof
...

It looks like that functionality landed in https://github.com/sigstore/sigstore-python/pull/634/files, which was between the sigstore-python 1.1.2 and 2.0.0rc1 releases.

... but again, I'm not sure what the right next step is. Where should the checkpoint information be coming from? Or maybe sigstore-python expecting something that really should be optional?

Originally posted by @steiza in #82 (comment)

[woodruffw]

I need to root-cause further, but the underlying error here is "correct" in the sense that checkpoints are required in the inclusion proof format, as specified in the bundle:

https://github.com/sigstore/protobuf-specs/blob/743cf10186e80b8e37b69fcdf952848fce4e4c6c/protos/sigstore_rekor.proto#L51-L68

Current SWAG is that these bundles are old, and don't include that checkpoint for whatever reason (which makes them technically invalid, although most clients probably didn't bother to check for it/enforce it if non-present).

[woodruffw]

@steiza How did you generate a.txt.sigstore? Was that from running sigstore sign locally and checking in the bundle?

[steiza]

@steiza How did you generate a.txt.sigstore? Was that from running sigstore sign locally and checking in the bundle?

That's correct, at the time I was running sigstore-python 1.1.2.

[woodruffw]

Gotcha, then this is our mess -- sorry for the confusion!

I think things should work if you re-generate that bundle using the current release candidate -- I'll also look into a more permanent resolution here, but that should unstick things on the PR itself.

[steiza]

I'll do that, but do note there are other non-bundle tests that failed when self-test was most recently run on main:

test/test_bundle.py ......                                               [ 42%]
test/test_certificate_verify.py .                                        [ 50%]
test/test_signature_verify.py ....F                                      [ 85%]
test/test_simple.py FF                                                   [100%]

[woodruffw]

Thanks -- it looks like something else is off about the ambient credential detection; I'll look into that as well.

[woodruffw]

sigstore/sigstore-python#705 should improve the bundle handling on the sigstore-python side: with that, we'll only perform the checkpoint check if the bundle version actually mandates it (i.e., is 0.2).

[woodruffw]

Some of the failures here are also potentially observed regressions from sigstore/sigstore-python@04f8fc1; the next RC I do will include that fix.

(This would also explain why the conformance suite is failing here, but not on sigstore-python's own repo.)

[woodruffw]

#89 should have all of the fixes that the conformance suite needs; it's blocked on the aforementioned sigstore-python RC, which in turn is waiting on sigstore/sigstore-python#705 🙂