feat:checking that the client is allowed to run against staging

[javanlacerda]

Closes #121

Summary

One conformance requirement for the clients is that they should be able to run the commands against staging. After this PR, the clients should be able to receive the --staging token in the command line and point to staging resources.

Release Note

Updated client run function to receive staging argument.
Updated run subprocess call to pass --staging flag to clients cli when required.
Modified CLI parser to receive --staging flag
Update CLI and conformance READMEs adding staging features

Documentation

[woodruffw]

LGTM overall, but with one broader design thought: rather than enabling individual tests to be run against staging, maybe we should allow the entire suite to be flipped between production and staging at a time?

My rationale: at least for testcases that do end-to-end signing (meaning they aren't implicitly tied to a specific instance already), we expect them to work on both instances and probably don't want to duplicate the code for both. If the entire suite supports being run in either mode (with skips where necessary for tests that are hard-baked against one or the other), then we get additional coverage against both instances for free.

Thoughts? This may be more effort than is immediately worth 🙂

[jku]

rather than enabling individual tests to be run against staging, maybe we should allow the entire suite to be flipped between production and staging at a time?

We can try that... but it may make sense to still enable just a subset of tests (the end-to-end ones) in the beginning: otherwise it might be quite a bit of work to re-create all of the test assets for staging (I assume that will be needed at least: I haven't created any of the assets)

If all (or many) tests can be run against staging, then we need to provide clients a way to enable/disable staging testing: xfail for individual tests is not viable... Some potential solutions:

• we add some sort of suite wide xfail value for all-staging-tests or
• we add a staging option to the GH action: this way clients can add a whole new action call to to their workflow when they are ready to test staging

I suppose the second one makes more sense?

[jku]

Waving hands about what tests should be executed by pytest:

• pytest --skip-signing already uses @pytest.mark.signing to skip tests: we could do something similar: if pytest gets "--staging", it could skip tests with @pytest.mark.no_staging_assets
• for practical purposes it might be easier to start tagging the tests that will work with staging though: I think most tests do use the (production) assets
• when/if we actually end up adding some staging assets, we need to figure out how to load the correct ones: I'm sure we can modify the make_materials*() functions at that time

[jku]

Some of results that currently PASS when running GHA_SIGSTORE_CONFORMANCE_XFAIL="test_verify_with_trust_root test_verify_dsse_bundle_with_trust_root" pytest test --staging --skip-signing --entrypoint=$PWD/sigstore-python-conformance are likely incorrect: if the suite uses production assets against staging infra, the client-under test is going to fail as the test expects (but it's failing for the wrong reasons) .

[woodruffw]

Yeah, getting the production/staging states right will probably require a larger refactor/discussion of how we want the CI to run the suite. I'm okay with punting on that for now 🙂

[woodruffw]

LGTM, thank you @javanlacerda!

[woodruffw]

(CCing @jku for review as well, since I'm now the last pusher.)

[di]

Ah, sorry @jku I merged too quickly here, please feel free to finish your review and suggest any changes if we missed something!

[haydentherapper]

Should we expect a significant increase in traffic to staging?

[woodruffw]

Should we expect a significant increase in traffic to staging?

I don't think so -- based on current integrations, this should only run a handful of times a day on the sigstore-python, Java, etc. repos. So the traffic should be roughly the same as the existing staging tests, e.g. the ones sigstore-python does in its own CI 🙂