v1.10.0 #255

webknjaz · 2024-09-01T01:17:03Z

webknjaz
Sep 1, 2024
Maintainer

🔏 Anything fancy, eh?

This time, @woodruffw 💰 implemented support for PEP 740 attestations functionality in #236 and #245. This is a big deal, as it is a huge step forward to replacing what the deprecated GPG signatures used to provide in a more meaningful way.

Important

✨ Please, do opt into trying this feature out early. It can be enabled as follows:

  with:
    attestations: true

Leave any feedback on this in this release discussion or the PR.

🙏 And please, thank William for working on this amazing improvement for the ecosystem! The overall effort is tracked @ pypi/warehouse#15871, by the way.

🪞 Full Diff: v1.9.0...v1.10.0

🧔‍♂️ Release Manager: @webknjaz 🇺🇦

This discussion was created from the release v1.10.0.

mnbf9rca · 2024-09-11T14:46:51Z

mnbf9rca
Sep 11, 2024

I turned on attestations and i'm getting this error:

Uploading distributions to https://upload.pypi.org/legacy/
INFO     /github/workspace/dist/pydantic_tfl_api-1.1.1-py3-none-any.whl (92.3   
         KB)                                                                    
INFO     /github/workspace/dist/pydantic_tfl_api-1.1.1.tar.gz (42.9 KB)         
INFO     password set by command options                                        
INFO     username: __token__                                                    
INFO     password: <hidden>                                                     
Uploading pydantic_tfl_api-1.1.1-py3-none-any.whl
INFO     Response from https://upload.pypi.org/legacy/:                         
         400 Could not verify the uploaded artifact using the included          
         attestation: Verification failed: 0 of 2 policies succeeded            
INFO     <html>                                                                 
          <head>                                                                
           <title>400 Could not verify the uploaded artifact using the included 
         attestation: Verification failed: 0 of 2 policies succeeded</title>    
          </head>                                                               
          <body>                                                                
           <h1>400 Could not verify the uploaded artifact using the included    
         attestation: Verification failed: 0 of 2 policies succeeded</h1>       
           The server could not comply with the request since it is either      
         malformed or otherwise incorrect.<br/><br/>                            
         Could not verify the uploaded artifact using the included attestation: 
         Verification failed: 0 of 2 policies succeeded                         
                                                                                
                                                                                
          </body>                                                               
         </html>                                                                
ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/        
         Could not verify the uploaded artifact using the included attestation: 
         Verification failed: 0 of 2 policies succeeded

happy to share more if that'd help. Here's the workflow - it works fine without attestations, and here's the specific job.

11 replies

mnbf9rca Sep 12, 2024

Thanks for looking in to it.

At the moment i can’t see any way to discover the claims in the token being sent to PyPi. If this was visible - perhaps in an event log - project owners could create a new trusted publisher which matches it.

woodruffw Sep 12, 2024

At the moment i can’t see any way to discover the claims in the token being sent to PyPi. If this was visible - perhaps in an event log - project owners could create a new trusted publisher which matches it.

This action currently only logs them when an error during TP exchange happens, which didn't happen in this case (since the error happened deeper in, with attestations).

I can look into improving the debuggability there, but in the mean time you could probably use https://github.com/github/actions-oidc-debugger to safely log the claims that your workflow is sending.

mnbf9rca Sep 12, 2024

Ok i installed that. Here's the publisher config on test.pypi.org:
Repository: mnbf9rca/pydantic_tfl_api
Workflow: deploy_to_pypi.yml
Environment name: pypi-test

And here's the token - the subject and audience seem ok (although you have to set that when you use https://github.com/github/actions-oidc-debugger i checked and it's the same as the github docs as far as i can tell):

 {
    "actor": "mnbf9rca",
    "actor_id": "1039058",
    "aud": "https://github.com/mnbf9rca",
    "base_ref": "",
    "environment": "pypi-test",
    "environment_node_id": "EN_kwDOMT68ds7J28it",
    "event_name": "workflow_dispatch",
    "exp": 1726169973,
    "head_ref": "",
    "iat": 1726169673,
    "iss": "https://token.actions.githubusercontent.com",
    "job_workflow_ref": "mnbf9rca/pydantic_tfl_api/.github/workflows/deploy_to_pypi.yml@refs/heads/chore/log_oidc_claims",
    "job_workflow_sha": "2c2c5829fcfb6bf63e4c66bb512cde4e81efdc46",
    "jti": "3e4ae702-d0c3-45a5-bf52-9488e709b547",
    "nbf": 1726169073,
    "ref": "refs/heads/chore/log_oidc_claims",
    "ref_protected": "false",
    "ref_type": "branch",
    "repository": "mnbf9rca/pydantic_tfl_api",
    "repository_id": "826195062",
    "repository_owner": "mnbf9rca",
    "repository_owner_id": "1039058",
    "repository_visibility": "public",
    "run_attempt": "1",
    "run_id": "10837523245",
    "run_number": "58",
    "runner_environment": "github-hosted",
    "sha": "2c2c5829fcfb6bf63e4c66bb512cde4e81efdc46",
    "sub": "repo:mnbf9rca/pydantic_tfl_api:environment:pypi-test",
    "workflow": "build artifact and publish to PyPi",
    "workflow_ref": "mnbf9rca/pydantic_tfl_api/.github/workflows/deploy_workflow_wrapper.yml@refs/heads/chore/log_oidc_claims",
    "workflow_sha": "2c2c5829fcfb6bf63e4c66bb512cde4e81efdc46"
  }

but the deployment failed with the same error.

woodruffw Sep 12, 2024

Thanks for pulling those claims. Here's the relevant policy construction section:

https://github.com/pypi/warehouse/blob/6b5f6bf005419be2260bafc6523d630eff8ec86f/warehouse/oidc/models/github.py#L280-L311

...and based on your workflow's output, here's an example of the claims as they appear in the attestation:

https://search.sigstore.dev/?logIndex=129964792

Based on the above, I'm pretty confident this is a limitation on reusable workflows that we can't yet lift -- our policy is to check the OIDCBuildConfigURI, which corresponds to the workflow's workflow_ref rather than job_workflow_ref. We could in theory switch to OIDCBuildSignerURI (which would match the reusable workflow filename), but we can't immediately to do that until we fix reusable workflow handling more generally within PyPI (pypi/warehouse#11096).

TL;DR: For the time being, I'd recommend either leaving attestations disabled or switching to a non-reusable workflow, like the docs suggest. The attestations feature will eventually work with reusable workflows, but not until we do a larger refactor to make them work fully correctly as Trusted Publisher sources first.

mnbf9rca Sep 12, 2024

thanks for this analysis. I've read through pypi/warehouse/issues/#11096 - very interesting. The reason i implemented a reusable workflow (with environments) is indeed least privilege.

i'll disable the feature until i see that it's now fixed.

steiza · 2024-09-11T14:55:01Z

steiza
Sep 11, 2024

The attestations feature worked for me! https://github.com/steiza/simplepypi/actions/runs/10813914412/job/29999063162

I see that the publish attestation is logged in the Action workflow logs and sent to Sigstore Rekor transparency log - that's great. Is the attestation currently being stored by Warehouse? Is there a CLI command I could run to see / verify the publish attestation, or an API where I could download it?

2 replies

woodruffw Sep 11, 2024

Is the attestation currently being stored by Warehouse? Is there a CLI command I could run to see / verify the publish attestation, or an API where I could download it?

Not yet -- we initially enabled persistence, but ran into a handful of severe performance issues around some of the original implementation's assumptions. We're currently working on re-enabling attestation storage however and should have it merged relatively soon (well before we flip the default on the action itself) 🙂

gaborbernat Sep 13, 2024

Also worked for https://github.com/tox-dev/tox-ini-fmt/actions/runs/10853423422/job/30121650173

gaborbernat · 2024-09-17T17:56:50Z

gaborbernat
Sep 17, 2024

@webknjaz @woodruffw does attestation work with a split build and publish env? Something like https://github.com/tox-dev/tox/blob/main/.github/workflows/release.yml#L10-L46. Or are we loosing build env info should we split it as such? 🤔

4 replies

webknjaz Sep 17, 2024
Maintainer Author

We were debating these and I maintain that it's irresponsible to give the transitive build deps access to OIDC. So we're talking about the publish attestations as opposed to the build ones. Strictly speaking, none of the build info or the env is signed, just the dists. You may still want to stick SLSA into your workflows.

gaborbernat Sep 17, 2024

SLSA?

webknjaz Sep 17, 2024
Maintainer Author

Our attestations are just one bit of the provenance metadata you might want to have. Here's an example where I integrated several things: https://github.com/ansible/awx-plugins/blob/01b2c31/.github/workflows/ci-cd.yml#L967-L1143.

webknjaz Sep 17, 2024
Maintainer Author

SLSA?

https://slsa.dev

Uh oh!

v1.10.0 #255

Uh oh!

Uh oh!

webknjaz Sep 1, 2024 Maintainer

🔏 Anything fancy, eh?

Replies: 3 comments · 17 replies

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

webknjaz Sep 17, 2024 Maintainer Author

Uh oh!

Uh oh!

Uh oh!

webknjaz Sep 17, 2024 Maintainer Author

Uh oh!

webknjaz Sep 17, 2024 Maintainer Author

webknjaz
Sep 1, 2024
Maintainer

Replies: 3 comments 17 replies

webknjaz Sep 17, 2024
Maintainer Author

webknjaz Sep 17, 2024
Maintainer Author

webknjaz Sep 17, 2024
Maintainer Author