Publishing to BCR stopped working for protobuf

[zhangskz]

The publish-to-bcr app seems to have stopped working for https://github.com/protocolbuffers/protobuf as of our recent v28.0-rc3 and v27.4 releases: protocolbuffers/protobuf#17998

No commit or PR was made by the app to/from our bazel-central-registry fork. It doesn't look like there have been recent changes to protobuf's .bcr files or MODULE.bazel, nor publish-to-bcr itself.
This PR was cherry-picked to both branches, but doesn't seem to be related. Creating PR manually to publish to BCR seems to have worked fine:

I have double checked / fixed our app configurations that access is granted to the ruleset repo and bazel-central-registry fork, but this was still broken for the latest v28.1 release

Are there logs somewhere to help understand what changed here?

[kormide]

It looks like for protobuf releases specifically the cloud function is running out of memory and crashes. I increased the limit and deployed to prod. Let's leave the ticket open until your next release to see if it works.

[zhangskz]

Thanks! Can confirm after our next release. Is there a trigger to run re-run publish-to-bcr manually when it fails? Are there logs somewhere we can access to investigate in the future?

Also, anything notable about protobuf releases s.t. we are hitting these OOM / crash but seems like not others?

[kormide]

Thanks! Can confirm after our next release. Is there a trigger to run re-run publish-to-bcr manually when it fails? Are there logs somewhere we can access to investigate in the future?

Unfortunately there isn't an easy way to retrigger as it runs on a release webhook event. You could manually delete and then re-publish the release. It's often easiest to just cut a new patch version. In the future I expect Publish to BCR will be offered as a GitHub Action, so you could have more control over re-runs, but likely not until next quarter.

You should receive an email notification on failure, but unfortunately the OOM case doesn't do this. I need to set up proper alerting so I can be a bit more proactive about failures like this, but haven't gotten around to it.

Also, anything notable about protobuf releases s.t. we are hitting these OOM / crash but seems like not others?

I'll have to investigate this some more.

[zhangskz]

Confirming that this seems to have worked for our latest release: bazelbuild/bazel-central-registry#2805 (comment)

[kormide]

Glad to hear it works! I landed this to see if it helps with the memory footprint. I'll close this issue for now but I'll keep an eye on the RAM metrics.