third_party protos are missing in 0.46.0

[webmaster128]

I'm trying to upgrade Cosmos SDK proto types from 0.45.3 to 0.46.0 for cosmjs-types, a proto -> TypeScript code generator. Unfortunately since the upgrade, third_party proto files are not found anymore.

cosmos_proto/cosmos.proto: File not found.
gogoproto/gogo.proto: File not found.
google/protobuf/any.proto:35:1: Import "gogoproto/gogo.proto" was not found or had errors.
cosmos/auth/v1beta1/auth.proto:4:1: Import "cosmos_proto/cosmos.proto" was not found or had errors.
cosmos/auth/v1beta1/auth.proto:5:1: Import "gogoproto/gogo.proto" was not found or had errors.
cosmos/auth/v1beta1/auth.proto:6:1: Import "google/protobuf/any.proto" was not found or had errors.
cosmos/auth/v1beta1/auth.proto:21:3: "google.protobuf.Any" is not defined.

Turns out the files were included here but are now missing.

I tried to get them locally using make proto-update-deps but the sed command seems to be GNU sed specific and fails on Mac. Also not all URLs have versions pinned so that approach would risk using different version than the SDK.

I wonder, was the removal of those files intended? For me it would be great if they were shipped with the cosmos-sdk source code as before.

[julienrbrt]

I don't know the reason, but as I am on linux, I went ahead, and I have re-generated the files, based on the pinned version: #12992. We can merge this PR from that branch and backport if it was an omission in v0.46.

[webmaster128]

Thanks a lot @julienrbrt. Having that on the 0.46 branch (even as an unreleased commit) would be very helpful for me.

[amaury1093]

This was actually intentional, it's to avoid needing to copy/paste all the proto files in each repo. So instead, we put all proto files in the buf registry.

For code generation, there are buf.yaml and buf.lock files for dependencies and their pinned versions. Then the buf generate command does all the magic. See protocgen.sh for an example. Cosmjs should have its own buf.yaml with buf.build/cosmos/cosmos-sdk as dep, to avoid having to copy paste the files.

What we probably need is a migration guide for how to go from local *.proto files to using buf, this is definitely missing. But I don't think we should put back the third_party proto files, that seems like a step back.

[webmaster128]

Fair point. But why would I use buf? I don't need it and the tooling we are working on for code generation does not use protoc at all.

[amaury1093]

the tooling we are working on for code generation does not use protoc

OK, what are you using? And why didn't you choose to use protoc (which seems to be industry standard)?

The SDK has been using buf/protoc as tooling for 2y+ now, so if you use other tooling, then I guess you'd need to download and track the .*proto files manually into your repo. Our buf.lock gives the exact commits we're using, but I think it's redundant to copy the proto files again in the SDK.

[webmaster128]

Right now we use protoc with the ts-proto plugin, which works reasonably well since Stargate launch. The next step is using telescope, which parses the .proto files directly, giving us tons of benefits wrt. the generated code.

Can I use buf to just provide me a third_party folder (i.e. do all the resolution/downloading) and I continue with plain .proto files from there?

[amaury1093]

Yes, buf export buf.build/cosmos/cosmos-sdk --output <some_folder>.

But thinking again, I do agree the transition to buf generate came too suddenly, without much migration docs. Maybe we can keep the third_party folder for v0.46, and also add a README.md in that folder saying that v0.46 will be the last version to use this folder? This way you don't need to introduce buf right now, but can do it sometime before v0.47.

Does that work as a compromise @webmaster128 ?

[webmaster128]

buf export looks very good to me. The output is all I need for the existing tooling to pick up. Is there a way to pin the cosmos-sdk version of commit? Right now I get master .protos. If version pinning can be done, I guess I don't need a backport of the files. Maybe buf export is Collin's friend too.

[julienrbrt]

If this solution work, we can add something about it in the UPGRADING.md.