Latest version of google.golang.org/protobuf (v1.33.0) is no longer compatible with the latest version of github.com/golang/protobuf (v1.5.3)

[vadimsht]

What version of protobuf and what language are you using?
google.golang.org/protobuf (v1.33.0)
github.com/golang/protobuf (v1.5.3)

What did you do?
Trying to compile a package that imports github.com/golang/protobuf/protoc-gen-go/descriptor results in

# github.com/golang/protobuf/protoc-gen-go/descriptor
../../../../../../../golang/modcache/github.com/golang/protobuf@v1.5.3/protoc-gen-go/descriptor/descriptor.pb.go:106:61: undefined: descriptorpb.Default_FileOptions_PhpGenericServices

What did you expect to see?

No build errors :)

What did you see instead?

Build errors :)

This seems to be a combination of protocolbuffers/protobuf-go@99e193e#diff-ae68ca51f2811349cc1f596970cc40bb7fa15fd893bea35e199f3cbd7ca2b486L2247 that removed Default_FileOptions_PhpGenericServices and https://github.com/golang/protobuf/blob/master/protoc-gen-go/descriptor/descriptor.pb.go#L106 that still references it.

[vadimsht]

This blocks picking up a fix for https://groups.google.com/g/golang-announce/c/ArQ6CDgtEjY/m/oLMrdq_GBQAJ in code bases that still uses deprecated Go package. While the "correct" solution is to stop using the deprecated package, it is not really possible to do fast. And running vulnerable code is not nice... Please consider releasing a new compatible github.com/golang/protobuf version.

[rootulp]

+1 I encountered this on celestiaorg/celestia-app#3155. See this error.

[neild]

Gah, apologies, I didn't expect that.

I think we have to make an updated version of github.com/golang/protobuf that updates descriptor.pb.go, thanks to a backwards-incompatible change in descriptor.proto.

[jerrybean]

Does google.golang.org/protobuf need a newer version than v1.33.0? I saw github.com/golang/protobuf (v1.5.4) already.

[rootulp]

Seems resolved via #1597 because https://github.com/golang/protobuf/releases/tag/v1.5.4 works for me.

[neild]

Sorry for the inconvenience! github.com/golang/protobuf v1.5.4 should be compatible with google.golang.org/protobuf v1.33.0.

The problem, for reference, is that:

• descriptor.proto (part of the upstream protobuf project) dropped the FileOptions.php_generic_services field.
• google.golang.org/protobuf includes the generated package for descriptor.proto.
• google.golang.org/protobuf v1.33.0 updates descriptor.proto, resulting in the const descriptorpb.Default_FileOptions_PhpGenericServices being removed.
• github.com/golang/protobuf, for historical reasons, includes a stub version of descriptor.proto that forwards all its symbols to the canonical one in google.golang.org/protobuf.
• github.com/golang/protobuf v1.5.3 is now incompatible with google.golang.org/protobuf v1.33.0, because it references a symbol that doesn't exist.

This is fixed in v1.5.4 by regenerating the forwarding package.

In retrospect, a better fix might have been in google.golang.org/protobuf: We could have added a declaration for Default_FileOptions_PhpGenericServices to that package for backwards compatibility. If we'd detected the problem in time, we could have done that in the v1.33.0 release and maintained compatibility with v1.5.3.

We should have a test that ensures this doesn't happen again in the future.