Tracking issue: Opaque API

[stapelberg]

This issue provides a central place for questions and discussion around the Opaque API.

Please read the Opaque API announcement blog post first!

Before posting questions, please check if the Opaque API FAQ answers your question.

Release Plan / Status

• File this tracking issue
• Merge code changes into protobuf repository
• Update documentation on protobuf.dev
• Publish go.dev announcement blog post: https://go.dev/blog/protobuf-opaque
• Publish open2opaque migration tool: https://google.golang.org/open2opaque

[stevvooe]

Is there a proposal somewhere that took into account community feedback? I can't seem to find anything on the mailing lists, github or code review older than a week ago. It seems like we just went through a major API break in Go protobuf and we have yet another in just 4 years. There are still projects that are trying to migrate to the last set of big changes.

The departure of the gogo project was a net loss for the protobuf community after the 2020 change. It required a massive amount of work and testing to move to the new API with a lot of risk and very little benefit. Looking at the new motivations enumerated in the blog post, I wouldn't really put any of those concerns on even my top 10 problems when using protobuf and grpc. Even when I have encountered these problems, I was able to workaround the problem with an existing plugin, bespoke code generation or static analysis (rulesguard for the pointer compare is trivial).

Looking at the changes, its unclear why this can't be an alternative plugin, rather than a core breaking feature. At a minimum, it would seem prudent to release the Opaque API as an experimental plugin to get some feedback before considering it as a "default" in the 2024 edition. Getting some outside usage will make sure we won't have the same issues that occurred with the last API change.

I've been a long term advocate of Protobuf and GRPC for a number of projects. I've taken some criticism for that backing over the years due to the problems that has brought to projects, so it's going to be hard to continue that advocacy with yet another major change to such a foundational project. While I don't want to take away the engineering accomplishment made here, I would just hope to see some time for feedback to ensure this is the right direction.

[stapelberg]

Hey Stephen

Thank you for your message. I will try and address some of your points as best as I can, but note that I joined the project in 2022, and the Opaque API project was going on for much longer.

Is there a proposal somewhere that took into account community feedback? I can't seem to find anything on the mailing lists, github or code review older than a week ago.

There is no public proposal for this change, but we did involve the community (privately) as early as 2023 and again, in multiple feedback rounds, over the recent few weeks leading up to yesterday’s release.

Your name did not pop up when I compiled the list of folks to reach out to. If you would like to be included in the future, please let me know which company or organization you are speaking for and provide an e-mail address so we can reach out to you.

It seems like we just went through a major API break in Go protobuf and we have yet another in just 4 years.

No, this change is not a breaking API change. Your existing programs continue to work as before (nothing breaks), and the announcement clearly states that there is no obligation or expectation to migrate.

Even when the default API level changes with edition 2024, it is easy to stick to the Open Struct API by invoking protoc with the --go_opt=default_api_level=API_OPEN command-line flag (if that’s easy in your environment) or by setting the api_level editions feature option in your .proto file.

The departure of the gogo project was a net loss for the protobuf community after the 2020 change. It required a massive amount of work and testing to move to the new API with a lot of risk and very little benefit.

Just to be clear: the gogo/protobuf project was a community-driven project which was not coordinated with the Go Protobuf project.

At a minimum, it would seem prudent to release the Opaque API as an experimental plugin to get some feedback before considering it as a "default" in the 2024 edition.

We have already incorporated a lot of feedback over the last few years as we migrated Google’s Go codebase to the Opaque API. While the implementation can always be improved, the generated API itself seems pretty stable at this point.

If you have suggestions for specific changes, feel free to file them as a feature request in this issue tracker.

Thanks

[stevvooe]

I'd love to give feedback early if given the opportunity (and do reach out to me, I'm at Docker now) but I would feel better if the project could commit to public proposals in the future. Involving the community privately here is the problem. That is a very unwelcoming and un-inclusive model for change and its easy to cherry pick positive responses in such an environment. I understand why its easier to do some things in private but this project is used widely.

While its stated this is not a breaking change, if one does nothing and upgrades protoc and the go proto compiler, the generated code will be broken with the 2024 edition. I know its been tried internally at Google, but Google has a very different environment than most developers. The world that uses protobuf is not contained in a monorepo with automated refactoring and collisions in generated code between the Open and Opaque APIs may lead to problems and expensive changes.

A few problems I foresee:

1. Anyone exporting generated protobuf types will be forced to maintain the hybrid generated code. Those package maintainers can't stay on the Open API without creating problems for users of the new Opaque API and they can't move to Opaque without breaking their Open API users and matching stability expectations.
2. Controlling generation behavior in files can make portability difficult. Let's say that a rust project sets api_level to be Opaque and we need to use them in a project that is still using the Open API. One must pre-process the file to remove the setting before generating the code or be forced to update to the Opaque API.
3. Do the performance gains reference hold even when having to use a Builder+Opaque object in performance sensitive code?
4. The proposed names collide with already existing things. For example, "Open Struct API" used as "Open API" in several places, collides with "OpenAPI", which is a REST IDL. The "Opaque API" collides with an old term for unknown but carried message fields that people use to call "opaque fields" (which were broken sometime around the "syntax3" release).

I'll give the new Opaque API a try and get back to you if I discover more issues or if these aren't that big of a problem.

Just to be clear: the gogo/protobuf project was a community-driven project which was not coordinated with the Go Protobuf project.

No, it wasn't but the decisions of the Go Protobuf project effectively killed the vibrant community around that project. The Gogo project solved a number of problems that the main implementation ignored or didn't see as an issue. I hope that the attitude isn't that it is lesser because it was community-driven. Either way, it's important that this project take care not to create unintended consequences.

The specific changes I'm asking for are the following:

1. The Opaque API be marked as unstable or experimental until an appropriate feedback period has been completed. It will take time for downstream projects to digest the changes. This will let folks digest the changes and provide fixes before the impact becomes unfixable. You can set the timeline but make it public.
2. This be released as a separate plugin, ie --go_opaque_out, so that we can add it seamlessly to existing build steps and test it out separately.

I think these are reasonable actions that can be taken to create trust across the community and show folks that this can continue to be a strong, foundational dependency in their products.

Appreciate you hearing me out.

[dsnet]

As a former maintainer, I can give some historical context (also as someone who's outside of Google now).

The protobuf reflection and opaque API both began around the same time in 2018. The opaque API was motivated by at least two reasons 1) better performance not using pointers to represent presence, and 2) better compliance with the protobuf language itself. Protobuf reflection was a prerequisite in order to pursue the opaque API because there was a lot of code that used Go reflection on protobuf messages and assumed that it was of a particular representation.

Another motivation for the protobuf reflection API was because Google consistently ran into panics due to Go reflection assuming that the protobuf message was produced by the golang/protobuf generator when it was actually produced by the gogo/protobuf generator, and vice-versa. Protobuf reflection is supposed to be agnostic to the underlying generated representation.

A proposal for protobuf reflection was announced to the Go community back in 2018 and feedback was sought. Back then people had already raised concerns about gogo/protobuf and the intention was for gogo/protobuf to support protobuf reflection (in which case there would be no compatibility issues. Unfortunately, the maintainer of gogo/protobuf was unable to commit time to implement protobuf reflection. I offered to help, but when I started diving into the gogo/protobuf codebase, I realized that the great many features it supported allowed users to specify behavior that fundamentally went against the protobuf language and was therefore impossible to properly represent through protobuf reflection. Technically, it probably would have been possible to implement reflection for most features, but the large feature space of gogo/protobuf (while enjoyed by many users) was also its fatal Achilles' heel as it led to a very high maintenance burden.

(As an aside: Personally I think some of the gogo/protobuf features are reasonable. Philosophically, gogo/protobuf takes a stance of making generated Go protobufs great in Go, which is stance that many Go enthusiasts are happy with. However, the golang/protobuf project is a sub-project is part of the wider protobuf project, which has the goal of making protobufs decent in all languages, but not become overly-specialized to any particular language. Many requested features of golang/protobuf were rejected because of this philosophical difference.)

Going back to the opaque API, the intention was to open source the opaque API after the release of protobuf reflection in 2020, which would have included a more thorough discussion and feedback about what it should eventually entail. However, due to internal company turmoil, I left Google not long after, which resulted in golang/protobuf staying somewhat stagnant and the opaque API (which was already being massively used inside Google) to functionally become frozen as stable over the years. Thus, the fault of the opaque API not being more widely discussed ahead-of-time is mine and not the present maintainers. In fact, they are the ones who have to deal with my sins, so I thank them for that.

[dsnet]

Controlling generation behavior in files can make portability difficult. Let's say that a rust project sets api_level to be Opaque and we need to use them in a project that is still using the Open API.

I'm not sure I see this problem. The option is scoped to the Go language:

option features.(pb.go).api_level = API_OPAQUE;

[stevvooe]

@dsnet I'll respond in more detail but want to give you a quick answer. I think the point about rust was confusing but the classic problem with this was having generation ignore language-specific directives.

Let's assume the entire projects are in Go and is set to use the Open API:

option features.(pb.go).api_level = API_OPEN;

But, then, I have a dependency that has this:

option features.(pb.go).api_level = API_OPAQUE;

Now, one is required to adopt the opaque api if they want to use the dependency or write some dependency post-processing to remove or ignore the line.