Compatibility of generated Python code with protoc >= 3.19.0

[aabmass]

I have some questions regarding the compatibility guarantees of generated code with individual language APIs, specifically Python. According to https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates:

each language has its own major version that can be incremented independently of other languages, as covered later in this topic with the Python release. [...] The first instance of this new versioning scheme is the new version of the Python API, 4.21.0

As everyone is aware from #10051

Python upb requires generated code that has been generated from protoc 3.19.0 or newer.

That makes sense going forward when trying to use upb with protobuf 4.x. Some questions:

1. Will code generated with protoc >= 3.19.0 continue to work with Python protobuf 3.x? When will protoc stop being compatible with protobuf 3.x?
2. Now that individual language API major versions are incremented independently, how are users expected to know which protoc version generates code compatible with which individual language API version? Right now, the latest protoc version is 3.21.10 which "targets" python 4.x afaict. The mismatch of major versions here is confusing.

I found #4945 referencing compatibility tests, but they were deleted in #8570. Apologies if this Is this documented somewhere and I missed it.

Additional context

open-telemetry/opentelemetry-python#2880 (comment)

[fowles]

Historically protobuf has been very bad about providing a crisp definition here. The good news is that I have something in progress that should define this more carefully and I will try to publish it in the next month or so (holidays may interfere)

[aabmass]

@fowles that's great to hear, looking forward to it.

In the shorter term, could I get some clarity on if code generated with latest protoc version 3.21 is backward compatible with protobuf 3.x python API? I am aware of the API breaking changes, just wondering about the generated code. Based on some experiments, things seem to work fine.

The context here is that many libraries authors would like to support both 3.x and 4.x for some time to avoid creating dependency conflicts for library users in their transitive dependencies.

[fowles]

The code generated by 4.21.x happens to be backwards compatible with 3.20.x and (I think) 3.19.x. The eventual criteria we are aiming for will not guarantee that code generated from a newer compiler supports an older runtime.

[aabmass]

The code generated by 4.21.x happens to be backwards compatible with 3.20.x and (I think) 3.19.x.

Anecdotally, protoc 3.19.4 works with opentelemetry-proto >= 3.19 and gives the upb benefits for 4.x. protoc 3.20 did not work with protobuf 3.19.x.

[tilsche]

I have to say I am truly puzzled. How is a Python project using protobuf supposed to cleanly manage its dependency?

I would really appreciate any documentation, best practices, or at least sanctioned reference projects.

Our thoughts were this:

• Compiled _pb2.py files don't belong under version control because they are compiled.
• Therefore we have to detect and use protoc during setup.py
• Since protoc doesn't come from pypi (except for unofficial short-term packages - we have to use the protoc from the developer or build environment
• However, we cannot make too strict assumptions on the protoc version due since they come from different sources
• So we have to support multiple protoc versions
• But we have had incompatibilities between protobuf and the generated _pb2.py files if the protoc and protobuf versions don't match
• So we need to set a dependency to the specific protobuf version based on the protc version during compilation
• Previous to the 2022 versioning change we basically depend on protobuf=m.n.* based on m.n.p from protoc --version (which felt already pretty unclean)
• But now, on the one hand, to keep ensuring protoc/protobuf compatibility we'd have to depend on protobuf=*.n.* - which to my knowledge is not possible to formulate as requirement
• And of course this raises the question of compatibility between our code and the protobuf library due to major version changes in protobuf.
• But how does one retain compatibility to a range of protoc versions while also specifying compatibility to protobuf?

What we came up with is to sort of hardcode a minor->major map in our setup.py, so the build works sort of like this

• Get m.n.p from protoc
• m' = 3 if n < 21 else 4
• Depend on protobuf>=m'.n,<m'n+1

But this means hardcoding these major version jumps.

Also when a major jump happens, let's say to 5.25.0 and we want to support the latest protoc 3.25.0 on development environments, we need to 1) check/hope that the major version is compatible with 5. and 2) add this to the hardcoded list because there will never be a 4.25.0 - right?