question: .proto packaging / repository layout

[kudla]

Hi
Trying to summarize the question is something like
How to write a pure reusable distributed proto packages with arbitrary depth of transitive dependencies.
E.g A, B->A, C->(A,B) : where A,B,C - are all separated repos with a proto declarations. The same time in a way it works for google/protobuf/empty.proto.

I'm not sure how better to structure the entire question. Instead there's a batch of tiny ones that looks contradicting the subject intention or just confusing the hole picture vision.

1. All the examples shows a package naming as "foo.bar.baz". Still google/protobuf/empty.proto is imported as some url(fs) path.
2. When referring import "some/other/mod.proto" does it really mean the referring (linking) on an initial photo-source level or at the compiled to target language one. As we can see google/protobuf/empty.proto as a source reference. Still the result compilation refers the real golang "google.golang.org/protobuf/types/known/emptypb" module.
3. The most resources say the only (best) composition for protos is a subtree/submodule linking
   3.1 So that how to keep code clean and consistent that case. As I can see every level of repo reference adds additional fs tree nesting to a target
   E.g.

repoA
   a.proto

repoB
   A(->repoA)
     a.proto
   B
     b.proto

repoC
  A(->repoA)
       a.proto
  B(->repoB)
     A(->repoA)
       a.proto
     B
       b.proto
   C
      c.proto

3.2 As well. How to avoid a code duplication in case of C->(B, A) if B is just a black box that doesn't leak it's dependancies and to compose all the reps within C as flat set.
3.3 Anyway in case of google/protobuf/empty.proto we don't subtree/fetch/declare whatever manually at all. So how to achieve. the same with custom repos.
4. How to make it even fluent and language agnostic in different packaging managers, e.g. go mod, npm etc. as the mentioned google/protobuf/empty.proto does.

Thank you

[stapelberg]

Hey @kudla

Your question is long and has many parts, not all of which I understand.

I’ll try to help as best as I can:

1. The package name (e.g. google.protobuf) is separate from the file system path (e.g. google/protobuf/empty.proto) of a .proto file. While they are separate, a common naming strategy is for the package to match the file system directory (as seen here).

2. The path google/protobuf/empty.proto refers to a “well-known proto”, i.e. one which is shipped with protoc itself.

3. If, during proto compilation (running protoc), you want to import other .proto files, you need to adjust protoc’s import search path by using the -I flag.

   • The well-known protos are found the same way: the import search path by default points to the location where the well-known protos can be found, which is the include/ directory next to the bin/ directory in which protoc lives:

   % unzip -l protoc-29.1-linux-x86_64.zip 
   Archive:  protoc-29.1-linux-x86_64.zip
     Length      Date    Time    Name
   ---------  ---------- -----   ----
           0  1980-01-01 00:00   bin/
     9623664  1980-01-01 00:00   bin/protoc
           0  1980-01-01 00:00   include/
           0  1980-01-01 00:00   include/google/
           0  1980-01-01 00:00   include/google/protobuf/
        6154  1980-01-01 00:00   include/google/protobuf/any.proto
        7729  1980-01-01 00:00   include/google/protobuf/api.proto
           0  1980-01-01 00:00   include/google/protobuf/compiler/
        8556  1980-01-01 00:00   include/google/protobuf/compiler/plugin.proto
        2185  1980-01-01 00:00   include/google/protobuf/cpp_features.proto
       52531  1980-01-01 00:00   include/google/protobuf/descriptor.proto
        4892  1980-01-01 00:00   include/google/protobuf/duration.proto
        2363  1980-01-01 00:00   include/google/protobuf/empty.proto
   […]
   

4. For repository b to work with a service that is implemented in repository a, it is typically sufficient to import the generated protobuf Go code (.pb.go files). You typically don’t need the .proto files from a in b.

5. (I don’t know enough about npm or other package managers to say anything about them.)

[kudla]

Hi @stapelberg . Thank you a lot for a such elaborate answer.

not all of which I understand.

I bet this is due to my initial lack of understanding either.

Looks I can see my crucial misunderstanding know. I though having any deps we should provide per each one both .proto sources (to statically match typings) and compiled libs to link the target lang deps.

Please confirm I see it the right way now. Do we have two SEPARATE options to resolve every single deps

• to provide a path with -I flag if we have the deps as a .proto sources so that to be compiled in place
• or just to provide a go mod url with -M in case we have compiled package to be linked straight on a go mod level?

[kudla]

Ah, looks it is still not))

Without deps .proto sources the actual references other.package.Message are stoped to be resolved.

Anyway the separation of terms package name vs file system path looks very helpful as well. Need to rethink the whole story once again.

[stapelberg]

Please confirm I see it the right way now. Do we have two SEPARATE options to resolve every single deps

• to provide a path with -I flag if we have the deps as a .proto sources so that to be compiled in place
• or just to provide a go mod url with -M in case we have compiled package to be linked straight on a go mod level?

Both -I and --go_opt=M… are command-line flags for protoc (the protobuf compiler):

• The -I flag extends the import search path, making protoc find .proto files on disk, i.e. translating from relative paths like google/protobuf/duration.proto to absolute file system paths like /opt/protobuf/include/google/protobuf/duration.proto.
• The M option overrides the Go package import path for generated code (only needed if you do not specify option go_package = "…"; in your .proto file), see https://protobuf.dev/reference/go/go-generated-opaque/#package

I would not call this “two separate options” at a high level — yes, these are two flags, and they are related, but they are not replacements for each other.

In general, protoc is only needed if you want to compile .proto files into Go code. Most often, you would do that compilation within one repository and then only export the generated Go code, because downstream users often do not need the .proto files.

Without deps .proto sources the actual references other.package.Message are stoped to be resolved.

Yes. If the repositories are not supposed to depend on each other’s .proto files directly, you probably need to reach for proto extensions: https://protobuf.dev/programming-guides/editions/#extensions