Allow excludes for buf generate

[zerospiel]

Hi! I have, for sure, pretty common protoc generation flow but when I tried to migrate from generic buf protoc or buf image && protoc, I've got an unexpected behavior of buf generate subcommand.

Preambula

Let's explore the next simple file.proto:

syntax = "proto3";

package some.service.v1;
option go_package = "github.com/user/repo/pkg;some_service";

import "google/api/annotations.proto";

service SomeService {
    rpc Get(Response) returns (Response) {
        option (google.api.http) = {
            get: "/version"
        };
    }
}

message Response {
    string some_name = 1;
}

message Request {}

The file structure then looks like this:

├─ protos/
│       ├─ google/api/
│       │         ├─ annotations.proto
│       │         ├─ http.proto
│       ├─ file.proto

And to generate files I use the following command, assuming I'm generating for golang using vanilla plugins and have module github.com/user/repo:

# here I mitigate some mappings, they do not matter
buf protoc --proto_path ./protos/ \
   --go_out=module=github.com/user/repo:. \
   --go-grpc_out=module=github.com/user/repo:. \
   ./protos/file.proto

And then I have perfectly and insanely fast generated files:

├─ pkg/
│    ├─ file.pb.go
│    ├─ file_grpc.pb.go

Issue

Now let's deep dive into the issue! Assume I have the following simple buf.yaml contents:

version: v1beta1
build:
  roots:
    - protos
lint:
  use:
    - MINIMAL
  except:
    - PACKAGE_DIRECTORY_MATCH
  ignore:
    - google

And then I try to reproduce the above generation using the next buf.gen.yaml template:

version: v1beta1

plugins:
  - name: go
    out: .
    opt: module=github.com/user/repo
    strategy: directory
  - name: go-grpc
    out: .
    opt: module=github.com/user/repo
    strategy: directory

And now using buf generate . I've started to get the following error:

plugin go: google.golang.org/genproto/googleapis/api/annotations/http.pb.go: generated file does not match prefix "github.com/user/repo"

Of course, this is because under the hood buf trying to generate (see strategy description) from all directories within roots from buf.yaml. But I need that protos only for vendor purposes without any generation for those files.

Unfortunately, I didn't found out any way to NOT generate those google/api proto files and it's not possible for me to use buf generate at this moment.

Let me know if I did something wrong or this is totally expected behavior and won't change in any observable future.

Possible solution / proposal

It'll be great to see an additional option to generate subcommand template something like vendor or maybe excludes (to correspond to roots from the main config) that will NOT generate any proto files from the described directories. Not sure if that option should be plugin-related or the whole template-related, so there are 2 options:

# buf.gen.yaml

version: v1beta1
plugins:
  - name: go
    out: .
    opt: module=github.com/user/repo
    path: ./bin/protoc-gen-go
    strategy: directory
    # plugin related
    excludes: google, third-party, any/other/directory
# or template related
excludes:
  - google
  - third-party
  - any/other/directory

Of course, directories used in this option should be consistent with roots otherwise the whole generation will be broken.

[bufdev]

We've considered this - for now, we want to avoid doing this as our upcoming protobuf module system will make it so you don't need to do vendoring whatsoever. We'll consider in the near future.

[zerospiel]

@bufdev is there any code that I can deep into to walkthrough? I've done a corporate tool to automatically vendor all imports and some stuff related to generation, that's why I'm very interested in the current state of this feature.

Also, I have to ask about priority: is there any possibility to use buf generate in the described above state? This is very important for me because my company actively using grpc-gateway, googleapis, and internal proto files, so I have to understand if it's possible to migrate from vanilla protoc to buf generate in the near future (I'm considering end of January 2021).

Maybe there is an option to use some kind of excludes as a work-in-progress state before releasing auto vendoring (or whatever you're developing now)

[bufdev]

There's no code to walkthrough here, and please do not submit a PR - we haven't decided yet to implement this, although we're thinking about it.

Priority for this is low - we want to get the whole picture right here, and don't want to release features that we'll later delete. We'll update this issue if/when we get to this. For now, I'd recommend sticking to buf protoc for your use case, or using buf generate --path to filter for includes.

[akshayjshah]

👋🏽 Hi friends! I ran into the same problem, but for a slightly different reason: I'd like to restrict grpc-go codegen to a subset of my protobuf IDL, even though many protobuf files define services. I can imagine the same scenario occurring during migrations, when you'd like to keep an allowlist of protobuf files allowed to use a deprecated plugin.

Some more (very predictable) context: I'm working on a small library that implements the gRPC protocol. All the protobuf IDL in the repo is mine - there's nothing vendored. To test my library for compatibility with other gRPC implementations, I've got a suite of cross-tests. Of course, I don't want my library to depend on other gRPC implementations, so the cross-tests are in their own module. Unfortunately, buf generate is dropping protoc-gen-go-grpc output all over the place.

It's easy enough to delete the files I don't want, but it makes the Buf experience feel less magical. ✨

[johanbrandhorst]

One current workaround is to use a separate generation template for protoc-gen-go-grpc and use the --path flag in a second invocation, only selecting the files/folders you need:

$ buf generate --template buf.go-grpc.gen.yaml --path my/test/protos --path my/other/test/protos

The grpc-gateway currently uses this pattern to generate files with many different combinations of generators and options: https://github.com/grpc-ecosystem/grpc-gateway/blob/master/Makefile#L97-L124.

[bufdev]

Quick update: I believe we'll cover this use case soon, we are going to extend the generate command a bit, stay tuned.

[akshayjshah]

👍🏽 I'll watch this issue for updates.

[doriable]

We just released 1.0.0-rc8, which adds the --exclude-path flag to generate (along with build, breaking, lint, and export). This will allow users to specify paths that will be excluded from the operation (similar to the --path flag specifying targets). The two flags, --exclude-path and --path, can be used together, however the exact same path cannot be set for both.

Closing the issue for now, but feel free to re-open or reach out with any questions!

[rbroggi]

Our organisation would have benefited of a plugin-level --exclude-path like in the original proposal:

# buf.gen.yaml

version: v1beta1
plugins:
  - name: go
    out: .
    opt: module=github.com/user/repo
    path: ./bin/protoc-gen-go
    strategy: directory
    # plugin related
    excludes: google, third-party, any/other/directory
# or template related
excludes:
  - google
  - third-party
  - any/other/directory

We have so far worked around it by splitting the buf.gen.yaml and making 2 separated calls to buf which sounds as a far less clean solution.

[ln-12]

Hey @doriable, are there any plans yet to (also) support the configuration approach shown above?

[doriable]

In v2 of buf.gen.yaml, we now have an input key: https://buf.build/docs/configuration/v2/buf-gen-yaml#inputs
After many discussions, we decided to go down the path of setting inputs explicitly rather than excluding.