Initial OpenAPI 3 support

[anjmao]

Initial implementation for OpenAPI v3.

Open API Go types are copied and modified https://github.com/getkin/kin-openapi/tree/master/openapi3
Proto Go types are copied and modified from https://github.com/google/gnostic/tree/master/openapiv3

There is still quite some work to do, but this PR could be merged and improved with separate PR's.

Build and run locally

Change bug.gen.yaml to point to openapiv3.

export PATH=$PATH:/path/to/grpc-gateway/bin
go build -o bin/protoc-gen-openapiv3 ./protoc-gen-openapiv3 && buf generate

[google-cla]

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

[google-cla]

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

[helbing]

Hi, is this feature still under development?

[anjmao]

Hi @helbing

I'm gonna fix CI pipeline and we could merge initial version. There are still some work to do but this one doesn't impact v2 generator.

[anjmao]

@johanbrandhorst I can't figure out why generate step fails with error:

go: downloading google.golang.org/grpc v1.40.0
go: downloading github.com/google/go-cmp v0.5.6
go: downloading github.com/antihax/optional v1.0.0
go: downloading golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f
go: downloading github.com/rogpeppe/fastuuid v1.2.0
go: downloading github.com/stretchr/testify v1.5.1
go: downloading gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127
go: downloading github.com/kr/pretty v0.1.0
go: downloading github.com/kr/text v0.1.0
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/davecgh/go-spew v1.1.0
go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go: downloading google.golang.org/appengine v1.6.6
go: downloading golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4
go: downloading golang.org/x/sys v0.0.0-20210510120138-977fb7262007
go: downloading golang.org/x/text v0.3.5
go: finding module for package github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv3/options
github.com/grpc-ecosystem/grpc-gateway/v2/internal/descriptor imports
	github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv3/options: module github.com/grpc-ecosystem/grpc-gateway@latest found (v1.16.0), but does not contain package github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv3/options

Exited with code exit status 1

[johanbrandhorst]

@johanbrandhorst I can't figure out why generate step fails with error:

go: downloading google.golang.org/grpc v1.40.0
go: downloading github.com/google/go-cmp v0.5.6
go: downloading github.com/antihax/optional v1.0.0
go: downloading golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f
go: downloading github.com/rogpeppe/fastuuid v1.2.0
go: downloading github.com/stretchr/testify v1.5.1
go: downloading gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127
go: downloading github.com/kr/pretty v0.1.0
go: downloading github.com/kr/text v0.1.0
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/davecgh/go-spew v1.1.0
go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go: downloading google.golang.org/appengine v1.6.6
go: downloading golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4
go: downloading golang.org/x/sys v0.0.0-20210510120138-977fb7262007
go: downloading golang.org/x/text v0.3.5
go: finding module for package github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv3/options
github.com/grpc-ecosystem/grpc-gateway/v2/internal/descriptor imports
	github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv3/options: module github.com/grpc-ecosystem/grpc-gateway@latest found (v1.16.0), but does not contain package github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv3/options

Exited with code exit status 1

Took me a while to narrow this down, but the issue is that we're running make clean which deletes the generated files in protoc-gen-openapiv3/options, then running make generate, which doesn't generate the files again because protoc-gen-openapiv3 has been added to the ignores in buf.yaml. When we run go mod tidy it fails because there are no go files in that directory. You need to ensure that the files are regenerated when running make generate, ideally by removing them from the ignore in buf.yaml.

[johanbrandhorst]

@anjmao I think all of the maintainers and users of this tool appreciates your work on this, but I must express my concern with the methodology of copying code from other open source projects. As an open source author and maintainer I think it is extremely important to provide the correct license attribution when taking inspiration from other code. It is good to see that you say where you copied code from, but we need to think about how to comply with the license of the sources. The getkin/kin-openapi project license is MIT, and the google/gnostic project license is Apache2. This is a BSD 3-clause project, and I'm not sure we're even able to comply with those licenses.

Additionally, this PR looks like basically a copy-paste of protoc-gen-openapiv2 with some new types overlaid on top. It's extremely hard to review, since openapiv2 is already a very complex piece of code. I stress that we want to be able to support an openapiv3 generator, but I'm just not sure how this could possibly be maintained by us. I would much prefer if we scaled back this initial attempt to something small, bootstrapping the new generator and then working towards improving it. I'd ideally not see it based on the protoc-gen-openapiv2 implementation at all. What do you think?

[anjmao]

@johanbrandhorst Hi. I totally agree. There are too many changes in openapi v3 spec and v2 generator makes it much more complex than it should be.

And for licences maybe it would make more sense to just generate go types and proto types from raw open api v3 spec JSON.

[johanbrandhorst]

How do you want to proceed? I think the only really sustainable way to build the openapiv3 generator would be to start small and incrementally add features. The first version could just output a basic schema with services and types, no annotation parsing, no configurability.

[casimcdaniels]

@johanbrandhorst It's been a while since this work has been looked at it seems. Is it possible to accept the current work from @anjmao as a stopgap solution and work towards a more standalone codebase going forwards? OpenAPI 3.0 support has been a long requested feature (several years now) for this project and I'm sure just getting the initial work out the door will accelerate improvements over time.

[johanbrandhorst]

I'm sympathetic to the community's wishes to have an OpenAPI 3 generator, but I think my latest comments made my stance clear - if someone wants to contribute this, start small, contribute a from-scratch MVP implementation that can be extended as necessary. This wholesale rewrite based on the OpenAPI 2 generator is not sustainable for a single contributor and such a small project as ours.

[zalavaari]

If there would be a way to donate/sponsor this task, I'd be happy to support. I can imagine there would be others.

[johanbrandhorst]

If there would be a way to donate/sponsor this task, I'd be happy to support. I can imagine there would be others.

I appreciate the message of support :). I'm not personally interested in taking sponsorship for a feature like this, and we don't have a history of sponsored contributions. What usually works for us is if an interested business can dedicate one of their engineers time to the effort. That has delivered many great contributions in the past.

[hypnoce]

@anjmao I would suggest that you start owning the generator on your own repo with releases, pre build binaries... If there is an appetite, community will follow, support maturity will increase. WDYT ?
I'm personally interested in the tool, can provide punctual contributions based on needs/bugs I face.

[anjmao]

Hi folks. Sorry for late response. Internally in our company we ended using https://github.com/OpenAPITools/openapi-generator-cli to convert v2 to v3 automatically. It's not ideal, but covers all our current needs. And as @johanbrandhorst mentioned it would be better to start small and from scratch.