Port Datastore onto GAPIC

[bjwatson]

What

Port the gRPC code path for Datastore onto GAPIC. The package is available here.

Why

This is not a beta-blocker, since there already is gRPC code for Datastore, but it is preferred.

[dhermes]

@bjwatson Two questions:

1. Why doesn't the new package depend on google-gax>=0.15 (which has the new auth helpers)?
2. Are we in a situation where some of the GAPIC packages depend on 0.13 <= google-gax < 0.14 while others depend on 0.14 <= google-gax < 0.15 (as this one does)?

[bjwatson]

@dhermes Only because @geigerj says that he needs to make some small updates to the GAPIC generator for it to be compatible with google-gax>=0.15. He might be able to get that done today, and then we can build all packages with the newer google-gax dependency.

I thought I would get this in your hands right away, rather than waiting for the necessary fixes to support the newer auth helpers. However, if you think it's a more efficient use of your time to wait, then we can do that.

[dhermes]

@bjwatson Sorry I didn't notice this until now, but the current form of grpc-google-cloud-datastore-v1 can't run without gRPC. This means we can't get access to the protobufs without forcing grpc to be installed.

Right now we support Protobuf-over-HTTP and Protobuf-over-gRPC, so we need access to the protobuf classes in use cases where gRPC isn't available (i.e. App Engine).

The generated protobuf classes currently live in datastore._generated, which is how we import them. In order to support gRPC we have datastore_pb2.py and datastore_grpc_pb2.py in the _generated module

/cc @jonparrott

[bjwatson]

@dhermes Does googleapis/api-client-staging@6649c45 do what you need? Do you also need changes to setup.py? I see that your setup.py depends on grpcio regardless.

Note that this will become unnecessary once grpc/grpc#8056 is released in grpcio==1.1.0.

[dhermes]

Yes it does do what I had in mind.

We had conditional checks in our setup.py, but @jonparrott discouraged that since it confuses wheel building. But maybe I am wrong on how I remember that.

AFAIK, even though grpcio is in our setup.py for packages like google-cloud-datastore, it doesn't stop from installing into a lib/ dir in an App Engine app. @jonparrott can confirm or deny or throw his hands up in sadness over packaging?

[theacodes]

grpcio will indeed be vendored into lib for app engine and explode on import.

[dhermes]

@bjwatson Ping me once released?

[geigerj]

@bjwatson Is this a Datastore-specific hand-edit, or do we intend to provide these conditional imports across the gRPC packages we maintain? If it's the latter, we should figure out what the timeline for grpcio 1.1.0 is and whether we need to automate this in our tooling.

[dhermes]

@geigerj @bjwatson We would love to see a new release of grpcio because of grpc/grpc#7849, so I am on team "pushing for release"

[bjwatson]

@dhermes We're going to wait on porting Datastore to GAPIC until after the beta announcement. We discussed and we don't want to take any chances on destabilizing the google-cloud-python code for the four APIs. The only one that will go beta with GAPIC is Logging.

[dhermes]

SGTM