Integrating NDB into gcloud.datastore?

[jgeewax]

Had a few discussions with @GoogleCloudPlatform/cloud-datastore (particularly @pcostell) so I wanted to summarize the things we covered. This issue is to clarify the goals for Datastore support in gcloud-python, and discuss and decide on a course of action that everyone likes for how to reconcile what we have today with the other libraries out there.

---

Current state of the world

Currently, I see two styles that people want to use when interacting with Datastore:

1. a "simple" more key-value-based style (similar to Amazon's Simple DB) where you're CRUDding dictionaries and then adding some querying
2. a more advanced ORM style where you're create models, and have some sort of schema defined in a Python file.

For the former, gcloud.datastore has had the goal of covering this use case. For the latter, ndb is the latest way (supported) way of doing this -- with others potentially existing, but ndb seems to be the clear leader.

We also have a unique situation where our code currently might have trouble running in App Engine -- whereas ndb can't run outside of App Engine. The layout sort of looks like this:

Looking forward

If our goals are ....

1. We agree that both styles of interacting with Datastore matter (and both should exist)
2. gcloud.datastore and ndb are our choices for each style respectively
3. gcloud.datastore and ndb should both run in App Engine and non-App Engine runtimes
4. gcloud.datastore is where all the recommended Python stuff to talk to Datastore should live (it is the "official source of truth")
5. People who want to write new Python libraries for Datastore can rely on code that exists in gcloud.datastore (and set gcloud as a Python dependency)

... then I'd like to suggest that we....

1. Port ndb over as gcloud.datastore.ndb (bringing with it datastore_rpc and datastore_query)
2. Rewrite gcloud.datastore to run on top of datastore_query
3. Rename gcloud.datastore to be a peer with ndb (using "simple" in this diagram, not set on that at all though).

which makes things look like this:

What do you guys think?

/cc @GoogleCloudPlatform/cloud-datastore @dhermes @tseaver @silvolu @proppy

[tseaver]

Do we know of issues using gcloud on GAE?

In the "new world order", I'm not sure what datastore_query.py and datastore_rpc.py are (I assume datastore_pb.py is really our current gcloud.datastore.datastore_v1_pb2.py). I think gcloud.datastore.simple might just be the APIs exposed currently from gcloud.datastore itself (re-exported from the underlying api, batch, connection, entity, key, query, and transaction modules).

ISTM that gcloud.datastore.ndb would sit "above" gcloud.datastore.simple in the stack, although it might be drawn as a "half layer" (to indicate that users had direct access to gcloud.datastore.simple). Also, I would think that the outer layer would just be "Anywhere" (we'd like GAE users to be able to reuse gcloud.datastore, right?)

[jgeewax]

Do we know of issues using gcloud on GAE?

I don't have confirmation either way. We certainly didn't design it with the Python App Engine sandboxing in mind... Might be worth a separate issue to look into that ?

In the "new world order", I'm not sure what datastore_query.py and datastore_rpc.py are

• datastore_rpc.py: https://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/datastore/datastore_rpc.py
• datastore_query.py: https://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/datastore/datastore_query.py

I think gcloud.datastore.simple might just be the APIs exposed currently from gcloud.datastore itself

Yes -- See item 3 in the list of suggestions ("Rename gcloud.datastore to be a peer with ndb (using "simple" in this diagram, not set on that at all though).")

ISTM that gcloud.datastore.ndb would sit "above" gcloud.datastore.simple in the stack

This was actually a point of contention -- because I don't think gcloud.datastore.simple (what we have as gcloud.datastore today) should support everything that's absolutely needed by a library as complex as NDB. I have a hunch that something like NDB would need access to protos or something lower level (like datastore_query).

I also suspect it will be more work to port NDB on top of gcloud.datastore.simple than it would to pull over datastore_rpc and datastore_query and have it continue living on top of those.

[pcostell]

gcloud should work asis in GAE right now, but its performance would be significantly worse than ndb's. Implementing on top of something like datastore_query and datastore_rpc would allow it to use either datastore_v1 or datastore_v3.

JJ was able to convince me that NDB should not be implemented on top of gcloud.datastore.simple. However, I do think that the simple implementation should be pretty minimal with most of the heft in datastore_query and datastore_rpc.

Importantly though, I think that datastore_rpc and datastore_query will need some work before they become part of gcloud. I'm almost done getting them to support both Cloud Datastore and datastore_v3. However I think they'll need some serious API cleanup before being part of the gcloud library.

Personally (I don't think we really talked about this), I would like to see them result in a nice API for library developers or for customers who want more control than ndb or simple provide. I think right now there is some overlap between the existing implementation in gcloud and datastore_rpc and datastore_query. It may be worthwhile to see how we can best merge the API of the first with more of the functionality of the second.

[dhermes]

RE: gcloud working on App Engine. It does: https://github.com/dhermes/test-gcloud-on-gae

As Patrick mentions, ndb saves take about 1/5 the amount of time the gcloud saves do because the RPCs happen directly in GAE instead of via the Cloud Datastore API.

[dhermes]

@jgeewax and @pcostell know, but for those who don't I've also taken a stab at converting ndb to a project that builds on Travis, uses tox, etc.:
https://github.com/dhermes/ndb-git

@jgeewax There is a lot going on in ndb besides just an ORM. I think the scope of it justifies being it's own library. It can always have gcloud as a dependency.

ISTM (given the observed timing difference on GAE) that we should use datastore_rpc and datastore_query under the covers instead of using the _datastore_v1_pb2.py module generated from the proto. @pcostell I'm happy to help in that effort if possible.

[jgeewax]

There is a lot going on in ndb besides just an ORM. I think the scope of it justifies being it's own library.

Can we look at what those things are? The reason I ask is that we have to find a happy balance between having one entry point for Google Cloud stuff in Python (pip install gcloud) and dividing things apart for us developers (issues, code maintenance, documentation, and everything else I'm sure I'm forgetting).

Would this be a good fit for a git submodule? Where as far as our users are concerned, they still type pip install gcloud and from gcloud.datastore import ndb, but the code, issues, and everything else would live in a separate GitHub repository?

[dhermes]

I recommend taking a peak at ndb's source. Every piece of code is in on the theme of async, and as a result, the code is quite complex.

The primary modules doing non-ORM type things are:

• context.py (3rd largest module)
• tasklets.py (4th largest module)
• eventloop.py

These are great features of ndb.

---

Just to get a sense of the size, we have about 2500 lines of non-test / non-generated code:

$ cd gcloud/datastore/
$ ls *py | egrep -v test | egrep -v pb2 | xargs wc -l
  223 api.py
  333 batch.py
  449 connection.py
  106 entity.py
  321 helpers.py
   12 _implicit_environ.py
  140 __init__.py
  362 key.py
  493 query.py
  160 transaction.py
 2599 total

---

On the other hand, ndb has nearly 12,000

$ cd ${GOOGLE_CLOUD_SDK}/platform/google_appengine/google/appengine/ext/ndb/
 $ wc -l *py
   459 blobstore.py
  1277 context.py
    65 django_middleware.py
   306 eventloop.py
    64 google_imports.py
    17 __init__.py
   805 key.py
   335 metadata.py
  3935 model.py
   432 msgprop.py
   242 polymodel.py
   194 prospective_search.py
  2042 query.py
   451 stats.py
  1149 tasklets.py
   220 utils.py
 11993 total

---

My current gcloud SDK environ at the time was:

$ gcloud -v
Google Cloud SDK 0.9.43

...
app-engine-python 1.9.17
...

[dhermes]

RE: Using a git submodule, @tseaver has expressed a distaste for it. We can ship ndb in our package on PyPI without using a submodule with custom build logic, so that may not really be an issue.