Having trouble querying by KeyProperty

[nlokare]

Hi,

I am having trouble querying entities in NDB from AppEngine by KeyProperty for entities created with a datastore.Key from the gcloud.datastore library. When I log the KeyProperty of the saved entity on AppEngine, I see a return value of Key('Company', '273', app='random_project_id'). Because I am attempting to query by Key Literal on AppEngine, it looks like NDB does not recognize the above Key with a query filter such as Entity.query(Entity.key_property == ndb.Key('Company', '273').fetch(). This query should return an iterator with various entities.

Is this a potential bug with gcloud or with AppEngine? @lexsf @vliang63

[dhermes]

@pcostell WDYT? Seems it may be related to #1639

[dhermes]

@nlokare Can we try to throw together a reproducible example? You're inserting the entities from gcloud-python and then the GAE / ndb queries are failing, yes?

[nlokare]

@dhermes Exactly.

On GCE

account_key = datastore.key("Company", account_id)
...
advertiser['client_key'] = account_key
datastore.put(advertiser)

On GAE

def return_all_advertisers(self, account_id):
   all_advertisers = Advertiser.query(Advertiser.client_key == ndb.Key('Company', account_id)).order(Advertiser.advertiser_name).fetch()

[nlokare]

@dhermes Just to clarify, my queries work on GAE when developing locally because we seed our datastore directly from GAE. Alternatively, in production/staging we are populating our datastore from GCE. Our GAE application is essentially only reading from the datastore.

[pcostell]

Sounds like the same issue as #1639. Same as that issue, I think this will be fixed in v1beta3 if we always populate the project in 'gcloud-python'.

@nlokare Can you try again but with something like this on GCE:

account_key = datastore.key("Company", account_id, project_id="<your-application-id>")
...
advertiser['client_key'] = account_key

Where the application contains your project ID but with the extra information (likely s~).

[nlokare]

@pcostell I will try this. However, if my GAE ndb.Key object doesn't contain the project_id keyword argument, I think that this will still fail, since the creation of the entity from GCE added the extra data to the Key that wasn't implicit in the GAE query. Does that make sense?

In the pseudo-code above, I initialized the datastore object with our project. I left that out in the code snippet.

[pcostell]

How do you print the key? Can you try printing key.app() directly? ndb.Key's default __repr__ won't print the application ID if it matches the current app.

[nlokare]

From GAE I print the key doing:

all_advertisers = Advertiser.query().order(Advertiser.advertiser_name).fetch(15)
advert_tester = all_advertisers[0]
test_key = advert_tester.client_key
logging.info(test_key)

This is how we discovered that the Entity was saved from GCE with the client_key property as Key('Company', '273', app='random_project_id')

[dhermes]

OK I just confirmed:

from gcloud import datastore

project = 'MY-PROJ'
client1 = datastore.Client(project=project)
entity1 = datastore.Entity(key=client1.key('Bar', 1))
entity1['foo'] = client1.key('Foo', 1234)
client1.put(entity1)
client2 = datastore.Client(project='s~' + project)
entity2 = datastore.Entity(key=client2.key('Bar', 2))
entity2['foo'] = client2.key('Foo', 1234)
client2.put(entity2)

and then from the remote API

$ ~/google-cloud-sdk/platform/google_appengine/remote_api_shell.py MY-PROJ
App Engine remote_api shell
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2]
The db, ndb, users, urlfetch, and memcache modules are imported.
s~MY-PROJ> from google.appengine.ext import ndb
s~MY-PROJ> class Foo(ndb.Model):
...            pass
...
s~MY-PROJ> class Bar(ndb.Model):
...            foo = ndb.KeyProperty(kind='Foo')
...
s~MY-PROJ> q = Bar.query(Bar.foo == ndb.Key(Foo, 1234))
s~MY-PROJ> q.fetch()
[Bar(key=Key('Bar', 2), foo=Key('Foo', 1234))]

[nlokare]

@dhermes So, adding the s~ before the project when initializing datastore.Client will solve this?

[dhermes]

Yup, that's what @pcostell mentioned above:

with the extra information (likely s~).

[nlokare]

@dhermes @pcostell Ok, great. I will try this and report back. Thanks for help!

[dhermes]

You could also use find_true_project but that all is moot in v1beta3.

[nlokare]

@dhermes @pcostell Reporting back. Looks like this was the fix we needed. Thank you gentlemen.

[dhermes]

Sounds good. Closing this for same reason we closed #1639.