Datastore Query with Key filter failing on GCE

[ndenny]

Since I upgraded to 0.11.0, I'm seeing a problem with my production environment on GCE, but not when I run it locally.

I am trying to query the datastore, using a filter on a property that is a Key. I have example code below - the basic problem is that when using the filter, I get no results back, even when querying using a key I know exists.

Looking at the print statements, the difference between my local and GCE environments appears to be that the key I create myself has project=project_id whereas the query returns a key with project=s~project_id.

Environment: GCE, Python 2.7.9, gcloud 0.11.0 [following code asserts with no results returned from 2nd query]
Local: Mac OS X, Python 2.7.10, gcloud 0.11.0 [following code works]

from gcloud import datastore

def main():
    client = datastore.Client()

    last_result = None
    query = client.query(kind='Foo')
    results = query.fetch(1)
    for result in results:
        last_result = result
    assert last_result is not None

    query2 = client.query(kind='Foo')
    key = client.key('Bar', last_result['bar'].id)
    print key
    print last_result['bar']
    query2.add_filter('bar', '=', key)
    results = query2.fetch(1)
    assert len(list(results)) > 0

if __name__ == '__main__':
    main()

[dhermes]

Thanks for the report! I'm in an airport but will try to dig around

[dhermes]

I can't reproduce. I just ran this on GCE and it passed:

daniel_j_hermes@instance-1:~$ python test.py
<Key[{'kind': 'Bar', 'id': 1234L}], project=PROJ-NAME>
<Key[{'kind': u'Bar', 'id': 1234L}], project=PROJ-NAME>
daniel_j_hermes@instance-1:~$ pip show gcloud
---
Name: gcloud
Version: 0.11.0
Location: /usr/local/lib/python2.7/dist-packages
Requires: googleapis-common-protos, oauth2client, pyOpenSSL, httplib2, six, protobuf

Maybe you're having issues with eventual consistency?

[ndenny]

Sorry for the delay in replying - I don't think it's a consistency issue - the data I'm reading hasn't changed recently. I'm getting a repro 100% of the time here.
This is an old project in App Engine terms - over a couple of years - there seems to be some inconsistency between the project id (e.g. "PROJ-NAME") and the datastore id (e.g. "s~PROJ-NAME") and which one to use when running on GCE.

[dhermes]

@pcostell Any ideas?

[pcostell]

This likely is a key normalization issue (which gets fixed substantially with the next version of the API).

In Datastore v1beta2, we only normalize the datasetId if you leave it blank. I can't remember how much gcloud-python does in order to try to generate the "correct" dataset -- does client.Key(...) have a dataset == "s~PROJ" or dataset == "PROJ"?

Can you try writing your query without using gcloud-python's key? Instead of using the client, create a key directly:

key = datastore.Key('Bar', last_result['bar'].id, project=None)

[dhermes]

@pcostell gcloud-python doesn't make any effort to get it correct, but wipes the dataset from a key before sending in a request.

[pcostell]

Ah ok. But you don't wipe the dataset when someone uses a key as a query argument unless they query by __key__. So in this case, the query is for 'my-project-id', but the actual data is stored in Datastore as 's~my-project-id'.

Note though that you can't necessarily wipe the project in this case, as someone could also query for 'my-other-project-id'.

I'm not sure this is worth fixing now -- in v1beta3 you should always populate the project id and should never have to deal with dataset ids.

[dhermes]

Gotcher. Let me see if I can repro.

[ndenny]

Just to add, I made a temp fix by, when creating the Datastore client, using datastore.Client(project="s~project_id") as the constructor instead of leaving it blank.

[dhermes]

@ndenny That is the best move for now. You could also use find_true_project but that all is moot in v1beta3.

@pcostell I was able to reproduce and it was definitely the project in the key:

>>> from gcloud import datastore
>>> client = datastore.Client(project='explicit-project-name')
>>> # Make the keys.
>>> k1 = client.key('Foo', 1234)
>>> k1a = k1._clone()
>>> k1a._project = 's~' + k1a._project
>>> k2 = client.key('Bar', 1234)
>>> # Store the data to be queried.
>>> e = datastore.Entity(key=k2)
>>> e['foo'] = k1
>>> client.put(e)
>>> # Query without s~.
>>> q = client.query(kind='Bar')
>>> q.add_filter('foo', '=', k1)
>>> q_iter = q.fetch()
>>> entities, more_results, cursor = q_iter.next_page()
>>> print(len(entities))
1
>>> # Query with s~.
>>> q = client.query(kind='Bar')
>>> q.add_filter('foo', '=', k1a)
>>> q_iter = q.fetch()
>>> entities, more_results, cursor = q_iter.next_page()
>>> print(len(entities))
0

[pcostell]

I don't think we normalize projectId => applicationId in v1beta2. So when
you store e['foo'] = k1, we'll store the key with the project Id instead
of the application Id. If you had stored this in GAE or left the
datasetId/projectId blank, we would store it with the application Id (s~...)

On Mon, Mar 28, 2016 at 11:46 AM Danny Hermes notifications@github.com
wrote:

@ndenny https://github.com/ndenny That is the best move for now. You
could also use find_true_project
https://github.com/GoogleCloudPlatform/gcloud-python/blob/4ccb7a10829075192f844479b315df612136e05c/gcloud/datastore/helpers.py#L33
but that all is moot in v1beta3.

@pcostell https://github.com/pcostell I was able to reproduce and it
was definitely the project in the key:

from gcloud import datastore>>> client = datastore.Client(project='explicit-project-name')>>> # Make the keys.>>> k1 = client.key('Foo', 1234)>>> k1a = k1._clone()>>> k1a._project = 's~' + k1a._project>>> k2 = client.key('Bar', 1234)>>> # Store the data to be queried.>>> e = datastore.Entity(key=k2)>>> e['foo'] = k1>>> client.put(e)>>> # Query without s~.>>> q = client.query(kind='Bar')>>> q.add_filter('foo', '=', k1)>>> q_iter = q.fetch()>>> entities, more_results, cursor = q_iter.next_page()>>> print(len(entities))1>>> # Query with s~.>>> q = client.query(kind='Bar')>>> q.add_filter('foo', '=', k1a)>>> q_iter = q.fetch()>>> entities, more_results, cursor = q_iter.next_page()>>> print(len(entities))0

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#1639 (comment)

[ndenny]

That makes sense - I realize the key bit of information here is that the data I'm querying was created in GAE.

[dhermes]

@pcostell I'm closing since v1beta3 will fix this eventually and the amount of code required to fix would be quite a lot (i.e. emulating an OR query every time a key is used).