Cannot do parallelized "X is one of [A,B,C]" queries

[mikelambert]

The old DB and NDB libraries offered a way to query a field as a member of a list:

CityTable.query(CityTable.city.IN(city_names)).fetch(100)

I think it should be possible to do this parallelized query in google.cloud.datastore as well.

The google.cloud.datastore offers no async API, and so can be an order of magnitude slower when manually stitching together this query, which makes it unusable for my purposes as a replacement for NDB. (I am getting off NDB as it is currently a requirement for getting off python-compat onto a generic python runtime.)

The async API bug mentions that an async API is not needed "because NDB exists", which doesn't make sense to me because NDB is more than an async API...its also an ORM and an eventloop library and more.

And even the NDB bug talks about how NDB positions as a complex ORM layer vs the simple key-value store of google.cloud.datastore.

I would like the simple key-value approach combined with a barebones async API, that one could use to implement a "where field is in list" query like the old DB and NDB libraries supported.

[dhermes]

IIUC

CityTable.city.IN(city_names)

is "more efficient" because IN is supported directly by the RPC API that App Engine has access to, not because of async. @jonparrott can you confirm this?

As for a generic async API, that seems to be a distinct issue than "it would be nice to have an IN query".

We used to have #557 as a target for just supporting ndb but that is no longer true.

[mikelambert]

Ahh I was not aware that there was special support for IN queries (I know that under the hood, they still counted as multiple queries).

Yes, async API is broader than supporting the IN API. There was an open issue for an async API, but you closed it in favor of supporting NDB. I was trying to present an argument here that an async API would be useful to get feature-parity with DB, outside of the whole NDB debacle.

(Not sure exactly what part you are referring to is "no longer true", though? You no longer plan to support ndb in google.cloud.datastore? You will no longer plan to support async in google.cloud.datastore?)

[theacodes]

ndb supports IN because behind the scenes it starts an individual query for each item in the clause and combines them. It's possible to emulate that behavior here by using izip_longest.

[dhermes]

@jonparrott

• Then the real issue would be sending multiple queries simultaneously (i.e. async) as noted by @mikelambert
• I'm not sure why we would be zip-ing the results, instead we'd be adding them to a set-like object which de-duplicates the result

[mikelambert]

And just to be clear, both appengine's NDB and DB libraries support IN queries. For example, this is a DB query from my code:

City.gql("where geohashes in :geohashes", geohashes=geohashes).fetch(1000)

[chemelnucfin]

Hello, feature requests will now be tracked in the project Feature Requests. I will close this issue now, please feel free to continue to address any issues/concerns here.

[dannymilsom]

+1 for adding an async API interface to the google cloud datastore library. I agree with @mikelambert that it should not a requirement to use NDB to gain this async ability (as ndb comes with a number of other things which might be surplus to requirements - e.g. the ORM).

I think this is particularly important as async Python web frameworks are growing in popularity, and other well established frameworks are actively working to support ASGI (biggest example being Django 3). It should not be a pre-requisite for existing apps to migrate to NDB IMO to unlock this functionality.