Datastore access is slow

[Paxa]

Access to datastore is slow, and even more slower when run on cloud functions.

Code example:

const Datastore = require('@google-cloud/datastore');

const ds = exports.ds = Datastore({
  projectId: 'my-test',
  keyFilename: './my-test-key.json'
});

console.time('query');
var query = ds.createQuery(['Cat']);
ds.runQuery(query, (error, entities, nextQuery) => {
  console.timeEnd('query');

  console.log(entities.length);
  console.log(entities);
});

Output:

query: 1156.979ms
8
[ { time: 2017-05-05T08:51:51.822Z,
    age: 123,
    name: 'miu',
    color: 'black' },
  { age: 123,
    name: 'miu',
    color: 'black',
    time: 2017-05-05T08:53:26.497Z },
... // 8 same objects

I ran many times, it's always 1.0 - 1.8 sec

Same code in ruby run in 0.7 seconds

---

Btw, how to use HTTP REST API for Datastore? I can't find how to insert or update data? Is there some example curl commands?

[stephenplusplus]

My experience testing locally using this same script:

1. 679.928ms
2. 577.103ms
3. 1505.129ms
4. 500.902ms
5. 520.056ms
6. 1796.278ms

I believe what this would tell us is that the code in this library is capable of running in the range of 500-2000ms for this type of command, but the API response time and network conditions will decide where in that range it falls.

Because of this, I don't think there is anything our library can do. If you feel the performance of Datastore, the service, is lacking, you can open an issue on the official tracker here and have a conversation with members of the Datastore team.

Btw, how to use HTTP REST API for Datastore? I can't find how to insert or update data? Is there some example curl commands?

If you want to use this API directly, you can see the documentation on the expected request and response objects here: https://cloud.google.com/datastore/docs/reference/rest/.

[Paxa]

Thank you
What is expected response time during perfect network condition?

I tried to access 1 record by Kind + ID via HTTP and it takes 700-2000 ms, is it normal?
I think it should be less then 30 ms, so people will consider to use it

[stephenplusplus]

Questions like those are better on the repo I linked above. We are just the client library that wraps the API. Consider that measuring the first request isn't always the best gauge of performance. There is the authentication step, where an access token is minted, which has to be run before the first API request. This step isn't repeated for future requests, which will show a better response time.

[Paxa]

Right, I didn't know that it makes multiple requests at first time.

Is there a way to save credentials to make first request faster?

I updated code:

const async = require('async');
const Datastore = require('@google-cloud/datastore');

const ds = exports.ds = Datastore({
  projectId: 'my-test',
  keyFilename: './my-test-key.json'
});

async.timesSeries(20, (n, next) => {
  console.time('query-' + n);
  var query = ds.createQuery(['Cat']);
  ds.runQuery(query, (error, entities, nextQuery) => {
    console.timeEnd('query-' + n);
    if (error) console.log(error);
    next(error, entities);
  });
}, function(err, cats) {
  console.log(cats.length);
});

Output:

query-0: 1823.483ms
query-1: 175.248ms
query-2: 339.068ms
query-3: 214.101ms
query-4: 687.255ms
query-5: 160.445ms
query-6: 456.044ms
query-7: 172.072ms
query-8: 563.079ms
query-9: 168.259ms
query-10: 411.999ms
query-11: 168.520ms
query-12: 564.410ms
query-13: 171.733ms
query-14: 850.954ms
query-15: 167.130ms
query-16: 861.601ms
query-17: 194.063ms
query-18: 849.674ms
query-19: 171.333ms
20

[stephenplusplus]

Is there a way to save credentials to make first request faster?

Not at this time. The access token management is done automatically, to handle auto-refreshing when it expires.

[charly37]

Same here. RT is very slow (too slow for our usecase to be usable sadly)