Adding one-time RPC to find unaliased / true dataset ID.

[dhermes]

Also removing unnecessary functions that are no longer
needed since there is no need to muck with the dataset ID.

@tseaver This is really an exploratory PR, no need to review just yet.

@pcostell Can you check out the _find_true_dataset_id method in gcloud/datastore/__init__.py (feel free to ignore the rest).

Is this a dumb idea? Is there an actual API available to convert from the "human friendly" ID foo to the unaliased / prefixed s~foo?

[coveralls]

Coverage remained the same when pulling 78bd1a2 on dhermes:non-fuzzy-app-id into 0525530 on GoogleCloudPlatform:master.

[coveralls]

Coverage remained the same when pulling e50b270 on dhermes:non-fuzzy-app-id into 0525530 on GoogleCloudPlatform:master.

[pcostell]

Pretty soon the datastore will normalize all types of keys, so you should never need to provide anything except the project id. Importantly, _find_true_dataset_id will only work with the current dataset which means it will break for users that store foreign keys.

In the near-future we don't want anyone to have to deal with the "s~", it should be purely an implementation detail of the Datastore. I think the client libraries should probably be written pretending the "s~" doesn't exist since it is cleaner that way (with the caveat that it is slightly broken atm).

[dhermes]

Thanks for the quick followup. That is great news! It will certainly be much cleaner if we can ignore s~. All the deleted code in this PR will still be able to be deleted even without _find_true_dataset_id.

How do you mean normalize? Currently it seems some normalization is happening, but that means if I send no dataset ID in the request and foo as the dataset ID in the URI, then I get back s~foo on the keys in the response. This still causes some issues / confusion / surprise.

[pcostell]

I believe the goal is that everywhere you will specify foo or an empty dataset and when you get the key back you will also only see foo in the dataset.

@eddavisson can you confirm?

[eddavisson]

That's correct. Note that if you want to do a one-time RPC in the mean time, you can do a Lookup request and check both the found and missing fields in the response for the resolved dataset id (basically what you're doing in this pull request but it's a read-only RPC).

[dhermes]

I didn't think to use missing! Good call. With the __key__ projection that'd be cheap enough, even if somehow the user had stored Key('__Foo', 1).

UPDATE: DERP, no projection on lookup.

[silvolu]

I think we should go forward with @eddavisson suggestion while waiting for the API updates to land.

[coveralls]

Coverage remained the same when pulling 3184184 on dhermes:non-fuzzy-app-id into 0525530 on GoogleCloudPlatform:master.

[dhermes]

@pcostell RE: foreign keys. I'm not sure why that's an issue if the application is managing those keys.

I double checked and the backend does not change the foreign dataset ID after storing (seems it shouldn't, for privacy reasons). From 0525530:

>>> from gcloud import datastore
>>> from gcloud.datastore import _implicit_environ
>>> from gcloud.datastore.entity import Entity
>>> from gcloud.datastore.key import Key
>>>
>>> datastore.set_default_connection()
>>> dataset_id = 'foo'
>>> datastore.set_default_dataset_id(dataset_id)
>>>
>>> e = Entity(key=Key('Foo', 1))
>>> e['bar'] = Key('Baz', 1, dataset_id='foreignappid')
>>> e.save()
>>>
>>> e_retrieved = datastore.get(e.key)
>>> e_retrieved['bar'].dataset_id
u'foreignappid'
>>>
>>> entity_pb, = _implicit_environ.CONNECTION.lookup(
...     dataset_id=dataset_id,
...     key_pbs=[e.key.to_protobuf()],
... )
>>> entity_pb.property
[<gcloud.datastore.datastore_v1_pb2.Property object at 0x7f4a5006d398>]
>>> print entity_pb.property[0].value.key_value
partition_id {
  dataset_id: "foreignappid"
}
path_element {
  kind: "Baz"
  id: 1
}

>>> e.key.delete()

FWIW I tried this both with a foreign dataset that my (user, not service account) OAuth 2.0 token had access to and one that the token did not.

[pcostell]

I believe we want to remove the notion of "s~" completely. So anywhere the user interacts with a dataset it should never have this extra prefix, including foreign keys.

Note that this is not implemented, so it isn't finalized and definitely won't work yet.

[coveralls]

Coverage remained the same when pulling 0cf4333 on dhermes:non-fuzzy-app-id into dcb73d3 on GoogleCloudPlatform:master.

[dhermes]

Had to rebase after #548 and #550, so did the renames in each commit through the rebase.

[coveralls]

Changes Unknown when pulling 31fd6fd on dhermes:non-fuzzy-app-id into * on GoogleCloudPlatform:master*.

[tseaver]

LGTM.

[dhermes]

@tseaver We can't / shouldn't merge this until the discussion above is resolved.

[coveralls]

Coverage remained the same when pulling 5d631a6 on dhermes:non-fuzzy-app-id into 55285f9 on GoogleCloudPlatform:master.

[tseaver]

Until this lands, or the back-end quits returning keys with s~ prefixes, I can't DTRT in #552 (fixing #447), unless I re-inject the "unmangling" logic we've already cleaned out elsewhere.

[coveralls]

Coverage remained the same when pulling 71a3dbe on dhermes:non-fuzzy-app-id into edfd5e2 on GoogleCloudPlatform:master.

[tseaver]

Ping me if more review is needed.

[dhermes]

Rebased and squashed into single commit (had to hack around after rebasing).

[coveralls]

Changes Unknown when pulling 0c4f4c6 on dhermes:non-fuzzy-app-id into * on GoogleCloudPlatform:master*.

[coveralls]

Coverage remained the same at 100.0% when pulling 9b0f87a on dhermes:non-fuzzy-app-id into 9518db7 on GoogleCloudPlatform:master.

[dhermes]

I'm closing this (since it is so out of date) and moving to discussion / tracking in #821.

Part of it was merged in #792 as well.