Adding one-time RPC to find unaliased / true dataset ID.

[dhermes]

Also removing unnecessary functions that are no longer
needed since there is no need to muck with the dataset ID.

@tseaver This is really an exploratory PR, no need to review just yet.

@pcostell Can you check out the _find_true_dataset_id method in gcloud/datastore/__init__.py (feel free to ignore the rest).

Is this a dumb idea? Is there an actual API available to convert from the "human friendly" ID foo to the unaliased / prefixed s~foo?

[coveralls]

Coverage remained the same when pulling 78bd1a2 on dhermes:non-fuzzy-app-id into 0525530 on GoogleCloudPlatform:master.

[coveralls]

Coverage remained the same when pulling e50b270 on dhermes:non-fuzzy-app-id into 0525530 on GoogleCloudPlatform:master.

[pcostell]

Pretty soon the datastore will normalize all types of keys, so you should never need to provide anything except the project id. Importantly, _find_true_dataset_id will only work with the current dataset which means it will break for users that store foreign keys.

In the near-future we don't want anyone to have to deal with the "s~", it should be purely an implementation detail of the Datastore. I think the client libraries should probably be written pretending the "s~" doesn't exist since it is cleaner that way (with the caveat that it is slightly broken atm).

[dhermes]

Thanks for the quick followup. That is great news! It will certainly be much cleaner if we can ignore s~. All the deleted code in this PR will still be able to be deleted even without _find_true_dataset_id.

How do you mean normalize? Currently it seems some normalization is happening, but that means if I send no dataset ID in the request and foo as the dataset ID in the URI, then I get back s~foo on the keys in the response. This still causes some issues / confusion / surprise.

[pcostell]

I believe the goal is that everywhere you will specify foo or an empty dataset and when you get the key back you will also only see foo in the dataset.

@eddavisson can you confirm?

[eddavisson]

That's correct. Note that if you want to do a one-time RPC in the mean time, you can do a Lookup request and check both the found and missing fields in the response for the resolved dataset id (basically what you're doing in this pull request but it's a read-only RPC).

[dhermes]

I didn't think to use missing! Good call. With the __key__ projection that'd be cheap enough, even if somehow the user had stored Key('__Foo', 1).

UPDATE: DERP, no projection on lookup.

[silvolu]

I think we should go forward with @eddavisson suggestion while waiting for the API updates to land.

[coveralls]

Coverage remained the same when pulling 3184184 on dhermes:non-fuzzy-app-id into 0525530 on GoogleCloudPlatform:master.

[dhermes]

@pcostell RE: foreign keys. I'm not sure why that's an issue if the application is managing those keys.

I double checked and the backend does not change the foreign dataset ID after storing (seems it shouldn't, for privacy reasons). From 0525530:

>>>fromgcloudimportdatastore>>>fromgcloud.datastoreimport_implicit_environ>>>fromgcloud.datastore.entityimportEntity>>>fromgcloud.datastore.keyimportKey>>>>>>datastore.set_default_connection()
>>>dataset_id='foo'>>>datastore.set_default_dataset_id(dataset_id)
>>>>>>e=Entity(key=Key('Foo', 1))
>>>e['bar'] =Key('Baz', 1, dataset_id='foreignappid')
>>>e.save()
>>>>>>e_retrieved=datastore.get(e.key)
>>>e_retrieved['bar'].dataset_idu'foreignappid'>>>>>>entity_pb, =_implicit_environ.CONNECTION.lookup(
... dataset_id=dataset_id,
... key_pbs=[e.key.to_protobuf()],
... )
>>>entity_pb.property
[<gcloud.datastore.datastore_v1_pb2.Propertyobjectat0x7f4a5006d398>]
>>>printentity_pb.property[0].value.key_valuepartition_id {
dataset_id: "foreignappid"
}
path_element {
kind: "Baz"id: 1
}
>>>e.key.delete()

FWIW I tried this both with a foreign dataset that my (user, not service account) OAuth 2.0 token had access to and one that the token did not.

[pcostell]

I believe we want to remove the notion of "s~" completely. So anywhere the user interacts with a dataset it should never have this extra prefix, including foreign keys.

Note that this is not implemented, so it isn't finalized and definitely won't work yet.

[coveralls]

Coverage remained the same when pulling 0cf4333 on dhermes:non-fuzzy-app-id into dcb73d3 on GoogleCloudPlatform:master.

[dhermes]

Had to rebase after #548 and #550, so did the renames in each commit through the rebase.

[coveralls]

Changes Unknown when pulling 31fd6fd on dhermes:non-fuzzy-app-id into * on GoogleCloudPlatform:master*.

[tseaver]

LGTM.

[dhermes]

@tseaver We can't / shouldn't merge this until the discussion above is resolved.

[coveralls]

Coverage remained the same when pulling 5d631a6 on dhermes:non-fuzzy-app-id into 55285f9 on GoogleCloudPlatform:master.

[coveralls]

Coverage remained the same at 100.0% when pulling 9b0f87a on dhermes:non-fuzzy-app-id into 9518db7 on GoogleCloudPlatform:master.

[dhermes]

I'm closing this (since it is so out of date) and moving to discussion / tracking in #821.

Part of it was merged in #792 as well.