NDB Compressed properties compatibility

[cpavon]

Using NDB you can define entity properties as compressed and its value will be stored as a blob result of the compression (zlib) of the original value.

I tried to emulate this behavior using google-cloud-datastore library and I inserted a value (blob) exactly equal that the one that NDB inserts.

The first row has been inserted using NDB, the second one using cloud datastore library.

The problem is that NDB is not be able to detect the second row as a compressed value and when I read it returns the compressed value... not the uncompressed one as expected to be.

How do I have to insert a "NDB compressed value" using the API and make it readable using NDB?

Trying to figure out how NDB does, it seems that when a value is compressed it set a "meaning_uri" param. Probably we need a way to set that meaning also using the API.

Thanks.

[dhermes]

@cpavon I'm pretty sure ndb sets a meaning on the property, which is a "bygone" feature of the datastore API (i.e. not officially supported). Can you retrieve the raw protobuf contents of each property and let me know the difference (I think it'd be faster than me deploying a GAE app)?

>>> from google.cloud import datastore
>>>
>>> client = datastore.Client()
>>> key1 = client.key('Foo', 'Bar')
>>> key2 = client.key('Foo', 'Baz')
>>>
>>> key_pbs = [key1.to_protobuf(), key2.to_protobuf()]
>>> lookup_response = client._datastore_api.lookup(
...     client.project, read_options=None, keys=key_pbs)
>>> lookup_response

[cpavon]

Yep, you're right, the value stored using NDB arrives with meaning "22".

 properties {
      key: "name_compressed"
      value {
        meaning: 22
        blob_value: "x\234+I-.\001\000\004]\001\301"
        exclude_from_indexes: true
      }
    }

And everything works fine if I set the proper meanings before the put operation using the API:

datastore_entity._meanings = {'name_compressed': (22, datastore_entity.get('name_compressed'))}
client.put(datastore_entity)

If I save the entity in that way NDB and API are be able to read it without problems.

Is that the right way to set the meanings?

[dhermes]

We don't "officially" support meanings. The non-public attribute _meanings is there so that we don't wipe meanings when put-ing an existing entity that has been updated.

Though we don't "officially" support it, you have identified a working code path so feel free to use it. The fact that _meanings is non-public means we may remove, rename or relocate it at any time.

I'm going to close this issue out, let us know if you need anything else?