Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PickleProperty put in Python 2.7 GAE NDB can't be read by Cloud NDB on Python 3 if it contains certain types #909

Open
Tony-Phillips05 opened this issue Aug 25, 2023 · 1 comment
Labels
api: datastore Issues related to the googleapis/python-ndb API.

Comments

@Tony-Phillips05
Copy link

Tony-Phillips05 commented Aug 25, 2023

Discovered while migrating to python 3

if type(value) is bytes: # pragma: NO BRANCH
return pickle.loads(value, encoding="bytes")

Related to #595

The current PickleProperty referenced in the github code link above uses encoding="bytes" when unpickling. This does not work for all data types that was previously pickled with NDB in python 2.7. According to: This post, if you had pickled objects that contained datetime and some others, the only way to unpickle correctly is to use encoding="latin1"

We have internally fixed this with:

`class PicklePropertyFixed(model.BlobProperty):

def _to_base_type(self, value):
    return pickle.dumps(value, pickle.HIGHEST_PROTOCOL)

def _from_base_type(self, value):
    """Because the google.cloud.ndb.model.PickleProperty ONLY decodes using the
    'bytes' encoding, we must use this custom class to override this.
    The reason is because the encoding 'latin1' has to be used if the pickled object
    has instances of datetime, date, time, or numpy arrayswhich. Our model uses datetime
    """
    if type(value) is bytes:
        return pickle.loads(value, encoding="latin1")
    return pickle.loads(value)`
@product-auto-label product-auto-label bot added the api: datastore Issues related to the googleapis/python-ndb API. label Aug 25, 2023
@sorced-jim
Copy link
Contributor

Did you happen to try pickle.loads(value)? Based on https://github.com/GoogleCloudPlatform/appengine-python-standard/blob/main/src/google/appengine/ext/ndb/model.py#L1901-L1917, it looks like the pickle.loads() should be first with a fallback to setting the encoding explicitly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: datastore Issues related to the googleapis/python-ndb API.
Projects
None yet
Development

No branches or pull requests

2 participants