Attribute error thrown in gcloud/exceptions.py

[craigloftus]

If I use an incorrect dataset_id I get an AttributeError when I try to make an API call, because payload in the exception is set to b'Backend Error'.

gcloud/exceptions.py in make_exception(response, content, use_json)
186
--> 187 message = payload.get('message', message)
188 errors = payload.get('error', {}).get('errors', ())
189

AttributeError: 'bytes' object has no attribute 'get'

This is with gcloud==0.4.1 on Python 3.4.2.

Code to reproduce:

from gcloud import datastore
datastore.set_defaults(dataset_id=123456789)
query = datastore.Query(kind="SomeKind")
list(query.fetch())

Expected behaviour would be for a helpfully worded exception to be thrown.

[dhermes]

Thanks for the report. I just "fixed" make_exception and it seems I broke it in some other way. Looking into it.

[dhermes]

Ahh I see now, it's not make_exception that's broken, you just expect better errors when using the "wrong" dataset ID.

>>> list(query.fetch())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "gcloud/datastore/query.py", line 420, in __iter__
    self.next_page()
  File "gcloud/datastore/query.py", line 387, in next_page
    transaction_id=transaction and transaction.id,
  File "gcloud/datastore/connection.py", line 246, in run_query
    datastore_pb.RunQueryResponse)
  File "gcloud/datastore/connection.py", line 93, in _rpc
    data=request_pb.SerializeToString())
  File "gcloud/datastore/connection.py", line 70, in _request
    raise make_exception(headers, content, use_json=False)
gcloud.exceptions.ServiceUnavailable: 503 Backend Error

---

The answer is that your dataset ID is not your numeric project ID, but is instead a word or phrase. The following are actually equivalent

datastore.set_defaults(dataset_id=123456789)
datastore.set_defaults(dataset_id='123456789')

Using your project ID is just like using FOOBARAZ as your dataset ID, and the backend returns a 503.

Maybe we should file a Cloud Datastore issue?

---

I'm closing this out since the issue is not the library, but either a) the incorrect dataset ID or b) the insufficient information returned in the backend response.

Please re-open if you'd like to discuss more. I'm happy to brainstorm on this.

[craigloftus]

@dhermes It is a 'real' AttributeError that I am seeing. If you're not able to reproduce it, it is probably down to my environment.

I had to jump and hack through some hoops to use a Python 3 only library I needed for a prototype.

[dhermes]

Can you send the stack trace? Also, maybe the output of env from the command line (at least what you are willing to send)?

[dhermes]

@craigloftus I have been trying to reproduce and got

AccessTokenRefreshError: invalid_grant

which is due to the issue fixed in googleapis/oauth2client#126

After installed from HEAD in oauth2client and using protobuf==3.0.0-alpha-1:

>>> from gcloud import datastore
>>> datastore.set_defaults(dataset_id=123456789)
>>> k = datastore.Key('Kind', 10)
>>> datastore.get([k])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/some/path/gcloud-python/gcloud/datastore/api.py", line 211, in get
    key_pbs=[k.to_protobuf() for k in keys],
  File "/some/path/gcloud-python/gcloud/datastore/api.py", line 211, in <listcomp>
    key_pbs=[k.to_protobuf() for k in keys],
  File "/some/path/gcloud-python/gcloud/datastore/key.py", line 250, in to_protobuf
    key.partition_id.dataset_id = self.dataset_id
  File "/usr/local/lib/python3.4/dist-packages/google/protobuf/internal/python_message.py", line 488, in field_setter
    self._fields[field] = type_checker.CheckValue(new_value)
  File "/usr/local/lib/python3.4/dist-packages/google/protobuf/internal/type_checkers.py", line 155, in CheckValue
    raise TypeError(message)
TypeError: 123456789 has type <class 'int'>, but expected one of: (<class 'bytes'>, <class 'str'>)

From HEAD in oauth2client, using a valid ID:

>>> from gcloud import datastore
>>> datastore.set_defaults()
>>> k = datastore.Key('Kind', 10)
>>> datastore.get([k])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/some/path/gcloud/datastore/api.py", line 214, in get
    transaction_id=transaction and transaction.id,
  File "/some/path/gcloud/datastore/api.py", line 144, in _extended_lookup
    transaction_id=transaction_id,
  File "/some/path/gcloud/datastore/connection.py", line 189, in lookup
    datastore_pb.LookupResponse)
  File "/some/path/gcloud/datastore/connection.py", line 109, in _rpc
    data=request_pb.SerializeToString())
  File "/some/path/gcloud/datastore/connection.py", line 82, in _request
    method='POST', headers=headers, body=data)
  File "/usr/local/lib/python3.4/dist-packages/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/oauth2client/client.py", line 551, in new_request
    redirections, connection_type)
  File "/usr/local/lib/python3.4/dist-packages/httplib2/__init__.py", line 1139, in request
    headers = self._normalize_headers(headers)
  File "/usr/local/lib/python3.4/dist-packages/httplib2/__init__.py", line 1107, in _normalize_headers
    return _normalize_headers(headers)
  File "/usr/local/lib/python3.4/dist-packages/httplib2/__init__.py", line 195, in _normalize_headers
    return dict([ (key.lower(), NORMALIZE_SPACE.sub(value, ' ').strip())  for (key, value) in headers.items()])
  File "/usr/local/lib/python3.4/dist-packages/httplib2/__init__.py", line 195, in <listcomp>
    return dict([ (key.lower(), NORMALIZE_SPACE.sub(value, ' ').strip())  for (key, value) in headers.items()])
TypeError: sequence item 0: expected str instance, bytes found

[dhermes]

I'm closing this out since it's being fixed upstream in httplib2 and oauth2client:
googleapis/oauth2client#104
googleapis/oauth2client#136
jcgregorio/httplib2#291
jcgregorio/httplib2#296