Firestore: invalid metadata error with grpcio 1.8.0 works with 1.7.3

[szferi]

The last update of grpcio 1.8.0 makes the firestore service fail with the following trace:

E1213 10:00:33.957789953    5598 ev_epollex_linux.cc:1482]   Skipping epollex becuase GRPC_LINUX_EPOLL is not defined.
E1213 10:00:33.957804187    5598 ev_epoll1_linux.cc:1261]    Skipping epoll1 becuase GRPC_LINUX_EPOLL is not defined.
E1213 10:00:33.957817171    5598 ev_epollsig_linux.cc:1761]  Skipping epollsig becuase GRPC_LINUX_EPOLL is not defined.
TypeError: Expected str, got unicode
Exception TypeError: 'Expected str, got unicode' in 'grpc._cython.cygrpc._store_c_metadata' ignored
E1213 10:00:33.959709563    5598 call.cc:1001]               validate_metadata: {"created":"@1513155633.959693129","description":"Metadata keys cannot be zero length","file":"src/core/lib/surface/validate_metadata.cc","file_line":66}
Traceback (most recent call last):
  File "t.py", line 11, in <module>
    db.collection('foo').document('foo').set({'foo': 'bar'})
  File "/home/szferi/envs/navega2/local/lib/python2.7/site-packages/google/cloud/firestore_v1beta1/document.py", line 224, in set
    write_results = batch.commit()
  File "/home/szferi/envs/navega2/local/lib/python2.7/site-packages/google/cloud/firestore_v1beta1/batch.py", line 135, in commit
    transaction=None, options=self._client._call_options)
  File "/home/szferi/envs/navega2/local/lib/python2.7/site-packages/google/cloud/firestore_v1beta1/gapic/firestore_client.py", line 851, in commit
    return self._commit(request, options)
  File "/home/szferi/envs/navega2/local/lib/python2.7/site-packages/google/gax/api_callable.py", line 452, in inner
    return api_caller(api_call, this_settings, request)
  File "/home/szferi/envs/navega2/local/lib/python2.7/site-packages/google/gax/api_callable.py", line 438, in base_caller
    return api_call(*args)
  File "/home/szferi/envs/navega2/local/lib/python2.7/site-packages/google/gax/api_callable.py", line 376, in inner
    return a_func(*args, **kwargs)
  File "/home/szferi/envs/navega2/local/lib/python2.7/site-packages/google/gax/retry.py", line 68, in inner
    return a_func(*updated_args, **kwargs)
  File "/home/szferi/envs/navega2/local/lib/python2.7/site-packages/grpc/_channel.py", line 484, in __call__
    credentials)
  File "/home/szferi/envs/navega2/local/lib/python2.7/site-packages/grpc/_channel.py", line 477, in _blocking
    _check_call_error(call_error, metadata)
  File "/home/szferi/envs/navega2/local/lib/python2.7/site-packages/grpc/_channel.py", line 86, in _check_call_error
    raise ValueError('metadata was invalid: %s' % metadata)
ValueError: metadata was invalid: [('google-cloud-resource-prefix', u'projects/platform-dev-187006/databases/(default)'), (u'x-goog-api-client', 'gl-python/2.7.13 gccl/0.28.0 gapic/0.28.0 gax/0.15.16 grpc/1.8.0')]

If I downgrade it to 1.7.3 it works just fine.

OS type and version

Linux, Ubuntu 17.04

Python version and virtual environment information python --version

Python 2.7.13

Relevant package versions

google-api-core==0.1.2
google-auth==1.2.1
google-cloud-core==0.28.0
google-cloud-firestore==0.28.0
google-cloud-storage==1.6.0
google-gax==0.15.16
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3
grpcio==1.8.0
firebase-admin==2.6.0

Code example

import firebase_admin
from firebase_admin import credentials
from firebase_admin import firestore

fb_cred = credentials.Certificate('./platform-dev-service-account.json')    
firebase_admin.initialize_app(fb_cred)
db = firestore.client()
db.collection('foo').document('foo').set({'foo': 'bar'})

[cpavon]

Same problem using google-cloud-datastore.
Downgrade to 1.7.3 also works.

[bcoghe]

same problem, same solution

[kmaillet]

Same here with google-cloud-datastore. Same fix

[igorpeshansky]

This is an issue with api_core/core, and seems to have been triggered by #4280 allowing a broken dependency in.

[dhermes]

@igorpeshansky Not quite. It's caused by the 1.8.0 release of grpcio breaking the way it accepts request metadata (which amounts to the equivalent of an HTTP header). The reason it just cropped up is because grpcio==1.8.0 was released last night.

[bmoyles0117]

Will that change be rolled back as a result? Or is this intended behavior?

[igorpeshansky]

@dhermes Right, 1.8.0 was the broken dependency... Can we restrict the version at 1.7.3 as a temporary fix, while grpcio gets their fix out?

[dhermes]

@igorpeshansky That won't help the already released versions, but depending on how difficult the fix is, that may be the best choice. I'm at lunch right now but hope to be able to address this soon.

[r00tat]

I've also opened a issue on the grpc github project before I found this issue. According to the issue #13752 this are the relevant code changes

[r00tat]

grpcio 1.8.0 is temporarily disabled on PyPI

[dhermes]

I am going to close this since 1.8.1 reverted the problematic change.