Logging stops when grpc call returns UNAVAILABLE

[flundermicha]

We experienced the following error:

Exception in thread google.cloud.logging.handlers.transport.Worker:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/google/gax/api_callable.py", line 430, in inner
return a_func(*args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/google/gax/api_callable.py", line 64, in inner
return a_func(*updated_args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/grpc/_channel.py", line 481, in __call__
return _end_unary_response_blocking(state, False, deadline)
File "/usr/local/lib/python3.5/site-packages/grpc/_channel.py", line 432, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, {"created":"@1478297096.016103965","description":"Secure read failed","file":"src/core/lib/security/transport/secure_endpoint.c","file_line":157,"grpc_status":14,"referenced_errors":[{"created":"@1478297096.016009469","description":"EOF","file":"src/core/lib/iomgr/tcp_posix.c","file_line":235}]})>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.5/site-packages/google/cloud/logging/handlers/transports/background_thread.py", line 80, in _run
self.batch.commit()    
File "/usr/local/lib/python3.5/site-packages/google/cloud/logging/logger.py", line 452, in commit
client.logging_api.write_entries(entries, **kwargs)
File "/usr/local/lib/python3.5/site-packages/google/cloud/logging/_gax.py", line 113, in write_entries
partial_success=partial_success, options=options)
File "/usr/local/lib/python3.5/site-packages/google/cloud/gapic/logging/v2/logging_service_v2_api.py", line 291, in write_log_entries
self._write_log_entries(request, options)
File "/usr/local/lib/python3.5/site-packages/google/gax/api_callable.py", line 481, in inner
return api_caller(api_call, this_settings, request)
File "/usr/local/lib/python3.5/site-packages/google/gax/api_callable.py", line 469, in base_caller
return api_call(*args)
File "/usr/local/lib/python3.5/site-packages/google/gax/api_callable.py", line 434, in inner
errors.create_error('RPC failed', cause=exception))
File "/usr/local/lib/python3.5/site-packages/future/utils/__init__.py", line 419, in raise_with_traceback
raise exc.with_traceback(traceback)
File "/usr/local/lib/python3.5/site-packages/google/gax/api_callable.py", line 430, in inner
return a_func(*args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/google/gax/api_callable.py", line 64, in inner
return a_func(*updated_args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/grpc/_channel.py", line 481, in __call__
return _end_unary_response_blocking(state, False, deadline)
File "/usr/local/lib/python3.5/site-packages/grpc/_channel.py", line 432, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
google.gax.errors.GaxError: GaxError(RPC failed, caused by <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, {"created":"@1478297096.016103965","description":"Secure read failed","file":"src/core/lib/security/transport/secure_endpoint.c","file_line":157,"grpc_status":14,"referenced_errors":[{"created":"@1478297096.016009469","description":"EOF","file":"src/core/lib/iomgr/tcp_posix.c","file_line":235}]})>)

We are using the following python modules:

gapic-google-logging-v2 (0.10.1)
gapic-google-pubsub-v1 (0.10.1)
google-cloud (0.20.0)
google-cloud-bigquery (0.20.0)
google-cloud-bigtable (0.20.0)
google-cloud-core (0.20.0)
google-cloud-datastore (0.20.1)
google-cloud-dns (0.20.0)
google-cloud-error-reporting (0.20.0)
google-cloud-language (0.20.0)
google-cloud-logging (0.20.0)
google-cloud-monitoring (0.20.0)
google-cloud-pubsub (0.20.0)
google-cloud-resource-manager (0.20.0)
google-cloud-storage (0.20.0)
google-cloud-translate (0.20.0)
google-cloud-vision (0.20.0)
google-gax (0.14.1)
googleapis-common-protos (1.4.0)
grpc-google-iam-v1 (0.10.1)
grpc-google-logging-v2 (0.10.1)
grpc-google-pubsub-v1 (0.10.1)
grpcio (1.0.1)

Proposed solution: Whenever a gRPC call fails with StatusCode.UNAVAILABLE the gRPC call should be retried with a backof.

[dhermes]

Closing as a dupe of #2683