pubsub frequently raises a gax DEADLINE_EXCEEDED error

[theacodes]

Debian Jessie, Python 2.7 and 3.4

Stacktrace:

tests/test_storage.py:44: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.tox/py34/lib/python3.4/site-packages/werkzeug/test.py:788: in post
    return self.open(*args, **kw)
.tox/py34/lib/python3.4/site-packages/flask/testing.py:113: in open
    follow_redirects=follow_redirects)
.tox/py34/lib/python3.4/site-packages/werkzeug/test.py:751: in open
    response = self.run_wsgi_app(environ, buffered=buffered)
.tox/py34/lib/python3.4/site-packages/werkzeug/test.py:668: in run_wsgi_app
    rv = run_wsgi_app(self.application, environ, buffered=buffered)
.tox/py34/lib/python3.4/site-packages/werkzeug/test.py:871: in run_wsgi_app
    app_rv = app(environ, start_response)
.tox/py34/lib/python3.4/site-packages/flask/app.py:2000: in __call__
    return self.wsgi_app(environ, start_response)
.tox/py34/lib/python3.4/site-packages/flask/app.py:1991: in wsgi_app
    response = self.make_response(self.handle_exception(e))
.tox/py34/lib/python3.4/site-packages/flask/app.py:1567: in handle_exception
    reraise(exc_type, exc_value, tb)
.tox/py34/lib/python3.4/site-packages/flask/_compat.py:33: in reraise
    raise value
.tox/py34/lib/python3.4/site-packages/flask/app.py:1988: in wsgi_app
    response = self.full_dispatch_request()
.tox/py34/lib/python3.4/site-packages/flask/app.py:1641: in full_dispatch_request
    rv = self.handle_user_exception(e)
.tox/py34/lib/python3.4/site-packages/flask/app.py:1544: in handle_user_exception
    reraise(exc_type, exc_value, tb)
.tox/py34/lib/python3.4/site-packages/flask/_compat.py:33: in reraise
    raise value
.tox/py34/lib/python3.4/site-packages/flask/app.py:1639: in full_dispatch_request
    rv = self.dispatch_request()
.tox/py34/lib/python3.4/site-packages/flask/app.py:1625: in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
bookshelf/crud.py:99: in add
    q = tasks.get_books_queue()
bookshelf/tasks.py:34: in get_books_queue
    client, 'books', extra_context=current_app.app_context)
.tox/py34/lib/python3.4/site-packages/psq/queue.py:41: in __init__
    self.topic = self._get_or_create_topic()
.tox/py34/lib/python3.4/site-packages/psq/queue.py:51: in _get_or_create_topic
    if not topic.exists():
.tox/py34/lib/python3.4/site-packages/google/cloud/pubsub/topic.py:179: in exists
    api.topic_get(topic_path=self.full_name)
.tox/py34/lib/python3.4/site-packages/google/cloud/pubsub/_gax.py:126: in topic_get
    topic_pb = self._gax_api.get_topic(topic_path)
.tox/py34/lib/python3.4/site-packages/google/cloud/gapic/pubsub/v1/publisher_client.py:315: in get_topic
    return self._get_topic(request, options)
.tox/py34/lib/python3.4/site-packages/google/gax/api_callable.py:481: in inner
    return api_caller(api_call, this_settings, request)
.tox/py34/lib/python3.4/site-packages/google/gax/api_callable.py:469: in base_caller
    return api_call(*args)
.tox/py34/lib/python3.4/site-packages/google/gax/api_callable.py:434: in inner
    errors.create_error('RPC failed', cause=exception))
.tox/py34/lib/python3.4/site-packages/future/utils/__init__.py:419: in raise_with_traceback
    raise exc.with_traceback(traceback)
.tox/py34/lib/python3.4/site-packages/google/gax/api_callable.py:430: in inner
    return a_func(*args, **kwargs)
.tox/py34/lib/python3.4/site-packages/google/gax/api_callable.py:64: in inner
    return a_func(*updated_args, **kwargs)
.tox/py34/lib/python3.4/site-packages/grpc/_channel.py:481: in __call__
    return _end_unary_response_blocking(state, False, deadline)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

state = <grpc._channel._RPCState object at 0x7fb3c4117c88>, with_call = False
deadline = 1486607618.4266632

    def _end_unary_response_blocking(state, with_call, deadline):
      if state.code is grpc.StatusCode.OK:
        if with_call:
          rendezvous = _Rendezvous(state, None, None, deadline)
          return state.response, rendezvous
        else:
          return state.response
      else:
>       raise _Rendezvous(state, None, None, deadline)
E       google.gax.errors.GaxError: GaxError(RPC failed, caused by <_Rendezvous of RPC that terminated with (StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded)>)

.tox/py34/lib/python3.4/site-packages/grpc/_channel.py:432: GaxError

test code is here

[tcroiset]

similar to #2983

[lukesneeringer]

@daspecster Why did you close this? Is it fixed?

[daspecster]

@lukesneeringer it was closed via #3130. I thought they were duplicates/related but perhaps not?

I'll run @jonparrott's tests and try and verify real quick.

[lukesneeringer]

That sounds good to me. Just verifying.

[daspecster]

@lukesneeringer in talking to @jonparrott, it sounds like this is an upstream issue with the service.

[lukesneeringer]

I actually learned more about this in an internal meeting today. It sounds like it is an issue with the grpc client itself, and we are expecting a new release soon (as in, probably within the next week or two).

Derp. Wrong issue.

[lukesneeringer]

Hello! :-)
First, a mea culpa; I have not done as good of a job at keeping up with issues as I should have. If you are getting this (I admit it) cut and paste, it is likely because your issue sat for too long.

In this case, I have been in the process of making a radical update to the PubSub library (#3637) to add significant performance improvements and a new surface, which we hope to launch soon. As such, I am clearing out issues on the old library. It is my sincere goal to do a better job of being on top of issues in the future.

As the preceding paragraph implies, I am closing this issue. If the revamped library does not solve your issue, however, please feel free to reopen.

Thanks!