[PubSub] Subscription.pull() not working (returns deadline exceeded)

[mykolasmith]

Any call pull() call to a PubSub subscription fails. There are about 1500 unacknowledged messages in the subscription in question. Note: I frequently switch this subscription between push and pull for draining and debugging purposes. (It would be nice to be able to peek messages and get message statistics by key without having to do this!)

Subscription details:

• Delivery type: pull
• Acknowledgement deadline: 600

Python 2.7.12
Relevant installed libraries:

• proto-google-cloud-pubsub-v1 (0.15.4)
• protobuf (3.2.0)
• gapic-google-cloud-pubsub-v1 (0.15.4)
• google-auth (0.10.0)
• google-auth-httplib2 (0.0.2)
• google-cloud-core (0.24.0)
• google-cloud-pubsub (0.24.0)
• google-gax (0.15.8)
• googleapis-common-protos (1.5.2)
• grpc-google-iam-v1 (0.11.1)
• grpcio (1.2.1)

---

After 600 seconds...

With GRPC:

Traceback (most recent call last):
  File "cli.py", line 21, in <module>
    rejected = repo.get_rejected_messages(subscription)
  File "/Users/mykola/src/pubsub/repository.py", line 33, in get_rejected_messages
    messages = subscription.pull(max_messages=1000, return_immediately=True)
  File "/Users/mykola/venv/lib/python2.7/site-packages/google/cloud/pubsub/subscription.py", line 332, in pull
    self.full_name, return_immediately, max_messages)
  File "/Users/mykola/venv/lib/python2.7/site-packages/google/cloud/pubsub/_gax.py", line 418, in subscription_pull
    return_immediately=return_immediately)
  File "/Users/mykola/venv/lib/python2.7/site-packages/google/cloud/gapic/pubsub/v1/subscriber_client.py", line 706, in pull
    return self._pull(request, options)
  File "/Users/mykola/venv/lib/python2.7/site-packages/google/gax/api_callable.py", line 419, in inner
    return api_caller(api_call, this_settings, request)
  File "/Users/mykola/venv/lib/python2.7/site-packages/google/gax/api_callable.py", line 407, in base_caller
    return api_call(*args)
  File "/Users/mykola/venv/lib/python2.7/site-packages/google/gax/api_callable.py", line 368, in inner
    return a_func(*args, **kwargs)
  File "/Users/mykola/venv/lib/python2.7/site-packages/google/gax/retry.py", line 143, in inner
    raise exc
google.gax.errors.RetryError: GaxError(Retry total timeout exceeded with exception, caused by <_Rendezvous of RPC that terminated with (StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded)>)

Without GRPC:

Traceback (most recent call last):
  File "cli.py", line 21, in <module>
    rejected = repo.get_rejected_messages(subscription)
  File "/Users/mykola/src/pubsub/repository.py", line 33, in get_rejected_messages
    messages = subscription.pull(max_messages=1000, return_immediately=True)
  File "/Users/mykola/venv/lib/python2.7/site-packages/google/cloud/pubsub/subscription.py", line 332, in pull
    self.full_name, return_immediately, max_messages)
  File "/Users/mykola/venv/lib/python2.7/site-packages/google/cloud/pubsub/_http.py", line 419, in subscription_pull
    response = self.api_request(method='POST', path=path, data=data)
  File "/Users/mykola/venv/lib/python2.7/site-packages/google/cloud/_http.py", line 303, in api_request
    error_info=method + ' ' + url)
google.cloud.exceptions.GatewayTimeout: 504 The service was unable to fulfill your request. Please try again. [code=8a75] (POST https://pubsub.googleapis.com/v1/projects/[redacted]/subscriptions/[redacted]:pull)

[mykolasmith]

May be similar to: #2993

[dhermes]

@mykolasmith Is this issue limited to the Python client or is it an issue with Pub / Sub itself? Can you get these requests to succeed elsewhere, e.g. in the APIs explorer?

[mykolasmith]

FYI, it seems like an issue with Pub/Sub and not this specific client library. Per Google Support:

Our Engineering team is actively looking into this issue right now. They've confirmed that there is indeed an issue on our side that caused this issue (which was triggered by you switching from pull to push and then back to push on that subscription). A colleague of mine will be further in touch as soon as there is further update (and, hopefully, a solution or a workaround).

[dhermes]

Thanks for the update @mykolasmith and sorry we can't be of more help.

[binodmanandhar]

Hi mykolasmith,
I have using php library google/pubsub to push the notification
I had create topic via console, and then set up a subscription on the topic with delivery Type as push. Then, I have setup up the endpoint url(https with POST). When every I try to publish the topic, I am getting 404 Not found error. What could the issue?
^ """

    <title>Not Found</title>

    <!-- Fonts -->
    <link rel="dns-prefetch" href="//fonts.gstatic.com">
    <link href="https://fonts.googleapis.com/css?family=Nunito" rel="stylesheet">

    <!-- Styles -->
    <style>
        html, body {
            background-color: #fff;
            color: #636b6f;
            font-family: 'Nunito', sans-serif;
            font-weight: 100;
            height: 100vh;
            margin: 0;
        }

        .full-height {
            height: 100vh;
        }

        .flex-center {
            align-items: center;
            display: flex;
            justify-content: center;
        }

        .position-ref {
            position: relative;
        }

        .code {
            border-right: 2px solid;
            font-size: 26px;
            padding: 0 15px 0 15px;
            text-align: center;
        }

        .message {
            font-size: 18px;
            text-align: center;
        }
    </style>
</head>
<body>
    <div class="flex-center position-ref full-height">
        <div class="code">
            404            </div>

        <div class="message" style="padding: 10px;">
            Not Found            </div>
    </div>
</body>