Calling `subscription.pull(return_immediately=True)` hangs

[GilJ]

I've got a fairly simple PubSub setup. The subscriber tries to pull from a topic with return_immediately set to True, and performs some actions based on the data that was returned. If there was no data, the consumer waits up to 5 seconds and tries again. In pseudocode:

client = pubsub.get_client()
topic = client.get_topic()
subscription = topic.get_subscription()
while True:
    data = subscription.pull(return_immediately=True)
    if data:
        do_work(data)
    else:
        sleep(5)

I've noticed that after running this for a while, my subscriber just hangs on the subscription.pull(return_immediately=True) call. I would expect that, if a network error occurs, the connection would time out and that the function call would result in an exception.

I've tried debugging this by setting the log level of the oauth2client and oauth2client.client loggers to logging.DEBUG, but this didn't help me. I've also looked at the gcloud.connection file, but didn't find where the error might occur.

If I'm doing anything wrong, or can help solve this problem in another way, then don't hesitate to ask!

[dhermes]

I'll try to reproduce. Anything specific about how the queue is populated from another client?

[GilJ]

I just threw some items on there (around 100 I think). The ack_deadline was set to 60 seconds, and I never ack messages. Then I pull max_messages=5 at every iteration, or sleep 5 seconds.

When trying to reproduce: it might be related to a network failure, but I would expect some sort of exception in that case. I'll encountered the problem when using scrapy (we pull messages from the queue in order to scrape them), but I'll try to reproduce it later in a simple setting.

[GilJ]

I've fixed the issue; the problem was a timeout occurred. Was able to fix the issue by creating my own http object with a timeout (httplib2.Http(timeout=TIMEOUT_SECONDS)), and authorising said object with my own credentials.

[dhermes]

Thanks for the heads up. It's strange that a connection to a Google server would hang indefinitely, @tmatsuo have you ever heard this? Should we be adding a default time-out to our default connection objects created?

[miyakawataku]

@dhermes I have experienced the same problem multiple times when calling subscription.pull(return_immediately=True), and ended up in the same workaround. It occured approximately once or twice a month, and on the occurrence, all subscribers got hung.

I think any API clients, not only the Pub/Sub client, should expect this kind of problem, and thus should set socket timeout.

[dhermes]

Thanks for the heads up. We're hoping to get better transport at some point.