PubSub: Stream never gets re-established after being killed

[guillaumelachaud]

As I updated today from pubsub 1.1.0 to 1.4.2, I noticed something strange on my async subscription:

Every 10 minutes, the stream dies out, but where it was getting re-established in 1.1.0, 1.4.2 ignores the error and thus all the subsequent messages:

020-04-16 22:32:16,537 - google.cloud.pubsub_v1.subscriber._protocol.leaser - DEBUG - Snoozing lease management for 6.620806 seconds.
2020-04-16 22:32:23,162 - google.cloud.pubsub_v1.subscriber._protocol.leaser - DEBUG - The current deadline value is 10 seconds.
2020-04-16 22:32:23,162 - google.cloud.pubsub_v1.subscriber._protocol.leaser - DEBUG - Snoozing lease management for 7.803144 seconds.
2020-04-16 22:32:30,396 - google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager - INFO - Observed non-terminating stream error 504 Deadline Exceeded
2020-04-16 22:32:30,463 - google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager - INFO - Observed recoverable stream error 504 Deadline Exceeded
2020-04-16 22:32:30,463 - google.api_core.bidi - DEBUG - Re-opening stream from gRPC callback.
2020-04-16 22:32:30,544 - google.api_core.bidi - DEBUG - Empty queue and inactive call, exiting request generator.
2020-04-16 22:32:30,970 - google.cloud.pubsub_v1.subscriber._protocol.leaser - DEBUG - The current deadline value is 10 seconds.
2020-04-16 22:32:30,970 - google.cloud.pubsub_v1.subscriber._protocol.leaser - DEBUG - Snoozing lease management for 8.288502 seconds.

Is this an expected behavior change?

Thanks

[guillaumelachaud]

After investigating, it looks like the issue is in python-api-core. I did an upgrade from 1.14.3 to 1.17.0 at the same time I did my upgrade of PubSub. Reverting PubSub to 1.1.0 didn't solve the problem, but reverting api-core did.

Could be linked to googleapis/python-api-core#25

[plamut]

@guillaumelachaud Thanks for reporting the issue, it appears to be a duplicate of #74. The api-core dependency introduced a regression in the latest version, thus it needs to be reverted back to 1.16.0 until the issue gets fixed there - as you already correctly discovered by yourself.

There is a new PubSub release (v1.4.3) that restricts the version of the troublesome dependency, thus you might want to use just that if you find it more convenient.