Description
Connector Name
zendesk-support
Connector Version
4.4.2
What step the error happened?
During the sync
Relevant information
During sync, we reached a point where consistently the zendesk support tickets stream failed to sync due to Gateway timeout.
The default behavior for Gateway timeout is to do exponential back off and retry 9 times before failing.
After looking into the internals of the airbyte source for Zendesk support and zendesk api for incremental ticket exporting, I replicated the issue with a script that fetched data directly from Zendesk.
My issue is that I stumbled upon a case where a single incremental ticket export response was so big that it resulted in a Gateway timeout. As a result, my connection is broken, and it cannot sync further since it always encounters this big block of tickets. Is there a way to salvage the connection?
While it makes sense to wait and retry on 429 errors that might occur due to rate limiting, in the Gateway timeout that occurs due to size, the ideal solution would be to utilize the per_page parameter of the incremental_exports. In my script, I halved the per_page size on every retry until it succeeded and then reset its size to the default 1000. This way, I managed to move the sync cursor past the big block of tickets that causes my Airbyte connection to break.
Relevant log output
source INFO Backing off _send(...) for 1.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException: Gateway timeout.)
source INFO Caught retryable error 'Gateway timeout.' after 1 tries. Waiting 1 seconds then retrying...
source INFO Backing off _send(...) for 2.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException: Gateway timeout.)
source INFO Caught retryable error 'Gateway timeout.' after 2 tries. Waiting 2 seconds then retrying...
source INFO Backing off _send(...) for 4.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException: Gateway timeout.)
source INFO Caught retryable error 'Gateway timeout.' after 3 tries. Waiting 4 seconds then retrying...Contribute
- Yes, I want to contribute