retries for failed API requests

[craigcitro]

Currently, pydrive2 uses googleapiclient for all Drive API operations, but doesn't retry any failed requests. However, the Drive API recommends retries with exponential backoff for many errors, and googleapiclient supports retries for all requests.

Would it be possible to add retry behavior as per the Drive API recommendations?

(Additionally, it'd be great to add this for the initial client creation, as the discovery service can also throw 500s.)

Thanks!

[shcheklein]

@craigcitro good feature request. It's not included in the PyDrive2 yet, but you can take a code that does this logic in a nice way here https://github.com/iterative/dvc/blob/b8aa8ad9511762c94c539680ab917f86ddc24569/dvc/remote/gdrive.py#L45

(may be we should add 429 in that function)

[shcheklein]

For the record:

from funcy import cached_property, retry, wrap_prop, wrap_with

def _gdrive_retry(func):
    def should_retry(exc):
        from pydrive2.files import ApiRequestError

        if not isinstance(exc, ApiRequestError):
            return False

        error_code = exc.error.get("code", 0)
        result = False
        if 500 <= error_code < 600:
            result = True

        if error_code == 403:
            result = exc.GetField("reason") in [
                "userRateLimitExceeded",
                "rateLimitExceeded",
            ]
        if result:
            logger.debug(f"Retrying GDrive API call, error: {exc}.")

        return result

    # 16 tries, start at 0.5s, multiply by golden ratio, cap at 20s
    return retry(
        16,
        timeout=lambda a: min(0.5 * 1.618 ** a, 20),
        filter_errors=should_retry,
    )(func)