google.auth.exceptions.TransportError not being retried

[nitishxp]

I am finding below error in one of my production environement the sdk should need to retry by default but its not retrying.
I am trying to access a bucket in Cloud function python38 environment

My package version
google-auth==1.24.0
google-cloud-storage==1.36.2

Error in app: Exception Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen httplib_response = self._make_request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request httplib_response = conn.getresponse() File "/opt/python3.8/lib/python3.8/http/client.py", line 1347, in getresponse response.begin() File "/opt/python3.8/lib/python3.8/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/opt/python3.8/lib/python3.8/http/client.py", line 276, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen retries = retries.increment( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/util/retry.py", line 410, in increment raise six.reraise(type(error), error, _stacktrace) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise raise value.with_traceback(tb) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen httplib_response = self._make_request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request httplib_response = conn.getresponse() File "/opt/python3.8/lib/python3.8/http/client.py", line 1347, in getresponse response.begin() File "/opt/python3.8/lib/python3.8/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/opt/python3.8/lib/python3.8/http/client.py", line 276, in _read_status raise RemoteDisconnected("Remote end closed connection without" urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/auth/transport/requests.py", line 182, in call response = self.session.request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/sessions.py", line 530, in request resp = self.send(prep, **send_kwargs) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/adapters.py", line 498, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/workspace/common/functions.py", line 55, in wrapper response = wrapped_func(*args, **kwargs) File "/workspace/main.py", line 970, in manifest stage_bucket = gcs.get_bucket('test-bucket') File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/client.py", line 402, in get_bucket bucket.reload( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/bucket.py", line 996, in reload super(Bucket, self).reload( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/_helpers.py", line 218, in reload api_response = client._connection.api_request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/_http.py", line 78, in api_request return call() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py", line 281, in retry_wrapped_func return retry_target( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py", line 184, in retry_target return target() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/_http.py", line 472, in api_request response = self._make_request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/_http.py", line 336, in _make_request return self._do_request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/_http.py", line 374, in _do_request return self.http.request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/auth/transport/requests.py", line 460, in request self.credentials.before_request(auth_request, method, url, request_headers) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/auth/credentials.py", line 133, in before_request self.refresh(request) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/oauth2/service_account.py", line 361, in refresh access_token, expiry, _ = _client.jwt_grant(request, self._token_uri, assertion) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/oauth2/_client.py", line 153, in jwt_grant response_data = _token_endpoint_request(request, token_uri, body) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/oauth2/_client.py", line 105, in _token_endpoint_request response = request(method="POST", url=token_uri, headers=headers, body=body) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/auth/transport/requests.py", line 188, in call six.raise_from(new_exc, caught_exc) File "", line 3, in raise_from google.auth.exceptions.TransportError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

[busunkim96]

Hi @nitishxp,

Could you share more context about when you see this error? Is it every time the function is executed, or only under certain circumstances?

Would you be able to share the call to the storage library as well?

[nitishxp]

This is the code that i am using in cloud function its the first part the Cloud function executes

googleCredentials = service_account.Credentials.from_service_account_info(authentication)
gcs = storage.Client(project=project, credentials=googleCredentials)
stage_bucket = gcs.get_bucket('test-bucket') 

Its not frequent but we r getting this in production system of ours

I also have bold the above traceback to show the bucket get operation

[busunkim96]

It looks like requests.ConnectionError is in the list of retryable types for the storage library, so the storage library intends for it to be retried.

I think it isn't actually being retried because google-auth re-wraps that error as a google.auth.exceptions.TransportError. It looks like the retry configs needs tweaking in the storage library so I'm going to close this in favor of tracking at googleapis/python-storage#414.