Closed
Description
From reading the invocation log, there seems to be many datasets. My suspicion is that now the project has many "datasets" which makes list_datasets to be more flaky. I think the function is fetching pages and pages under the cover.
I think manually deleting those datasets will greatly mitigate
Log:
Traceback (most recent call last):
File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 826, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1588849942.771695832","description":"Error received from peer ipv4:108.177.98.81:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/tmpfs/src/github/python-docs-samples/datalabeling/manage_dataset_test.py", line 72, in test_list_dataset
run_sample()
File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/backoff/_sync.py", line 94, in retry
ret = target(*args, **kwargs)
File "/tmpfs/src/github/python-docs-samples/datalabeling/manage_dataset_test.py", line 70, in run_sample
manage_dataset.list_datasets(PROJECT_ID)
File "/tmpfs/src/github/python-docs-samples/datalabeling/manage_dataset.py", line 73, in list_datasets
for element in response:
File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 212, in _items_iter
for page in self._page_iter(increment=False):
File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 249, in _page_iter
page = self._next_page()
File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 534, in _next_page
response = self._method(self._request)
File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
return wrapped_func(*args, **kwargs)
File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/tmpfs/src/github/python-docs-samples/datalabeling/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/retry.py", line 206, in retry_target
last_exc,
File "<string>", line 3, in raise_from
google.api_core.exceptions.RetryError: Deadline of 600.0s exceeded while calling functools.partial(<function _wrap_unary_errors.<locals>.error_remapped_callable at 0x7f44b77f1320>, parent: "projects/python-docs-samples-tests"
page_token: "2020-02-26T00:23:46.026853412-08:00"
, metadata=[('x-goog-request-params', 'parent=projects/python-docs-samples-tests'), ('x-goog-api-client', 'gl-python/3.7.5 grpc/1.28.1 gax/1.17.0 gapic/0.4.0')]), last exception: 504 Deadline Exceeded