batch.tests.test_basics: test_container_job failed

[flaky-bot]

Note: #8899 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.

---

commit: c13058d
buildURL: Build Status, Sponge
status: failed

Test output

Traceback (most recent call last):
  File "/workspace/batch/tests/test_basics.py", line 100, in test_container_job
    _test_body(job, additional_test=lambda: _check_tasks(job_name))
  File "/workspace/batch/tests/test_basics.py", line 56, in _test_body
    pytest.fail("Timed out while waiting for job to complete!")
  File "/workspace/batch/.nox/py-3-11/lib/python3.11/site-packages/_pytest/outcomes.py", line 194, in fail
    raise Failed(msg=reason, pytrace=pytrace)
Failed: Timed out while waiting for job to complete!

[flaky-bot]

Test passed for commit 3e28b78 (Build Status, Sponge)! Closing this issue.

[flaky-bot]

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

---

commit: c99c9c0
buildURL: Build Status, Sponge
status: failed

Test output

Traceback (most recent call last):
  File "/workspace/batch/.nox/py-3-9/lib/python3.9/site-packages/google/api_core/retry.py", line 191, in retry_target
    return target()
  File "/workspace/batch/.nox/py-3-9/lib/python3.9/site-packages/google/api_core/future/polling.py", line 120, in _done_or_raise
    raise _OperationNotComplete()
google.api_core.future.polling._OperationNotComplete
The above exception was the direct cause of the following exception:
Traceback (most recent call last):

File "/workspace/batch/.nox/py-3-9/lib/python3.9/site-packages/google/api_core/future/polling.py", line 137, in _blocking_poll

polling(self._done_or_raise)(retry=retry)

File "/workspace/batch/.nox/py-3-9/lib/python3.9/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func

return retry_target(

File "/workspace/batch/.nox/py-3-9/lib/python3.9/site-packages/google/api_core/retry.py", line 207, in retry_target

raise exceptions.RetryError(

google.api_core.exceptions.RetryError: Deadline of 900.0s exceeded while calling target function, last exception:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):

File "/workspace/batch/tests/test_basics.py", line 100, in test_container_job

_test_body(job, additional_test=lambda: _check_tasks(job_name))

File "/workspace/batch/tests/test_basics.py", line 71, in _test_body

delete_job(PROJECT, REGION, test_job.name.rsplit('/', maxsplit=1)[1]).result()

File "/workspace/batch/.nox/py-3-9/lib/python3.9/site-packages/google/api_core/future/polling.py", line 256, in result

self._blocking_poll(timeout=timeout, retry=retry, polling=polling)

File "/workspace/batch/.nox/py-3-9/lib/python3.9/site-packages/google/api_core/future/polling.py", line 139, in _blocking_poll

raise concurrent.futures.TimeoutError(

concurrent.futures._base.TimeoutError: Operation did not complete within the designated timeout of 900 seconds.

[atulep]

@m-strzelczyk PTAL

Looks like the issue happens sometimes when delete_job takes more than the default value of 900 seconds to complete.

To fix, I suggest:

delete_job(PROJECT, REGION, test_job.name.rsplit('/', maxsplit=1)[1]).result(timeout=23*60*60) # or some other default value.