Fixes test flakiness caused by "unsupported operand type(s) for -=: 'Retry' and 'int'"

[Bobgy]

I want to first test if this fix actually help.
part of #2556
The fix come from https://stackoverflow.com/a/37531821

/kind bug

---

This change is 

[numerology]

Thanks @Bobgy ! This might need some time to run, though...

[Bobgy]

@numerology np, I will retry a few time today to verify.

[Ark-kun]

/lgtm

[Ark-kun]

Thanks for fixing the issue!

[Bobgy]

First try succeeded
/retest

[Bobgy]

/test kubeflow-pipeline-e2e-test
/test kubeflow-pipeline-sample-test

[Bobgy]

Second try succeeded
/test kubeflow-pipeline-e2e-test
/test kubeflow-pipeline-sample-test

[Bobgy]

Third try succeeded
/test kubeflow-pipeline-e2e-test
/test kubeflow-pipeline-sample-test

[Bobgy]

Fourth try succeeded
/test kubeflow-pipeline-e2e-test
/test kubeflow-pipeline-sample-test

[Bobgy]

Fifth try succeeded
/test kubeflow-pipeline-e2e-test
/test kubeflow-pipeline-sample-test

[Bobgy]

Sixth try failed with

Step #0: go: error loading module requirements
Step #0: The command '/bin/sh -c GO111MODULE=on go build -o /bin/persistence_agent backend/src/agent/persistence/*.go' returned a non-zero code: 1

Not related to this fix, but it shouldn't happen. Because image building should be cached. It seems inverse-proxy-agent is wrongly queried even though it is not built.

UPDATE: fixed the inverse proxy image issue in #2566

/test kubeflow-pipeline-e2e-test
/test kubeflow-pipeline-sample-test

[Bobgy]

Seventh try succeeded.
I think we can merge and later take an eye on if this error shows up again.

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Bobgy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• test/sample-test/OWNERS [Bobgy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment