Add retry options to BigQuery

[Yanson]

What this PR does / why we need it:
Add two fields to job properties and use them to set retry options in all BigQuery job waitFor calls, as per Google example.

This fixes an issue where job metadata is sometimes used on an object that does not have it set, since the job was not reassigned to the completed job reference that waitFor returns.

Additionally handles errors with best practice and allows some retry options to be configured by the user.

Does this PR introduce a user-facing change?:

Users should add the following settings under feast.jobs in their application yaml for batch serving:
    bigquery-initial-retry-delay-secs: 1
    bigquery-total-timeout-secs: 21600

[feast-ci-bot]

Hi @Yanson. Thanks for your PR.

I'm waiting for a gojek member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Yanson]

/assign @woop

[zhilingc]

/ok-to-test

[zhilingc]

/test test-end-to-end-batch

[khorshuheng]

/test test-end-to-end-batch

[Yanson]

/test test-end-to-end-batch

[Yanson]

Hopefully this retest will pass. I will update the user-facing notes on Monday as it's worth mentioning.

[woop]

Thanks for this @Yanson!

Assigning @zhilingc to this one.

[zhilingc]

/lgtm

I think the whole serving jobs configuration needs a bit of a refactor but I think that's a PR for another day.

[davidheryanto]

/approve

[feast-ci-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: davidheryanto, Yanson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [davidheryanto]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Yanson]

Failure to specify the configs described may result in the following error message in logs:

Client:

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Unable to load entity dataset to Bigquery"

Server:

total timeout or maximum number of attempts exceeded; current settings: RetrySettings{totalTimeout=PT0S, initialRetryDelay=PT0S, retryDelayMultiplier=1.0, maxRetryDelay=PT3S, maxAttempts=0, jittered=true, initialRpcTimeout=PT0S, rpcTimeoutMultiplier=1.0, maxRpcTimeout=PT0S}