2682: Support a custom kube config location in K8sJobHelper

[pahask8]

Fixes #2682

This change is 

[k8s-ci-robot]

Hi @pahask8. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pahask8]

@Ark-kun please take a look.

[Ark-kun]

/ok-to-test

[pahask8]

@Ark-kun , @gaoning777 , could you please take a look? Thank you

[Ark-kun]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Ark-kun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• sdk/OWNERS [Ark-kun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[pahask8]

/retest

[pahask8]

Hi @Ark-kun, is master broken? Tests failed:

build-basic-e2e-tests-image: ERROR: google-auth 1.8.1 has requirement setuptools>=40.3.0, but you'll have setuptools 33.1.1 which is incompatible

Thank you

[pahask8]

/retest

[pahask8]

/retest

[Ark-kun]

/retest

[pahask8]

/retest

[pahask8]

/retest

[pahask8]

/test kubeflow-pipeline-e2e-test

[pahask8]

/test kubeflow-pipeline-e2e-test

[pahask8]

Hi @Ark-kun, my pull request is still failing with e2e tests even when I've rebased with the master:

ERROR: google-auth 1.9.0 has requirement setuptools>=40.3.0, but you'll have setuptools 33.1.1 which is incompatible.. Looks like it is not related to my changes, could you please suggest?

Thank you

[Ark-kun]

Looks like it is not related to my changes, could you please suggest?

The actual error is:

run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	List of buckets:
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	Traceback (most recent call last):
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	  File "<string>", line 6, in <module>
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	  File "/usr/local/lib/python2.7/dist-packages/google/api_core/page_iterator.py", line 212, in _items_iter
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	    for page in self._page_iter(increment=False):
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	  File "/usr/local/lib/python2.7/dist-packages/google/api_core/page_iterator.py", line 243, in _page_iter
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	    page = self._next_page()
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	  File "/usr/local/lib/python2.7/dist-packages/google/api_core/page_iterator.py", line 369, in _next_page
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	    response = self._get_next_page_response()
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	  File "/usr/local/lib/python2.7/dist-packages/google/api_core/page_iterator.py", line 419, in _get_next_page_response
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	    method=self._HTTP_METHOD, path=self.path, query_params=params
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	  File "/usr/local/lib/python2.7/dist-packages/google/cloud/_http.py", line 417, in api_request
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	    timeout=timeout,
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	  File "/usr/local/lib/python2.7/dist-packages/google/cloud/_http.py", line 275, in _make_request
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	    method, url, headers, data, target_object, timeout=timeout
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	  File "/usr/local/lib/python2.7/dist-packages/google/cloud/_http.py", line 313, in _do_request
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	    url=url, method=method, headers=headers, data=data, timeout=timeout
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	  File "/usr/local/lib/python2.7/dist-packages/google/auth/transport/requests.py", line 277, in request
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	    self.credentials.before_request(auth_request, method, url, request_headers)
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	  File "/usr/local/lib/python2.7/dist-packages/google/auth/credentials.py", line 124, in before_request
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	    self.refresh(request)
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	  File "/usr/local/lib/python2.7/dist-packages/google/auth/compute_engine/credentials.py", line 102, in refresh
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	    six.raise_from(new_exc, caught_exc)
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	  File "/usr/lib/python2.7/dist-packages/six.py", line 737, in raise_from
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	    raise value
run-sample-tests-loop(4:secret):	using-google-cloud-apis-with-auth:	google.auth.exceptions.RefreshError: ('Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Enginemetadata service. Status: 500 Response:\nGKE Metadata Server encountered an error: Could not recursively fetch uri\n', <google.auth.transport.requests._Response object at 0x7f32e3834650>)
run-sample-tests-loop(4:secret):	access-gcs-using-auth-token:	gs://ml-pipeline-playground/shakespeare1.txt
run-sample-tests-loop(4:secret):	access-gcs-using-auth-token:	Auth token is located at /secret/gcp-credentials/user-gcp-sa.json

[Ark-kun]

/retest

[pahask8]

Thank you, looks like it passed now.