feat(sdk+backend): Add support for generic ephemeral volume

[abaland]

Description of your changes:

This change is the second half of the intended change in #10602 but adds the actual support for ephemeral volume in SDK and KFP backend.

I haven't actually tested this locally yet (I think I'll need to update the go.mod once the other PR is approved/merged/deployed), but I think the code will mostly stay the same as long as no bugs occurs when i try this out in our KFP fork

Checklist:

• The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

[google-oss-prow]

Hi @abaland. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[abaland]

completely unrelated, just saw this when debugging unit tests

[Tomcli]

/ok-to-test

[abaland]

/retest

[abaland]

/retest

[abaland]

Is there a way to get access to the Artifacts produced by the above tests?
I found the cause for 2 of the failures (forgot to mod tidy and to updates license) but the rest one is an error at integration tests preparation and if i try to look at the artifacts in https://oss.gprow.dev/view/gs/oss-prow/pr-logs/pull/kubeflow_pipelines/10605/kubeflow-pipeline-upgrade-test/1772473169093857280 I need to authenticate with a google.com account it seems.

[abaland]

Mhhh, one of the errors (samples-v2) is

  File "sample_test.py", line 21, in <module>
    import kfp.deprecated as kfp
  File "/usr/local/lib/python3.7/site-packages/kfp/deprecated/__init__.py", line 21, in <module>
    from ._client import Client
  File "/usr/local/lib/python3.7/site-packages/kfp/deprecated/_client.py", line 97, in <module>
    class Client(object):
  File "/usr/local/lib/python3.7/site-packages/kfp/deprecated/_client.py", line 396, in Client
    def get_kfp_healthz(self) -> kfp_server_api.ApiGetHealthzResponse:
AttributeError: module 'kfp_server_api' has no attribute 'ApiGetHealthzResponse'

One (backend-test) is

ok  	github.com/kubeflow/pipelines/backend/src/v2/objectstore	0.056s	coverage: 14.0% of statements
FAIL	github.com/kubeflow/pipelines/backend/test/initialization [build failed]
FAIL	github.com/kubeflow/pipelines/backend/test/integration [build failed]
FAIL	github.com/kubeflow/pipelines/backend/test/v2/initialization [build failed]
FAIL	github.com/kubeflow/pipelines/backend/test/v2/integration [build failed]

which i could reproduce locally by just executing the test folder and got

# github.com/kubeflow/pipelines/backend/src/common/client/api_server/v1
../../src/common/client/api_server/v1/experiment_client.go:33:24: undefined: params.CreateExperimentV1Params
../../src/common/client/api_server/v1/experiment_client.go:34:21: undefined: params.GetExperimentV1Params
../../src/common/client/api_server/v1/experiment_client.go:35:22: undefined: params.ListExperimentsV1Params
../../src/common/client/api_server/v1/experiment_client.go:36:25: undefined: params.ListExperimentsV1Params
../../src/common/client/api_server/v1/experiment_client.go:37:25: undefined: params.ArchiveExperimentV1Params
../../src/common/client/api_server/v1/experiment_client.go:38:27: undefined: params.UnarchiveExperimentV1Params
../../src/common/client/api_server/v1/healthz_client.go:31:24: undefined: params.GetHealthzOK
../../src/common/client/api_server/v1/job_client.go:33:24: undefined: params.CreateJobParams
../../src/common/client/api_server/v1/job_client.go:34:21: undefined: params.GetJobParams
../../src/common/client/api_server/v1/job_client.go:35:24: undefined: params.DeleteJobParams
../../src/common/client/api_server/v1/job_client.go:35:24: too many errors
FAIL	github.com/kubeflow/pipelines/backend/test/initialization [build failed]

I don't think this MR touched anything related to the above, which makes me think this is related to the multiple MR from last night's and the KFP api server release that came with it.

[abaland]

For the first error, I see that it used to be ApiGetHealthzResponse in

pipelines/backend/api/v1beta1/python_http_client/kfp_server_api/__init__.py

Line 39 in 5399585

from kfp_server_api.models.api_get_healthz_response import ApiGetHealthzResponse

for the v1beta1, but got switched to V2beta1GetHealthzResponse in https://github.com/kubeflow/pipelines/blob/5399585b6a0f92446bcfc5a7588f2a85ea0fe6a3/backend/api/v2beta1/python_http_client/kfp_server_api/__init__.py#L55C1-L56C1
so i guess i could switch other all kfp_server_api.Api* to kfp_server_api.V2beta1*, but I'll admit i have no idea why this is happening and where/how this changed. So any pointer is more than welcome

[rimolive]

@abaland With #10602 merged, you're free to remove WIP label in this PR. Let's see with the new changes if tests will keep breaking.

[abaland]

@rimolive I rebased the current branch onto master after the previous PR had been merged (and also pulled the most recent version of go kubernetes_platform module) but unfortunately that did not fix the tests.
At least https://github.com/abaland/pipelines/tree/feat/ephemeral_volume2 does not seem to indicate I have commits from master not included.
Did I miss something else to run an update?

[rimolive]

/retest

[Tomcli]

@abaland Can you rebase to the latest commit to include the new API client code? Thanks

[abaland]

Just did 🤞

[google-oss-prow]

@abaland: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
kubeflow-pipelines-samples-v2	01bd04b	link	false	/test kubeflow-pipelines-samples-v2
kfp-kubernetes-execution-tests	01bd04b	link	false	/test kfp-kubernetes-execution-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[abaland]

Ok the required tests passed this time :dance:

The samples tests failing: I tried compiling one of the kfp-kubernetes-execution-tests test case locally, uploading it to our KFP environment (running on KF 1.8 release) and running it but it did not cause any error, so i'm not sure what's going on.

the samples-v2 is having the same issues as before and using the wrong api endpoint in kfp_server_api, which seems unrelated to the current PR

[rimolive]

Since they are not required tests, I believe we can just ignore
/lgtm
/assign @Tomcli @chensun

[Tomcli]

/lgtm

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chensun, Tomcli

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [chensun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment