Training and Serving Pipeline leveraging WML #800

animeshsingh · 2019-02-08T19:56:20Z

The Watson Train and Serve sample pipeline runs training, storing and deploying a Tensorflow model with MNIST handwriting recognition using IBM Watson Studio and IBM Watson Machine Learning service.

This change is

k8s-ci-robot · 2019-02-08T19:56:33Z

Hi @animeshsingh. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2019-02-08T19:56:33Z

Hi @animeshsingh. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

IronPan · 2019-02-09T05:05:35Z

/retest

animeshsingh · 2019-02-11T22:41:34Z

/retest

k8s-ci-robot · 2019-02-11T22:42:00Z

@animeshsingh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2019-02-11T22:42:00Z

@animeshsingh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

IronPan · 2019-02-12T20:18:46Z

/test kubeflow-pipeline-e2e-test
/test kubeflow-pipeline-sample-test

Removing references to wml-base image, and recreating it with python-slim as base image

googlebot · 2019-02-14T07:39:21Z

So there's good news and bad news.

👍 The good news is that everyone that needs to sign a CLA (the pull request submitter and all commit authors) have done so. Everything is all good there.

😕 The bad news is that it appears that one or more commits were authored or co-authored by someone other than the pull request submitter. We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that here in the pull request.

Note to project maintainer: This is a terminal state, meaning the cla/google commit status will not change from this state. It's up to you to confirm consent of all the commit author(s), set the cla label to yes (if enabled on your project), and then merge this pull request when appropriate.

animeshsingh · 2019-02-14T07:43:19Z

@vicaire we confirm the PR is good from our side, and has consent from all the authors. Please review

components/ibm-components/watson/deploy/src/wml-deploy.py

vicaire · 2019-02-15T06:55:46Z

components/ibm-components/watson/deploy/src/wml-deploy.py

+    # download scoring payload
+    payload_file = os.path.join('/app', wml_scoring_payload)
+
+    s3 = boto3.resource('s3',


Could you please use Minio client instead of S3 directly?

https://github.com/minio/mc

Minio client abstracts away AWS S3, GCS, Azure Cloud, and Minio Server (on prem solution), so that the component is compatible with all these clouds.

@vicaire this is something we can definitely come back to in future, as we strive to take the message of 'Watson anywhere' forward. Currently s3 client is something most of the developer use

Hi animeshsingh@, we are moving towards making all the components checked into the repository compatible with multiple clouds.

Minio Client should be simple to use (as much as using an S3 client directly) and would make the component work across multiple clouds.

It would also make it easy to add a end-to-end test for this component (our e2e test framework uses GCP). e2e tests are important as they automatically verify that the component keeps working at each PR commit.

Thanks @vicaire . We will look into minio client. Vis a vis the end to end test, how are we doing it for components which use GCP, for e.g. Cloud Dataflow service? Do we create a dedicated set of credentials? If not, the tests are mostly mocking the behaviour?

@vicaire We have switched to minio client. End to end tests - want to get your input as mentioned above

components/ibm-components/watson/store/src/wml-store.py

components/ibm-components/watson/train/src/wml-train.py

vicaire · 2019-02-15T07:01:38Z

samples/ibm-samples/watson/watson_train_serve_pipeline.py

+if __name__ == '__main__':
+    # compile the pipeline
+    import kfp.compiler as compiler
+    pipeline_filename = kfp_wml_pipeline.__name__ + '.tar.gz'


Did you think of a way tests could be written for this pipeline? And end to end test would be useful but the call to IBM watson would have to be mocked. Let me know your thoughts.

Will invstigate

vicaire · 2019-02-20T09:20:49Z

Thanks animeshsingh@,

The changes look good. Could you please just modify the code/documentation so that it does not seem s3 specific but multi-cloud instead? Using "object store" instead of "s3" in the doc would be great, using minio or minio_client instead of s3 in the code would be great as well.

We are working on doing the same thing for parts of the codebase where GCS is mentioned. The community feels strongly about Kubeflow being a project that is independent of any cloud.

Let's skip e2e tests for now as we are trying to figure out to organize this for contributions. We will also need to release your containers, which requires a review process. Thanks!

animeshsingh · 2019-02-21T17:41:12Z

Thanks @vicaire. We have changed s3 to cos, standing for cloud object store. Please take a look and see if we can get this in now

vicaire · 2019-02-22T22:20:32Z

Thank you. This is great.
/lgtm
/approve
/retest

k8s-ci-robot · 2019-02-22T22:20:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vicaire

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~components/OWNERS~~ [vicaire]
~~samples/OWNERS~~ [vicaire]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

vicaire · 2019-02-22T23:00:43Z

@animeshsingh, this is blocked by the CLA agreement.

animeshsingh · 2019-02-22T23:01:48Z

@vicaire there are two authors, and the other one has given consent. He has signed the CLA as well..do we need anything else?

vicaire · 2019-02-22T23:07:48Z

According to the robot:

So there's good news and bad news.

👍 The good news is that everyone that needs to sign a CLA (the pull request submitter and all commit authors) have done so. Everything is all good there.

😕 The bad news is that it appears that one or more commits were authored or co-authored by someone other than the pull request submitter. We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that here in the pull request.

Note to project maintainer: This is a terminal state, meaning the cla/google commit status will not change from this state. It's up to you to confirm consent of all the commit author(s), set the cla label to yes (if enabled on your project), and then merge this pull request when appropriate.

vicaire · 2019-02-22T23:08:52Z

So the second author just needs to confirm approval in the pull request. Then I can add the label and we can check this in.

adrian555 · 2019-02-23T00:41:48Z

LGTM. Looks good to me. Approve.

animeshsingh · 2019-02-23T00:42:44Z

@vicaire please check above

vicaire · 2019-02-23T11:52:51Z

Sounds good. Thanks for your contribution!

* initial watson pipeline * updates * updates * Update creds * updated dockerhub base files * update watson pipeline (kubeflow#2) Removing references to wml-base image, and recreating it with python-slim as base image * address review comment (kubeflow#3) * change s3 to cos (kubeflow#4)

* Add custom inference example using BentoML * nit: add introductary to setup and improve instructions * Remove note about kfserving-ingressgateway and update curl command

animeshsingh added 5 commits February 6, 2019 22:08

initial watson pipeline

5dc0cfc

updates

f8c606e

updates

0f8060b

Update creds

acf4d57

updated dockerhub base files

04275f3

k8s-ci-robot requested review from gaoning777 and hongye-sun February 8, 2019 19:56

k8s-ci-robot added size/XL labels Feb 8, 2019

k8s-ci-robot added the needs-ok-to-test label Feb 8, 2019

update watson pipeline (#2)

690fa79

Removing references to wml-base image, and recreating it with python-slim as base image

vicaire reviewed Feb 15, 2019

View reviewed changes

address review comment (#3)

44ecd07

change s3 to cos (#4)

28f8fc2

k8s-ci-robot assigned vicaire Feb 22, 2019

k8s-ci-robot added the lgtm label Feb 22, 2019

k8s-ci-robot added the approved label Feb 22, 2019

vicaire merged commit 89e21df into kubeflow:master Feb 23, 2019

Linchin pushed a commit to Linchin/pipelines that referenced this pull request Apr 11, 2023

test(gcp): fix cleanup minor mistakes (kubeflow#800)

ce9a0f6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Training and Serving Pipeline leveraging WML #800

Training and Serving Pipeline leveraging WML #800

animeshsingh commented Feb 8, 2019 •

edited by jlewi

Loading

k8s-ci-robot commented Feb 8, 2019

k8s-ci-robot commented Feb 8, 2019

IronPan commented Feb 9, 2019

animeshsingh commented Feb 11, 2019

k8s-ci-robot commented Feb 11, 2019

k8s-ci-robot commented Feb 11, 2019

IronPan commented Feb 12, 2019

googlebot commented Feb 14, 2019

animeshsingh commented Feb 14, 2019

vicaire Feb 15, 2019

animeshsingh Feb 15, 2019

vicaire Feb 15, 2019

animeshsingh Feb 18, 2019

animeshsingh Feb 20, 2019

vicaire Feb 15, 2019

animeshsingh Feb 15, 2019

vicaire commented Feb 20, 2019

animeshsingh commented Feb 21, 2019

vicaire commented Feb 22, 2019

k8s-ci-robot commented Feb 22, 2019

vicaire commented Feb 22, 2019

animeshsingh commented Feb 22, 2019

vicaire commented Feb 22, 2019

vicaire commented Feb 22, 2019

adrian555 commented Feb 23, 2019

animeshsingh commented Feb 23, 2019

vicaire commented Feb 23, 2019

Training and Serving Pipeline leveraging WML #800

Training and Serving Pipeline leveraging WML #800

Conversation

animeshsingh commented Feb 8, 2019 • edited by jlewi Loading

k8s-ci-robot commented Feb 8, 2019

k8s-ci-robot commented Feb 8, 2019

IronPan commented Feb 9, 2019

animeshsingh commented Feb 11, 2019

k8s-ci-robot commented Feb 11, 2019

k8s-ci-robot commented Feb 11, 2019

IronPan commented Feb 12, 2019

googlebot commented Feb 14, 2019

animeshsingh commented Feb 14, 2019

vicaire Feb 15, 2019

Choose a reason for hiding this comment

animeshsingh Feb 15, 2019

Choose a reason for hiding this comment

vicaire Feb 15, 2019

Choose a reason for hiding this comment

animeshsingh Feb 18, 2019

Choose a reason for hiding this comment

animeshsingh Feb 20, 2019

Choose a reason for hiding this comment

vicaire Feb 15, 2019

Choose a reason for hiding this comment

animeshsingh Feb 15, 2019

Choose a reason for hiding this comment

vicaire commented Feb 20, 2019

animeshsingh commented Feb 21, 2019

vicaire commented Feb 22, 2019

k8s-ci-robot commented Feb 22, 2019

vicaire commented Feb 22, 2019

animeshsingh commented Feb 22, 2019

vicaire commented Feb 22, 2019

vicaire commented Feb 22, 2019

adrian555 commented Feb 23, 2019

animeshsingh commented Feb 23, 2019

vicaire commented Feb 23, 2019

animeshsingh commented Feb 8, 2019 •

edited by jlewi

Loading