Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

updated ffdl notebook using component yaml file #1241

Merged
merged 3 commits into from
Apr 27, 2019
Merged

updated ffdl notebook using component yaml file #1241

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
683b903
updating ffdl notebook to use component yaml file
animeshsingh Apr 26, 2019
184335f
using just create experiment
animeshsingh Apr 26, 2019
0539bf3
removing the images for now
animeshsingh Apr 26, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
163 changes: 85 additions & 78 deletions samples/ibm-samples/ffdl-seldon/ffdl_pipeline.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,27 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# FfDL Kubeflow Pipeline Notebook demo\n"
"# Train and deploy with FfDL and Seldon demo\n"
]
},

{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### A simple IBM OSS pipeline demonstrates how to train a model using Fabric for Deep Learning and then deploy it with Seldon.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Model for this pipeline\n",
"\n",
"We will be training a PyTorch model that can classify the gender of a human face image. This PyTorch model is a simple convolutional neural network (CNN) with 3 convolutional layers and 2 fully connected layers using the [UTKFace](https://susanqq.github.io/UTKFace/) dataset. We will be training for 5 epochs for the purpose of this demo.\n",
"\n",
"<br>"
]
},
{
Expand All @@ -16,9 +36,7 @@
"\n",
"**Please fill in the below environment variables with you own settings.**\n",
"\n",
"- **EXPERIMENT_NAME**: A unique experiment name that will be created for this notebook demo.\n",
"- **KFP_PACKAGE**: The latest release of kubeflow pipeline platform library.\n",
"- **KUBEFLOW_PIPELINE_LINK**: The link to access the KubeFlow pipeline API."
"- **KFP_PACKAGE**: The release of kubeflow pipeline library."
]
},
{
Expand All @@ -27,18 +45,7 @@
"metadata": {},
"outputs": [],
"source": [
"EXPERIMENT_NAME = 'jupyter-demo'\n",
"KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.6/kfp.tar.gz'\n",
"KUBEFLOW_PIPELINE_LINK = ''"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Then, define the GitHub credentials manaully or load it from the credentials folder\n",
"- **config_file_url**: GitHub raw content link to the pipeline credentials file\n",
"- **github_token**: GitHub Token that can access your private repository"
"KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.17/kfp.tar.gz'"
]
},
{
Expand All @@ -47,11 +54,8 @@
"metadata": {},
"outputs": [],
"source": [
"import configparser\n",
"config = configparser.ConfigParser()\n",
"config.read('credentials/github-creds.ini')\n",
"config_file_url = config['CREDENTIALS']['config_file_url']\n",
"github_token = config['CREDENTIALS']['github_token']"
"config_file_url = ''\n",
"github_token = ''"
]
},
{
Expand All @@ -60,7 +64,7 @@
"source": [
"### Install the necessary python packages\n",
"\n",
"Note: Please change pip3 to the package manager that's used for this Notebook Kernel."
"Note: Please change pip to the package manager that's used for this Notebook Kernel."
]
},
{
Expand All @@ -69,8 +73,8 @@
"metadata": {},
"outputs": [],
"source": [
"!pip3 install ai_pipeline_params --upgrade\n",
"!pip3 install $KFP_PACKAGE --upgrade"
"!pip install $KFP_PACKAGE --upgrade\n",
"!pip install ai_pipeline_params --upgrade"
]
},
{
Expand All @@ -88,18 +92,23 @@
"source": [
"import kfp\n",
"from kfp import compiler\n",
"import kfp.dsl as dsl\n",
"import kfp.notebook\n",
"import kfp.gcp as gcp\n",
"import kfp\n",
"from kfp import components\n",
"from kfp import dsl\n",
"from kfp import notebook\n",
"\n",
"# Run client with KUBEFLOW_PIPELINE_LINK if this notebook server\n",
"# is running on localhost without enterprise gateway.\n",
"\n",
"# KUBEFLOW_PIPELINE_LINK = ''\n",
"# client = kfp.Client(KUBEFLOW_PIPELINE_LINK)\n",
"\n",
"client = kfp.Client(KUBEFLOW_PIPELINE_LINK)\n",
"client = kfp.Client()\n",
"\n",
"# Uncomment the below line if you want to create an experiment, \n",
"# else we will assume the EXPERIMENT_NAME is already exist.\n",
"\n",
"# exp = client.create_experiment(name=EXPERIMENT_NAME)\n",
"EXPERIMENT_NAME = 'FfDL-Seldon Experiments'\n",
"\n",
"exp = client.get_experiment(experiment_name=EXPERIMENT_NAME)"
"exp = client.create_experiment(name=EXPERIMENT_NAME)\n"
]
},
{
Expand All @@ -109,6 +118,19 @@
"### 2. Define pipeline tasks using the kfp library. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# define secret name that contains the credentials for this pipeline, and load components\n",
"secret_name = 'kfp-creds'\n",
"configuration_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/master/components/ibm-components/commons/config/component.yaml')\n",
"train_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/master/components/ibm-components/ffdl/train/component.yaml')\n",
"serve_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/master/components/ibm-components/ffdl/serve/component.yaml')"
]
},
{
"cell_type": "code",
"execution_count": null,
Expand All @@ -118,56 +140,40 @@
"import kfp.dsl as dsl\n",
"import ai_pipeline_params as params\n",
"\n",
"# generate default secret name\n",
"secret_name = 'kfp-creds'\n",
"\n",
"\n",
"# create pipeline\n",
"@dsl.pipeline(\n",
" name='FfDL pipeline',\n",
" description='A pipeline for machine learning workflow using Fabric for Deep Learning and Seldon.'\n",
")\n",
"def ffdlPipeline(\n",
" GITHUB_TOKEN=dsl.PipelineParam(name='github-token'),\n",
" CONFIG_FILE_URL=dsl.PipelineParam(name='config-file-url'),\n",
" model_def_file_path=dsl.PipelineParam(name='model-def-file-path',\n",
" value='gender-classification.zip'),\n",
" manifest_file_path=dsl.PipelineParam(name='manifest-file-path',\n",
" value='manifest.yml'),\n",
" model_deployment_name=dsl.PipelineParam(name='model-deployment-name',\n",
" value='gender-classifier'),\n",
" model_class_name=dsl.PipelineParam(name='model-class-name',\n",
" value='ThreeLayerCNN'),\n",
" model_class_file=dsl.PipelineParam(name='model-class-file',\n",
" value='gender_classification.py')\n",
" GITHUB_TOKEN=github_token,\n",
" CONFIG_FILE_URL=config_file_url,\n",
" model_def_file_path='gender-classification.zip',\n",
" manifest_file_path='manifest.yml',\n",
" model_deployment_name='gender-classifier',\n",
" model_class_name='ThreeLayerCNN',\n",
" model_class_file='gender_classification.py'\n",
"):\n",
" \"\"\"A pipeline for end to end machine learning workflow.\"\"\"\n",
" config_op = dsl.ContainerOp(\n",
" name=\"config\",\n",
" image=\"aipipeline/wml-config\",\n",
" command=['python3'],\n",
" arguments=['/app/config.py',\n",
" '--token', GITHUB_TOKEN,\n",
" '--url', CONFIG_FILE_URL,\n",
" '--name', secret_name],\n",
" file_outputs={'secret-name': '/tmp/' + secret_name}\n",
"\n",
" get_configuration = configuration_op(\n",
" token = GITHUB_TOKEN,\n",
" url = CONFIG_FILE_URL,\n",
" name = secret_name\n",
" )\n",
"\n",
" train = dsl.ContainerOp(\n",
" name='train',\n",
" image='aipipeline/ffdl-train:0.6',\n",
" command=['sh', '-c'],\n",
" arguments=['echo %s > /tmp/logs.txt; python -u train.py --model_def_file_path %s --manifest_file_path %s;'\n",
" % (config_op.output, model_def_file_path, manifest_file_path)],\n",
" file_outputs={'output': '/tmp/training_id.txt'}).apply(params.use_ai_pipeline_params(secret_name))\n",
"\n",
" serve = dsl.ContainerOp(\n",
" name='serve',\n",
" image='aipipeline/ffdl-serve:0.11',\n",
" command=['sh', '-c'],\n",
" arguments=['python -u serve.py --model_id %s --deployment_name %s --model_class_name %s --model_class_file %s;'\n",
" % (train.output, model_deployment_name, model_class_name, model_class_file)],\n",
" file_outputs={'output': '/tmp/deployment_result.txt'}).apply(params.use_ai_pipeline_params(secret_name))"
" train = train_op(\n",
" model_def_file_path,\n",
" manifest_file_path\n",
" ).apply(params.use_ai_pipeline_params(secret_name))\n",
"\n",
" serve = serve_op(\n",
" train.output, \n",
" model_deployment_name, \n",
" model_class_name, \n",
" model_class_file\n",
" ).apply(params.use_ai_pipeline_params(secret_name))"
]
},
{
Expand All @@ -190,11 +196,12 @@
"\n",
"compiler.Compiler().compile(ffdlPipeline, 'ffdl-pipeline.tar.gz')\n",
"\n",
"run = client.run_pipeline(exp.id, 'ffdl-pipeline', 'ffdl-pipeline.tar.gz', \n",
"run = client.run_pipeline(exp.id, 'ffdl-seldon-pipeline', 'ffdl-pipeline.tar.gz', \n",
" params=parameters)\n",
"\n",
"print('The above run link is assuming you ran this cell on JupyterHub that is deployed on the same cluster. ' +\n",
" 'The actual run link is ' + KUBEFLOW_PIPELINE_LINK + '/#/runs/details/' + run.id)"
"import IPython\n",
"html = ('<p id=\"link\"> </p> <script> document.getElementById(\"link\").innerHTML = \"Actual Run link <a href=//\" + location.hostname + \"%s/#/runs/details/%s target=_blank >here</a>\"; </script>'\n",
" % (client._get_url_prefix(), run.id))\n",
"IPython.display.HTML(html)"
]
},
{
Expand All @@ -207,9 +214,9 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "Python on Kubernetes",
"language": "python",
"name": "python3"
"name": "python_kubernetes"
},
"language_info": {
"codemirror_mode": {
Expand All @@ -221,7 +228,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
"version": "3.6.8"
}
},
"nbformat": 4,
Expand Down