Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Custom server container registry #713

Merged
merged 7 commits into from
Sep 12, 2022
Merged

Custom server container registry #713

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
48e290c
custom server requirements.txt file generation added to the notebook
saeid93 Sep 1, 2022
3639a19
fixed typo
saeid93 Sep 2, 2022
24aa56d
complied
saeid93 Sep 2, 2022
344b57a
information added about container registry
saeid93 Sep 2, 2022
0e1e718
added the note block
saeid93 Sep 6, 2022
48d6815
lint fix
saeid93 Sep 7, 2022
a86033f
typo fix
saeid93 Sep 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 52 additions & 33 deletions docs/examples/custom/README.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,27 +54,29 @@
"from jax import random\n",
"from numpyro.infer import MCMC, NUTS\n",
"\n",
"DATASET_URL = 'https://raw.githubusercontent.com/rmcelreath/rethinking/master/data/WaffleDivorce.csv'\n",
"dset = pd.read_csv(DATASET_URL, sep=';')\n",
"DATASET_URL = \"https://raw.githubusercontent.com/rmcelreath/rethinking/master/data/WaffleDivorce.csv\"\n",
"dset = pd.read_csv(DATASET_URL, sep=\";\")\n",
"\n",
"standardize = lambda x: (x - x.mean()) / x.std()\n",
"\n",
"dset['AgeScaled'] = dset.MedianAgeMarriage.pipe(standardize)\n",
"dset['MarriageScaled'] = dset.Marriage.pipe(standardize)\n",
"dset['DivorceScaled'] = dset.Divorce.pipe(standardize)\n",
"dset[\"AgeScaled\"] = dset.MedianAgeMarriage.pipe(standardize)\n",
"dset[\"MarriageScaled\"] = dset.Marriage.pipe(standardize)\n",
"dset[\"DivorceScaled\"] = dset.Divorce.pipe(standardize)\n",
"\n",
"\n",
"def model(marriage=None, age=None, divorce=None):\n",
" a = numpyro.sample('a', dist.Normal(0., 0.2))\n",
" M, A = 0., 0.\n",
" a = numpyro.sample(\"a\", dist.Normal(0.0, 0.2))\n",
" M, A = 0.0, 0.0\n",
" if marriage is not None:\n",
" bM = numpyro.sample('bM', dist.Normal(0., 0.5))\n",
" bM = numpyro.sample(\"bM\", dist.Normal(0.0, 0.5))\n",
" M = bM * marriage\n",
" if age is not None:\n",
" bA = numpyro.sample('bA', dist.Normal(0., 0.5))\n",
" bA = numpyro.sample(\"bA\", dist.Normal(0.0, 0.5))\n",
" A = bA * age\n",
" sigma = numpyro.sample('sigma', dist.Exponential(1.))\n",
" sigma = numpyro.sample(\"sigma\", dist.Exponential(1.0))\n",
" mu = a + M + A\n",
" numpyro.sample('obs', dist.Normal(mu, sigma), obs=divorce)\n",
" numpyro.sample(\"obs\", dist.Normal(mu, sigma), obs=divorce)\n",
"\n",
"\n",
"# Start from this source of randomness. We will split keys for subsequent operations.\n",
"rng_key = random.PRNGKey(0)\n",
Expand All @@ -85,7 +87,9 @@
"# Run NUTS.\n",
"kernel = NUTS(model)\n",
"mcmc = MCMC(kernel, num_warmup=num_warmup, num_samples=num_samples)\n",
"mcmc.run(rng_key_, marriage=dset.MarriageScaled.values, divorce=dset.DivorceScaled.values)\n",
"mcmc.run(\n",
" rng_key_, marriage=dset.MarriageScaled.values, divorce=dset.DivorceScaled.values\n",
")\n",
"mcmc.print_summary()"
]
},
Expand Down Expand Up @@ -113,9 +117,9 @@
"serialisable = {}\n",
"for k, v in samples.items():\n",
" serialisable[k] = np.asarray(v).tolist()\n",
" \n",
"\n",
"model_file_name = \"numpyro-divorce.json\"\n",
"with open(model_file_name, 'w') as model_file:\n",
"with open(model_file_name, \"w\") as model_file:\n",
" json.dump(serialisable, model_file)"
]
},
Expand Down Expand Up @@ -309,14 +313,7 @@
"\n",
"x_0 = [28.0]\n",
"inference_request = {\n",
" \"inputs\": [\n",
" {\n",
" \"name\": \"marriage\",\n",
" \"shape\": [1],\n",
" \"datatype\": \"FP32\",\n",
" \"data\": x_0\n",
" }\n",
" ]\n",
" \"inputs\": [{\"name\": \"marriage\", \"shape\": [1], \"datatype\": \"FP32\", \"data\": x_0}]\n",
"}\n",
"\n",
"endpoint = \"http://localhost:8080/v2/models/numpyro-divorce/infer\"\n",
Expand All @@ -335,6 +332,26 @@
"With that goal in mind, the rough outline of steps will be to first build a custom image containing our code, and then deploy it.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Specifying requirements\n",
"MLServer will automatically find your requirements.txt file and install necessary python packages"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%writefile requirements.txt\n",
"numpy==1.22.0\n",
"numpyro==0.8.0\n",
"jax==0.2.24"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down Expand Up @@ -390,21 +407,14 @@
"\n",
"x_0 = [28.0]\n",
"inference_request = {\n",
" \"inputs\": [\n",
" {\n",
" \"name\": \"marriage\",\n",
" \"shape\": [1],\n",
" \"datatype\": \"FP32\",\n",
" \"data\": x_0\n",
" }\n",
" ]\n",
" \"inputs\": [{\"name\": \"marriage\", \"shape\": [1], \"datatype\": \"FP32\", \"data\": x_0}]\n",
"}\n",
"\n",
"endpoint = \"http://localhost:8080/v2/models/numpyro-divorce/infer\"\n",
"response = requests.post(endpoint, json=inference_request)\n",
"\n",
"print(response)\n",
"print(response.text)\n"
"print(response.text)"
]
},
{
Expand All @@ -428,6 +438,10 @@
"There is a large number of tools out there to deploy images.\n",
"However, for our example, we will focus on deploying it to a cluster running [Seldon Core](https://docs.seldon.io/projects/seldon-core/en/latest/).\n",
"\n",
"```{note}\n",
"Also consider that depending on your Kubernetes installation Seldon Core might excpect to get the container image from a public container registry like [Docker hub](https://hub.docker.com/) or [Google Container Registry](https://cloud.google.com/container-registry). For that you need to do an extra step of pushing the container to the registry using `docker tag <image name> <container registry>/<image name>` and `docker push <container registry>/<image name>` and also updating the `image` section of the yaml file to `<container registry>/<image name>`. \n",
"```\n",
"\n",
"For that, we will need to create a `SeldonDeployment` resource which instructs Seldon Core to deploy a model embedded within our custom image and compliant with the [V2 Inference Protocol](https://github.com/kserve/kserve/tree/master/docs/predict-api/v2).\n",
"This can be achieved by _applying_ (i.e. `kubectl apply`) a `SeldonDeployment` manifest to the cluster, similar to the one below:"
]
Expand Down Expand Up @@ -467,7 +481,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"display_name": "Python 3.8.10 64-bit",
"language": "python",
"name": "python3"
},
Expand All @@ -481,7 +495,12 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.8"
"version": "3.8.10"
},
"vscode": {
"interpreter": {
"hash": "e7370f93d1d0cde622a1f8e1c04877d8463912d04d973331ad4851f04de6915a"
}
}
},
"nbformat": 4,
Expand Down
15 changes: 15 additions & 0 deletions docs/examples/custom/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -251,6 +251,17 @@ Now that we have written and tested our custom model, the next step is to deploy
With that goal in mind, the rough outline of steps will be to first build a custom image containing our code, and then deploy it.


### Specifying requirements
MLServer will automatically find your requirements.txt file and install necessary python packages


```python
%%writefile requirements.txt
numpy==1.22.0
numpyro==0.8.0
jax==0.2.24
```

### Building a custom image

```{note}
Expand Down Expand Up @@ -311,6 +322,10 @@ Now that we've built a custom image and verified that it works as expected, we c
There is a large number of tools out there to deploy images.
However, for our example, we will focus on deploying it to a cluster running [Seldon Core](https://docs.seldon.io/projects/seldon-core/en/latest/).

```{note}
Also consider that depending on your Kubernetes installation Seldon Core might excpect to get the container image from a public container registry like [Docker hub](https://hub.docker.com/) or [Google Container Registry](https://cloud.google.com/container-registry). For that you need to do an extra step of pushing the container to the registry using `docker tag <image name> <container registry>/<image name>` and `docker push <container registry>/<image name>` and also updating the `image` section of the yaml file to `<container registry>/<image name>`.
adriangonz marked this conversation as resolved.
Show resolved Hide resolved
saeid93 marked this conversation as resolved.
Show resolved Hide resolved
```

For that, we will need to create a `SeldonDeployment` resource which instructs Seldon Core to deploy a model embedded within our custom image and compliant with the [V2 Inference Protocol](https://github.com/kserve/kserve/tree/master/docs/predict-api/v2).
This can be achieved by _applying_ (i.e. `kubectl apply`) a `SeldonDeployment` manifest to the cluster, similar to the one below:

Expand Down