Skip to content

Commit

Permalink
add nuclio components (to build/deploy, delete, invoke functions) (#1295
Browse files Browse the repository at this point in the history
)

* add support for flexible config (via env var) for the pipline service and UI, fix broken links (pointed to API vs UI service)

* add namespace and change address to endpoint
  • Loading branch information
yaronha authored and k8s-ci-robot committed May 8, 2019
1 parent b61bef0 commit 442c804
Show file tree
Hide file tree
Showing 4 changed files with 107 additions and 0 deletions.
44 changes: 44 additions & 0 deletions components/nuclio/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Nuclio (Serverless) Components

[Nuclio](https://nuclio.io/) is a native and high performance serverless platform over Kubernetes
which automate the process of build, deployment, monitoring, and auto-scaling of micro-services.
Nuclio support variety of data and data-science related features (e.g. stream processing,
GPUs, volume/DB mounts, high concurrency, etc.)

To install Nuclio over Kubernetes follow the [instruction in Github](https://github.com/nuclio/nuclio),
or this [interactive tutorial](https://www.katacoda.com/javajon/courses/kubernetes-serverless/nuclio).

Nuclio functions can be used in the following ML pipline tasks:
* Data collectors, ETL, stream processing
* Data preparation and analysis
* Hyper parameter model training
* Real-time model serving
* Feature vector assembly (real-time data preparation)

Read more on the use of Nuclio in [data-science here](https://towardsdatascience.com/serverless-can-it-simplify-data-science-projects-8821369bf3bd).
Nuclio functions can be generated automatically from 8 code languages, from Jupyter Notebooks, Zip, Git, Docker, etc.
The [nuclio-jupyter repo](https://github.com/nuclio/nuclio-jupyter) provide guidance and many examples.

## Components

There are currently 3 components in this package:
* [deploy](deploy/component.yaml) - Automatically build and deploy/re-deploy functions
from code/zip/notebooks/git/.. and/or override various deployment configurations such as
setting cpu/mem/gpu resources, scaling, environment variables, triggers, etc.
* [delete](delete/component.yaml) - Delete a function
* [invoker](invoker/component.yaml) - invoke a function and return the results/logs

Additional components and examples will be added soon for parallel batch/stream processing

## Examples

**Deploy a function (from Github)**

```python
nuclio_dep = kfp.components.load_component_from_file('deploy/component.yaml')

def my_pipeline():
new_func = nuclio_dep(url='git://github.com/nuclio/nuclio#master:/hack/examples/python/helloworld', name='myfunc', project='myproj', tag='0.11')

...
```
14 changes: 14 additions & 0 deletions components/nuclio/delete/component.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
name: nuclio delete
description: delete a nuclio function.
inputs:
- {name: Name, type: String, description: 'function name'}
- {name: Namespace, type: String, description: 'Kubernetes namespace', default: ''}
- {name: Dashboard, type: String, description: 'nuclio dashboard service url', default: 'http://nuclio-dashboard.nuclio:8070'}
implementation:
container:
image: nuclio/pydeploy
command: [
python, -m, nuclio, del, {inputValue: Name},
--dashboard-url, {inputValue: Dashboard},
--namespace, {inputValue: Namespace},
]
28 changes: 28 additions & 0 deletions components/nuclio/deploy/component.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: nuclio deploy
description: auto build and deploy nuclio function.
inputs:
- {name: Url, type: String, description: 'url/path to source code, zip archive, git path, or notebook'}
- {name: Name, type: String, description: 'function name'}
- {name: Project, type: String, description: 'project name', default: 'default'}
- {name: Tag, type: String, description: 'function version tag', default: ''}
- {name: Dashboard, type: String, description: 'nuclio dashboard service url', default: 'http://nuclio-dashboard.nuclio:8070'}
- {name: Spec, type: String, description: 'override function spec, Json {key: value, ..}', default: ''}
- {name: Env, type: String, description: 'override function env var, Json {key: value, ..}', default: ''}
- {name: Mount, type: String, description: 'volume mount, [vol-type:]<vol-url>:<dst-path>', default: ''}
outputs:
- {name: Endpoint, type: String, description: 'function endpoint url'}
implementation:
container:
image: nuclio/pydeploy
command: [
python, -m, nuclio, deploy, {inputValue: Url},
--dashboard-url, {inputValue: Dashboard},
--name, {inputValue: Name},
--project, {inputValue: Project},
--tag, {inputValue: Tag},
--env-json, {inputValue: Env},
--spec-json, {inputValue: Spec},
--mount, {inputValue: Mount},
]
fileOutputs:
Endpoint: /tmp/output
21 changes: 21 additions & 0 deletions components/nuclio/invoker/component.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
name: nuclio invoker
description: invoke nuclio function.
inputs:
- {name: Url, type: String, description: 'function URL/endpoint'}
- {name: Body, type: String, description: 'request body', default: ''}
- {name: Log level, type: String, description: 'log level', default: 'info'}
- {name: Method, type: String, description: 'HTTP method GET|POST|..', default: ''}
outputs:
- {name: output, type: String, description: 'function output'}
implementation:
container:
image: nuclio/invoker
command: [
invoke,
-a, {inputValue: Url},
-b, {inputValue: Body},
-m, {inputValue: Method},
-l, {inputValue: Log level},
]
fileOutputs:
output: /tmp/output

0 comments on commit 442c804

Please sign in to comment.