-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
add nuclio components (to build/deploy, delete, invoke functions) (#1295
) * add support for flexible config (via env var) for the pipline service and UI, fix broken links (pointed to API vs UI service) * add namespace and change address to endpoint
- Loading branch information
1 parent
b61bef0
commit 442c804
Showing
4 changed files
with
107 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,44 @@ | ||
# Nuclio (Serverless) Components | ||
|
||
[Nuclio](https://nuclio.io/) is a native and high performance serverless platform over Kubernetes | ||
which automate the process of build, deployment, monitoring, and auto-scaling of micro-services. | ||
Nuclio support variety of data and data-science related features (e.g. stream processing, | ||
GPUs, volume/DB mounts, high concurrency, etc.) | ||
|
||
To install Nuclio over Kubernetes follow the [instruction in Github](https://github.com/nuclio/nuclio), | ||
or this [interactive tutorial](https://www.katacoda.com/javajon/courses/kubernetes-serverless/nuclio). | ||
|
||
Nuclio functions can be used in the following ML pipline tasks: | ||
* Data collectors, ETL, stream processing | ||
* Data preparation and analysis | ||
* Hyper parameter model training | ||
* Real-time model serving | ||
* Feature vector assembly (real-time data preparation) | ||
|
||
Read more on the use of Nuclio in [data-science here](https://towardsdatascience.com/serverless-can-it-simplify-data-science-projects-8821369bf3bd). | ||
Nuclio functions can be generated automatically from 8 code languages, from Jupyter Notebooks, Zip, Git, Docker, etc. | ||
The [nuclio-jupyter repo](https://github.com/nuclio/nuclio-jupyter) provide guidance and many examples. | ||
|
||
## Components | ||
|
||
There are currently 3 components in this package: | ||
* [deploy](deploy/component.yaml) - Automatically build and deploy/re-deploy functions | ||
from code/zip/notebooks/git/.. and/or override various deployment configurations such as | ||
setting cpu/mem/gpu resources, scaling, environment variables, triggers, etc. | ||
* [delete](delete/component.yaml) - Delete a function | ||
* [invoker](invoker/component.yaml) - invoke a function and return the results/logs | ||
|
||
Additional components and examples will be added soon for parallel batch/stream processing | ||
|
||
## Examples | ||
|
||
**Deploy a function (from Github)** | ||
|
||
```python | ||
nuclio_dep = kfp.components.load_component_from_file('deploy/component.yaml') | ||
|
||
def my_pipeline(): | ||
new_func = nuclio_dep(url='git://github.com/nuclio/nuclio#master:/hack/examples/python/helloworld', name='myfunc', project='myproj', tag='0.11') | ||
|
||
... | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
name: nuclio delete | ||
description: delete a nuclio function. | ||
inputs: | ||
- {name: Name, type: String, description: 'function name'} | ||
- {name: Namespace, type: String, description: 'Kubernetes namespace', default: ''} | ||
- {name: Dashboard, type: String, description: 'nuclio dashboard service url', default: 'http://nuclio-dashboard.nuclio:8070'} | ||
implementation: | ||
container: | ||
image: nuclio/pydeploy | ||
command: [ | ||
python, -m, nuclio, del, {inputValue: Name}, | ||
--dashboard-url, {inputValue: Dashboard}, | ||
--namespace, {inputValue: Namespace}, | ||
] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
name: nuclio deploy | ||
description: auto build and deploy nuclio function. | ||
inputs: | ||
- {name: Url, type: String, description: 'url/path to source code, zip archive, git path, or notebook'} | ||
- {name: Name, type: String, description: 'function name'} | ||
- {name: Project, type: String, description: 'project name', default: 'default'} | ||
- {name: Tag, type: String, description: 'function version tag', default: ''} | ||
- {name: Dashboard, type: String, description: 'nuclio dashboard service url', default: 'http://nuclio-dashboard.nuclio:8070'} | ||
- {name: Spec, type: String, description: 'override function spec, Json {key: value, ..}', default: ''} | ||
- {name: Env, type: String, description: 'override function env var, Json {key: value, ..}', default: ''} | ||
- {name: Mount, type: String, description: 'volume mount, [vol-type:]<vol-url>:<dst-path>', default: ''} | ||
outputs: | ||
- {name: Endpoint, type: String, description: 'function endpoint url'} | ||
implementation: | ||
container: | ||
image: nuclio/pydeploy | ||
command: [ | ||
python, -m, nuclio, deploy, {inputValue: Url}, | ||
--dashboard-url, {inputValue: Dashboard}, | ||
--name, {inputValue: Name}, | ||
--project, {inputValue: Project}, | ||
--tag, {inputValue: Tag}, | ||
--env-json, {inputValue: Env}, | ||
--spec-json, {inputValue: Spec}, | ||
--mount, {inputValue: Mount}, | ||
] | ||
fileOutputs: | ||
Endpoint: /tmp/output |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
name: nuclio invoker | ||
description: invoke nuclio function. | ||
inputs: | ||
- {name: Url, type: String, description: 'function URL/endpoint'} | ||
- {name: Body, type: String, description: 'request body', default: ''} | ||
- {name: Log level, type: String, description: 'log level', default: 'info'} | ||
- {name: Method, type: String, description: 'HTTP method GET|POST|..', default: ''} | ||
outputs: | ||
- {name: output, type: String, description: 'function output'} | ||
implementation: | ||
container: | ||
image: nuclio/invoker | ||
command: [ | ||
invoke, | ||
-a, {inputValue: Url}, | ||
-b, {inputValue: Body}, | ||
-m, {inputValue: Method}, | ||
-l, {inputValue: Log level}, | ||
] | ||
fileOutputs: | ||
output: /tmp/output |