Dockerized kernel is designed to run the kernel remotely, if you are interested in the implementation, please see the design document.
- Jupyter Notebook v4.4.0 or above (We do not test other versions now.)
- Kubernetes v1.9 or above
- Docker
Update ./hack/config.yaml
:
kubeconfig: {kubeconfig}
s2i:
provider: img
username: {input-your-username}
password: {input-your-pwd}
registry: {input-your-registry}
worker:
cpu: {default-worker-cpu}
memory: {default-worker-memory}
ps:
cpu: {default-ps-cpu}
memory: {default-ps-memory}
master:
cpu: {default-master-cpu}
memory: {default-master-memory}
If you choose img
or s2i
to provide the functionality of converting source code to image, username
and password
are necessary to push the image to the Docker registry. registry
is optional, default to https://index.docker.io/v1/
.
If you choose configmap
, then no need to input the username
, password
and registry
. The code will be mounted into the pod with the help of ConfigMap. Thus we recommend using configmap
as s2i provider.
Run the command:
docker build -t caicloud/ciao .
Install nb2kg
Jupyter Notebook Extension to Kernel Gateway (nb2kg) is used to communicate with the kernel in the docker image, please install the package using the command:
pip install nb2kg
Then enable it:
jupyter serverextension enable --py nb2kg --sys-prefix
Run the command:
docker run -v {kubeconfig}:{kubeconfig} -p 8889:8889 caicloud/ciao
Then you could run the notebook with nb2kg extension:
export KG_URL=http://127.0.0.1:8889
jupyter notebook \
--NotebookApp.session_manager_class=nb2kg.managers.SessionManager \
--NotebookApp.kernel_manager_class=nb2kg.managers.RemoteKernelManager \
--NotebookApp.kernel_spec_manager_class=nb2kg.managers.RemoteKernelSpecManager
- Jupyter Notebook v4.4.0 or above (We do not test other versions now.)
- Kubernetes v1.9 or above
- Kubeflow v0.4 or above
- S2I v1.1.10 or above
- img
- Docker
We do not release any version of the kernel, so please build it from the source, please see the Development Guide.
Please see Getting Started with Kubeflow. Currently, we only support one-node cluster since we do not push the image to the Docker Registry.
Run the script hack/install.sh
, then the specification of the kernel will be installed to ${HOME}/.local/share/jupyter/kernels/kubeflow
, then Jupyter will know the information about Ciao.
Then we need to create a configuration file $HOME/.ciao/config.yaml
:
kubeconfig: {path to your kubeconfig}
namespace: {Kubernetes namespace to create resources (TFJob, ConfigMap and so on), please make sure that the namespace exists in Kubernetes}
s2i:
provider: {img or s2i}
registry: {registry to be used to push images, optional}
username: {username}
password: {password}
worker:
cpu: {default-worker-cpu}
memory: {default-worker-memory}
ps:
cpu: {default-ps-cpu}
memory: {default-ps-memory}
master:
cpu: {default-master-cpu}
memory: {default-master-memory}
There are two options about tools to convert the source code in Jupyter Notebook to Docker image:
- img (Recommended), which is a daemon-free tool to build and push Docker images.
- s2i, which is a source to image tool.
For better performance, we recommend pulling the builder images from Docker Registry ahead of time. There are two builder images for different ML frameworks:
gaocegege/tensorflow-s2i:1.10.1-py3
gaocegege/pytorch-s2i:v0.2
tensorflow/tensorflow:1.10.1-py3
pytorch/pytorch:v0.2
Or the time of the first run will be extremely long (which depends on your network).
First, we need to set the environment variable KUBECONFIG
to tell the kernel where to find the kubeconfig:
export KUBECONFIG={path to your kubeconfig}
Then run Jupyter Notebook or Lab, choose Kubeflow kernel:
jupyter notebook