Skip to content

Add example of creating Google cloud cluster via commandline. #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add example of creating Google cloud cluster via commandline. #1

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 15 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,20 @@ It uses Google Cloud Storage bucket to store your notebook for persistence so th
0. Create a GCS bucket for storing your notebooks
1. Change `c.GoogleStorageContentManager.default_path` in `jupyter-config.py` to your GCS path
3. Create a GKE cluster of your choice (Recommend 2CPU 7.5G or larger each node), make sure turn on **legacy authorisation mode**
4. `kubectl apply -f ./kube/`
5. Connect to service using port forwarding `kubectl port-forward svc/svc-notebooks 8888:8888`, or use the public ip from `kubectl get svc`
6. Start using cluster!

For example the following bash script:

```
project_name=brightwrite-211800
zone_name=us-west1-b
zone_name2=us-west1
gcloud beta container --project "${project_name}" clusters create "jupyter-1" --zone "${zone_name}" --username "admin" --cluster-version "1.9.7-gke.6" --machine-type "n1-highcpu-64" --image-type "COS" --disk-type "pd-ssd" --disk-size "100" --scopes "https://www.googleapis.com/auth/cloud-platform" --min-cpu-platform "Intel Skylake" --num-nodes "1" --enable-cloud-logging --enable-cloud-monitoring --network "projects/${project_name}/global/networks/default" --subnetwork "projects/${project_name}/regions/${zone_name2}/subnetworks/default" --enable-legacy-authorization --addons HorizontalPodAutoscaling,HttpLoadBalancing --no-enable-autoupgrade --enable-autorepair
```
4. Make sure the container image specified in kube/30-deployment.yaml is as desired. This docker image will be deployed to the workers.
5. Check that the limits specified in kube/30-deployment.yamls is consistent with your cluster definition above in terms of number of CPUs, memory, etc.
6. Run `kubectl apply -f ./kube/ --validate=false`
7. Connect to service using port forwarding `kubectl port-forward svc/svc-notebooks 8888:8888`, or use the public ip from `kubectl get svc`
8. Start using cluster!
```
from dask_kubernetes import KubeCluster

Expand All @@ -28,4 +39,4 @@ It uses Google Cloud Storage bucket to store your notebook for persistence so th

1. Change the `Dockerfile`, build your image, and push it to any of the image storage service.
2. Change the image name in `30-deployment.yaml` file
3. Apply your kubernetes configuration
3. Apply your kubernetes configuration