Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[autoscaler] Kubernetes autoscaler backend #5492

Merged
merged 48 commits into from
Oct 3, 2019
Merged

[autoscaler] Kubernetes autoscaler backend #5492

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
86d9c6f
Add Kubernetes NodeProvider to autoscaler
edoakes Aug 12, 2019
be1b96b
Split off SSHCommandRunner
edoakes Aug 19, 2019
ffe00a6
Add KubernetesCommandRunner
edoakes Aug 20, 2019
0915df9
Cleanup
edoakes Aug 22, 2019
4558a76
More config options
edoakes Aug 22, 2019
15d1856
Check if auth present
edoakes Aug 22, 2019
8ed1ed9
More auth checks
edoakes Aug 22, 2019
8ec5f23
Better output
edoakes Aug 22, 2019
3c454c2
Always bootstrap config
edoakes Aug 22, 2019
56e3bf0
All working
edoakes Aug 23, 2019
4379355
Add k8s-rsync comment
edoakes Aug 23, 2019
44327bb
Clean up manual k8s examples
edoakes Aug 23, 2019
2d6fcc3
Fix up submit.yaml
edoakes Aug 23, 2019
9c98da7
Automatically configure permissisons
edoakes Aug 24, 2019
a7265ad
Fix get_node_provider arg
edoakes Aug 24, 2019
3174aa1
Fix permissions
edoakes Aug 24, 2019
aa13c22
Fill in empty auth
edoakes Aug 26, 2019
158876f
Merge branch 'master' into k8s
edoakes Aug 26, 2019
6ff3c2a
Remove ray-cluster from this PR
edoakes Aug 26, 2019
c218965
No hard dep on kubernetes library
edoakes Aug 26, 2019
f904982
Move permissions into autoscaler config
edoakes Aug 26, 2019
2452e1b
lint
edoakes Aug 26, 2019
a4e16d6
Fix indentation
edoakes Aug 26, 2019
e13f4a1
namespace validation
edoakes Aug 28, 2019
0fc7a0f
Use cluster name tag
edoakes Aug 28, 2019
9dd310f
Remove kubernetes from setup.py
edoakes Aug 28, 2019
da0ad01
Comment in example configs
edoakes Aug 28, 2019
34b6592
Same default autoscaling config as aws
edoakes Aug 28, 2019
8a0bb24
Add Kubernetes quickstart
edoakes Aug 28, 2019
6df676c
lint
edoakes Aug 28, 2019
15d3573
Revert changes to submit.yaml (other PR)
edoakes Aug 28, 2019
c610f34
Install kubernetes in travis
edoakes Aug 29, 2019
0cc97d6
address comments
edoakes Aug 29, 2019
e1c6fa2
Improve autoscaling doc
edoakes Aug 29, 2019
3bb44d6
Merge remote-tracking branch 'upstream/master' into k8s
edoakes Aug 29, 2019
2e449ea
kubectl command in setup
edoakes Sep 3, 2019
8fa6126
Force use_internal_ips
edoakes Sep 3, 2019
6559635
Merge remote-tracking branch 'upstream/master' into k8s
edoakes Sep 3, 2019
3fdb850
comments
edoakes Sep 3, 2019
c01d471
backend env in docs
edoakes Sep 4, 2019
670d819
Merge remote-tracking branch 'upstream/master' into k8s
edoakes Sep 4, 2019
86d5cc4
Change namespace config
edoakes Sep 4, 2019
1b80017
comments
edoakes Sep 18, 2019
16adb26
Merge remote-tracking branch 'upstream/master' into k8s
edoakes Sep 18, 2019
e88ec5f
comments
edoakes Sep 18, 2019
fee41e4
Merge branch 'master' into k8s
edoakes Sep 20, 2019
734320a
Merge branch 'master' into k8s
edoakes Oct 2, 2019
b29d172
Fix yaml test
edoakes Oct 3, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge remote-tracking branch 'upstream/master' into k8s
  • Loading branch information
@edoakes
edoakes committed Sep 3, 2019
commit 655963553cdaeee7e80ac8f05e402e3f1fdc9ede
8 changes: 4 additions & 4 deletions ci/travis/install-dependencies.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ if [[ "$PYTHON" == "2.7" ]] && [[ "$platform" == "linux" ]]; then
bash miniconda.sh -b -p $HOME/miniconda
export PATH="$HOME/miniconda/bin:$PATH"
pip install -q scipy tensorflow cython==0.29.0 gym opencv-python-headless pyyaml pandas==0.24.2 requests \
feather-format lxml openpyxl xlrd py-spy setproctitle faulthandler pytest-timeout mock flaky networkx tabulate psutil kubernetes
feather-format lxml openpyxl xlrd py-spy setproctitle faulthandler pytest-timeout pytest-sugar mock flaky networkx tabulate psutil kubernetes
elif [[ "$PYTHON" == "3.5" ]] && [[ "$platform" == "linux" ]]; then
sudo apt-get update
sudo apt-get install -y python-dev python-numpy build-essential curl unzip tmux gdb
Expand All @@ -34,21 +34,21 @@ elif [[ "$PYTHON" == "3.5" ]] && [[ "$platform" == "linux" ]]; then
bash miniconda.sh -b -p $HOME/miniconda
export PATH="$HOME/miniconda/bin:$PATH"
pip install -q scipy tensorflow cython==0.29.0 gym opencv-python-headless pyyaml pandas==0.24.2 requests \
feather-format lxml openpyxl xlrd py-spy setproctitle pytest-timeout flaky networkx tabulate psutil aiohttp kubernetes
feather-format lxml openpyxl xlrd py-spy setproctitle pytest-timeout pytest-sugar flaky networkx tabulate psutil aiohttp kubernetes
elif [[ "$PYTHON" == "2.7" ]] && [[ "$platform" == "macosx" ]]; then
# Install miniconda.
wget https://repo.continuum.io/miniconda/Miniconda2-4.5.4-MacOSX-x86_64.sh -O miniconda.sh -nv
bash miniconda.sh -b -p $HOME/miniconda
export PATH="$HOME/miniconda/bin:$PATH"
pip install -q cython==0.29.0 tensorflow gym opencv-python-headless pyyaml pandas==0.24.2 requests \
feather-format lxml openpyxl xlrd py-spy setproctitle faulthandler pytest-timeout mock flaky networkx tabulate psutil kubernetes
feather-format lxml openpyxl xlrd py-spy setproctitle faulthandler pytest-timeout pytest-sugar mock flaky networkx tabulate psutil kubernetes
elif [[ "$PYTHON" == "3.5" ]] && [[ "$platform" == "macosx" ]]; then
# Install miniconda.
wget https://repo.continuum.io/miniconda/Miniconda3-4.5.4-MacOSX-x86_64.sh -O miniconda.sh -nv
bash miniconda.sh -b -p $HOME/miniconda
export PATH="$HOME/miniconda/bin:$PATH"
pip install -q cython==0.29.0 tensorflow gym opencv-python-headless pyyaml pandas==0.24.2 requests \
feather-format lxml openpyxl xlrd py-spy setproctitle pytest-timeout flaky networkx tabulate psutil aiohttp kubernetes
feather-format lxml openpyxl xlrd py-spy setproctitle pytest-timeout pytest-sugar flaky networkx tabulate psutil aiohttp kubernetes
elif [[ "$LINT" == "1" ]]; then
sudo apt-get update
sudo apt-get install -y build-essential curl unzip
Expand Down
8 changes: 4 additions & 4 deletions doc/source/autoscaling.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Test that it works by running the following commands from your local machine:
# Get a remote screen on the head node.
$ ray attach ray/python/ray/autoscaler/aws/example-full.yaml
$ source activate tensorflow_p36
$ # Try running a Ray program with 'ray.init(redis_address="localhost:6379")'.
$ # Try running a Ray program with 'ray.init(address="localhost:6379")'.

# Tear down the cluster.
$ ray down ray/python/ray/autoscaler/aws/example-full.yaml
Expand All @@ -52,7 +52,7 @@ Test that it works by running the following commands from your local machine:
# Get a remote screen on the head node.
$ ray attach ray/python/ray/autoscaler/gcp/example-full.yaml
$ source activate tensorflow_p36
$ # Try running a Ray program with 'ray.init(redis_address="localhost:6379")'.
$ # Try running a Ray program with 'ray.init(address="localhost:6379")'.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
$ # Try running a Ray program with 'ray.init(address="localhost:6379")'.
$ # Try running a Ray program with 'ray.init(address="localhost:6379")'
$ # inside of a Python interpreter.

Same comment applies to GCP and Kubernetes and private cluster.


# Tear down the cluster.
$ ray down ray/python/ray/autoscaler/gcp/example-full.yaml
Expand Down Expand Up @@ -80,7 +80,7 @@ Test that it works by running the following commands from your local machine:

# Get a remote screen on the head node.
$ ray attach ray/python/ray/autoscaler/gcp/example-full.yaml
$ # Try running a Ray program with 'ray.init(redis_address="localhost:6379")'.
$ # Try running a Ray program with 'ray.init(address="localhost:6379")'.

# Tear down the cluster
$ ray down ray/python/ray/autoscaler/kubernetes/example-full.yaml
Expand All @@ -101,7 +101,7 @@ Test that it works by running the following commands from your local machine:

# Get a remote screen on the head node.
$ ray attach ray/python/ray/autoscaler/gcp/example-full.yaml
$ # Try running a Ray program with 'ray.init(redis_address="localhost:6379")'.
$ # Try running a Ray program with 'ray.init(address="localhost:6379")'.

# Tear down the cluster
$ ray down ray/python/ray/autoscaler/local/example-full.yaml
Expand Down
You are viewing a condensed version of this merge commit. You can view the full changes here.