Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update release information. #9124

Merged
merged 4 commits into from
Jul 24, 2020
Merged

Update release information. #9124

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f393597
[BRING BACK TO MASTER] Update release information.
rkooo567 Jun 24, 2020
d5e8ade
[MERGE TO MASTER] Add microbenchmark result.
rkooo567 Jun 9, 2020
84a7b9b
Merge branch 'master' into release-update
rkooo567 Jul 24, 2020
9d0bc75
Update asan tests to the doc.
rkooo567 Jul 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions doc/dev/RELEASE_PROCESS.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ This document describes the process for creating new releases.
These suite of tests are similar to the standard long running tests, except these actually run in a multi-node cluster instead of just a simulated one.
These tests should also run for at least 24 hours without erroring or hanging.

2. Multi-node regression tests
3. Multi-node regression tests

Follow the same instruction as long running stress tests. The large scale distributed
regression tests identify potential performance regression in distributed environment.
Expand All @@ -68,7 +68,7 @@ This document describes the process for creating new releases.
The summaries printed by each test should be checked in under
``doc/dev/release_logs/<version>``.

3. Microbenchmarks
4. Microbenchmarks

Run the ``ci/microbenchmark`` with the commit. Under the hood, the session will
run `ray microbenchmark` on an `m4.16xl` instance running `Ubuntu 18.04` with `Python 3`
Expand All @@ -79,6 +79,11 @@ This document describes the process for creating new releases.
You can also get the performance change rate from the previous version using
microbenchmark_analysis.py

5. ASAN tests

Run the ``ci/asan_tests`` with the commit. This will enable ASAN build and run the
whole Python tests to detect memory leaks.

5. **Resolve release-blockers:** If a release blocking issue arises, there are
two ways the issue can be resolved: 1) Fix the issue on the master branch and
cherry-pick the relevant commit (using ``git cherry-pick``) onto the release
Expand Down Expand Up @@ -160,12 +165,13 @@ This document describes the process for creating new releases.
scripts. Make sure that it is finding the version of Ray that you just
installed by checking ``ray.__version__`` and ``ray.__file__``.

Do this at least for MacOS and Linux.
Do this for MacOS, Linux, and Windows.

This process is automated. Run ./bin/pip_download_test.sh.
This will download the ray from the test pypi repository and run the minimum
sanity check from all the Python version supported. (3.6, 3.7, 3.8)

Windows sanity check test is currently not automated.

9. **Upload to PyPI:** Now that you've tested the wheels on the PyPI test
repository, they can be uploaded to the main PyPI repository. Be careful,
Expand Down
4 changes: 4 additions & 0 deletions doc/dev/download_wheels.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,7 @@ wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_H
wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-macosx_10_13_intel.whl"
# Wheel name convention has been changed from Python 3.8.
wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp38-cp38-macosx_10_13_x86_64.whl"
# Make sure Windows wheels are downloadable without errors.
wget https://ray-wheels.s3-us-west-2.amazonaws.com/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp36-cp36m-win_amd64.whl
wget https://ray-wheels.s3-us-west-2.amazonaws.com/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-win_amd64.whl
wget https://ray-wheels.s3-us-west-2.amazonaws.com/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp38-cp38-win_amd64.whl
19 changes: 19 additions & 0 deletions doc/dev/release_logs/0.8.6/microbenchmark.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
single client get calls per second 8746.49 +- 454.49
single client put calls per second 5449.42 +- 86.57
single client put gigabytes per second 13.59 +- 3.27
multi client put calls per second 11662.1 +- 153.21
multi client put gigabytes per second 17.26 +- 7.94
single client tasks sync per second 1272.57 +- 33.26
single client tasks async per second 14621.87 +- 200.32
multi client tasks async per second 41588.79 +- 4142.82
1:1 actor calls sync per second 1914.85 +- 20.58
1:1 actor calls async per second 6541.67 +- 127.36
1:1 actor calls concurrent per second 5663.49 +- 155.04
1:n actor calls async per second 10495.3 +- 133.15
n:n actor calls async per second 34453.12 +- 329.25
n:n actor calls with arg async per second 12500.39 +- 167.03
1:1 async-actor calls sync per second 1291.48 +- 48.9
1:1 async-actor calls async per second 3510.1 +- 48.84
1:1 async-actor calls with args async per second 2321.3 +- 70.18
1:n async-actor calls async per second 8961.23 +- 111.43
n:n async-actor calls async per second 27483.48 +- 113.91
39 changes: 39 additions & 0 deletions doc/dev/release_logs/0.8.6/rllib_regression.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
== Status ==
Memory usage on this node: 14.3/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/64 CPUs, 0.0/8 GPUs, 0.0/323.49 GiB heap, 0.0/98.39 GiB objects
Result logdir: /home/ubuntu/ray_results/apex
Result logdir: /home/ubuntu/ray_results/atari-a2c
Result logdir: /home/ubuntu/ray_results/atari-basic-dqn
Result logdir: /home/ubuntu/ray_results/atari-impala
Result logdir: /home/ubuntu/ray_results/atari-ppo-tf
Result logdir: /home/ubuntu/ray_results/atari-ppo-torch
Number of trials: 24 (24 TERMINATED)
+-------------------------------------------+------------+-------+--------+------------------+---------+----------+
| Trial name | status | loc | iter | total time (s) | ts | reward |
|-------------------------------------------+------------+-------+--------+------------------+---------+----------|
| IMPALA_BreakoutNoFrameskip-v4_926d2_00000 | TERMINATED | | 357 | 3601.43 | 7523500 | 315.74 |
| IMPALA_BreakoutNoFrameskip-v4_926d2_00001 | TERMINATED | | 356 | 3603.43 | 7455000 | 340.84 |
| IMPALA_BreakoutNoFrameskip-v4_926d2_00002 | TERMINATED | | 357 | 3601.72 | 7622250 | 306.88 |
| IMPALA_BreakoutNoFrameskip-v4_926d2_00003 | TERMINATED | | 357 | 3601.02 | 7561500 | 331.75 |
| PPO_BreakoutNoFrameskip-v4_926d2_00004 | TERMINATED | | 692 | 3600.28 | 3460000 | 28.44 |
| PPO_BreakoutNoFrameskip-v4_926d2_00005 | TERMINATED | | 946 | 3600.06 | 4730000 | 53.61 |
| PPO_BreakoutNoFrameskip-v4_926d2_00006 | TERMINATED | | 943 | 3602.39 | 4715000 | 37.03 |
| PPO_BreakoutNoFrameskip-v4_926d2_00007 | TERMINATED | | 944 | 3602.4 | 4720000 | 36.35 |
| PPO_BreakoutNoFrameskip-v4_926d2_00008 | TERMINATED | | 27 | 3600.14 | 135000 | 2.02 |
| PPO_BreakoutNoFrameskip-v4_926d2_00009 | TERMINATED | | 27 | 3723.31 | 135000 | 5.37 |
| PPO_BreakoutNoFrameskip-v4_926d2_00010 | TERMINATED | | 21 | 3704.86 | 105000 | 1.5 |
| PPO_BreakoutNoFrameskip-v4_926d2_00011 | TERMINATED | | 22 | 3630.76 | 110000 | 1.98 |
| APEX_BreakoutNoFrameskip-v4_926d2_00012 | TERMINATED | | 106 | 3607.17 | 5372960 | 74.44 |
| APEX_BreakoutNoFrameskip-v4_926d2_00013 | TERMINATED | | 107 | 3603.72 | 8074080 | 87.35 |
| APEX_BreakoutNoFrameskip-v4_926d2_00014 | TERMINATED | | 90 | 3633.64 | 6787680 | 59.02 |
| APEX_BreakoutNoFrameskip-v4_926d2_00015 | TERMINATED | | 90 | 3616.73 | 6784160 | 61.9 |
| A2C_BreakoutNoFrameskip-v4_926d2_00016 | TERMINATED | | 354 | 3603.13 | 3347000 | 74.53 |
| A2C_BreakoutNoFrameskip-v4_926d2_00017 | TERMINATED | | 353 | 3601.95 | 3374000 | 174.69 |
| A2C_BreakoutNoFrameskip-v4_926d2_00018 | TERMINATED | | 353 | 3601.02 | 3377000 | 149.69 |
| A2C_BreakoutNoFrameskip-v4_926d2_00019 | TERMINATED | | 353 | 3605.72 | 3732000 | 203.06 |
| DQN_BreakoutNoFrameskip-v4_926d2_00020 | TERMINATED | | 27 | 3695.93 | 280000 | 14.71 |
| DQN_BreakoutNoFrameskip-v4_926d2_00021 | TERMINATED | | 26 | 3613.54 | 270000 | 16.37 |
| DQN_BreakoutNoFrameskip-v4_926d2_00022 | TERMINATED | | 27 | 3661.86 | 280000 | 15.43 |
| DQN_BreakoutNoFrameskip-v4_926d2_00023 | TERMINATED | | 27 | 3680.85 | 280000 | 15.62 |
+-------------------------------------------+------------+-------+--------+------------------+---------+----------+
14 changes: 14 additions & 0 deletions doc/dev/release_logs/0.8.6/stress_tests/application_stress_test.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
== Status ==
Memory usage on this node: 19.9/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/640 CPUs, 0/8 GPUs, 0.0/1906.49 GiB heap, 0.0/566.21 GiB objects
Result logdir: /home/ubuntu/ray_results/atari-impala
Number of trials: 4 (4 TERMINATED)
+------------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------+
| Trial name | status | loc | env | iter | total time (s) | ts | reward |
|------------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------|
| IMPALA_BreakoutNoFrameskip-v4_a4e31_00000 | TERMINATED | | BreakoutNoFrameskip-v4 | 680 | 6847.72 | 30047500 | 358.37 |
| IMPALA_BeamRiderNoFrameskip-v4_a4e31_00001 | TERMINATED | | BeamRiderNoFrameskip-v4 | 694 | 6987.42 | 30001500 | 2657.04 |
| IMPALA_QbertNoFrameskip-v4_a4e31_00002 | TERMINATED | | QbertNoFrameskip-v4 | 676 | 6821.02 | 30016500 | 3606.75 |
| IMPALA_SpaceInvadersNoFrameskip-v4_a4e31_00003 | TERMINATED | | SpaceInvadersNoFrameskip-v4 | 684 | 6889.83 | 30025000 | 653.4 |
+------------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------+
4 changes: 4 additions & 0 deletions doc/dev/release_logs/0.8.6/stress_tests/test_dead_actors.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Finished in: 457.8229465484619s
Average iteration time: 4.578227622509003s
Max iteration time: 28.17513918876648s
Min iteration time: 0.31386494636535645s
16 changes: 16 additions & 0 deletions doc/dev/release_logs/0.8.6/stress_tests/test_many_tasks.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
Stage 0 results:
Total time: 21.636702299118042
Stage 1 results:
Total time: 150.684504032135
Average iteration time: 15.068440675735474
Max iteration time: 16.537473440170288
Min iteration time: 14.317448616027832
Stage 2 results:
Total time: 1103.2014956474304
Average iteration time: 220.63988456726074
Max iteration time: 229.98777270317078
Min iteration time: 203.2540054321289
Stage 3 results:
Actor creation time: 0.09029483795166016
Total time: 3138.855129480362