Update release information from 0.8.6. (#9124)

* [BRING BACK TO MASTER] Update release information. * [MERGE TO MASTER] Add microbenchmark result. * Update asan tests to the doc.
2025-03-05 18:11:42 -05:00 · 2020-07-24 10:04:07 -07:00 · 2020-07-24 10:04:07 -07:00 · 18981bad59
commit 18981bad59
parent 58a38e81d1
7 changed files with 105 additions and 3 deletions
--- a/doc/dev/RELEASE_PROCESS.rst
+++ b/doc/dev/RELEASE_PROCESS.rst
@ -49,7 +49,7 @@ This document describes the process for creating new releases.
   These suite of tests are similar to the standard long running tests, except these actually run in a multi-node cluster instead of just a simulated one.
   These tests should also run for at least 24 hours without erroring or hanging.

-   2. Multi-node regression tests
+   3. Multi-node regression tests

   Follow the same instruction as long running stress tests. The large scale distributed
   regression tests identify potential performance regression in distributed environment.
@ -68,7 +68,7 @@ This document describes the process for creating new releases.
   The summaries printed by each test should be checked in under
   ``doc/dev/release_logs/<version>``.

-   3. Microbenchmarks
+   4. Microbenchmarks

   Run the ``ci/microbenchmark`` with the commit. Under the hood, the session will
   run `ray microbenchmark` on an `m4.16xl` instance running `Ubuntu 18.04` with `Python 3`
@ -79,6 +79,11 @@ This document describes the process for creating new releases.
   You can also get the performance change rate from the previous version using
   microbenchmark_analysis.py

+   5. ASAN tests
+
+   Run the ``ci/asan_tests`` with the commit. This will enable ASAN build and run the
+   whole Python tests to detect memory leaks.
+
 5. **Resolve release-blockers:** If a release blocking issue arises, there are
   two ways the issue can be resolved: 1) Fix the issue on the master branch and
   cherry-pick the relevant commit  (using ``git cherry-pick``) onto the release
@ -160,12 +165,13 @@ This document describes the process for creating new releases.
   scripts. Make sure that it is finding the version of Ray that you just
   installed by checking ``ray.__version__`` and ``ray.__file__``.

-   Do this at least for MacOS and Linux.
+   Do this for MacOS, Linux, and Windows.

   This process is automated. Run ./bin/pip_download_test.sh.
   This will download the ray from the test pypi repository and run the minimum
   sanity check from all the Python version supported. (3.6, 3.7, 3.8)

+   Windows sanity check test is currently not automated.

 9. **Upload to PyPI:** Now that you've tested the wheels on the PyPI test
   repository, they can be uploaded to the main PyPI repository. Be careful,
--- a/doc/dev/download_wheels.sh
+++ b/doc/dev/download_wheels.sh
@ -21,3 +21,7 @@ wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_H
 wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-macosx_10_13_intel.whl"
 # Wheel name convention has been changed from Python 3.8.
 wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp38-cp38-macosx_10_13_x86_64.whl"
+# Make sure Windows wheels are downloadable without errors.
+wget https://ray-wheels.s3-us-west-2.amazonaws.com/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp36-cp36m-win_amd64.whl
+wget https://ray-wheels.s3-us-west-2.amazonaws.com/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-win_amd64.whl
+wget https://ray-wheels.s3-us-west-2.amazonaws.com/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp38-cp38-win_amd64.whl
--- a/doc/dev/release_logs/0.8.6/microbenchmark.txt
+++ b/doc/dev/release_logs/0.8.6/microbenchmark.txt
@ -0,0 +1,19 @@
+single client get calls per second 8746.49 +- 454.49
+single client put calls per second 5449.42 +- 86.57
+single client put gigabytes per second 13.59 +- 3.27
+multi client put calls per second 11662.1 +- 153.21
+multi client put gigabytes per second 17.26 +- 7.94
+single client tasks sync per second 1272.57 +- 33.26
+single client tasks async per second 14621.87 +- 200.32
+multi client tasks async per second 41588.79 +- 4142.82
+1:1 actor calls sync per second 1914.85 +- 20.58
+1:1 actor calls async per second 6541.67 +- 127.36
+1:1 actor calls concurrent per second 5663.49 +- 155.04
+1:n actor calls async per second 10495.3 +- 133.15
+n:n actor calls async per second 34453.12 +- 329.25
+n:n actor calls with arg async per second 12500.39 +- 167.03
+1:1 async-actor calls sync per second 1291.48 +- 48.9
+1:1 async-actor calls async per second 3510.1 +- 48.84
+1:1 async-actor calls with args async per second 2321.3 +- 70.18
+1:n async-actor calls async per second 8961.23 +- 111.43
+n:n async-actor calls async per second 27483.48 +- 113.91
--- a/doc/dev/release_logs/0.8.6/rllib_regression.txt
+++ b/doc/dev/release_logs/0.8.6/rllib_regression.txt
@ -0,0 +1,39 @@
+== Status ==
+Memory usage on this node: 14.3/480.3 GiB
+Using FIFO scheduling algorithm.
+Resources requested: 0/64 CPUs, 0.0/8 GPUs, 0.0/323.49 GiB heap, 0.0/98.39 GiB objects
+Result logdir: /home/ubuntu/ray_results/apex
+Result logdir: /home/ubuntu/ray_results/atari-a2c
+Result logdir: /home/ubuntu/ray_results/atari-basic-dqn
+Result logdir: /home/ubuntu/ray_results/atari-impala
+Result logdir: /home/ubuntu/ray_results/atari-ppo-tf
+Result logdir: /home/ubuntu/ray_results/atari-ppo-torch
+Number of trials: 24 (24 TERMINATED)
+-------------------------------------------+------------+-------+--------+------------------+---------+----------+
+| Trial name                                | status     | loc   |   iter |   total time (s) |      ts |   reward |
+|-------------------------------------------+------------+-------+--------+------------------+---------+----------|
+| IMPALA_BreakoutNoFrameskip-v4_926d2_00000 | TERMINATED |       |    357 |          3601.43 | 7523500 |   315.74 |
+| IMPALA_BreakoutNoFrameskip-v4_926d2_00001 | TERMINATED |       |    356 |          3603.43 | 7455000 |   340.84 |
+| IMPALA_BreakoutNoFrameskip-v4_926d2_00002 | TERMINATED |       |    357 |          3601.72 | 7622250 |   306.88 |
+| IMPALA_BreakoutNoFrameskip-v4_926d2_00003 | TERMINATED |       |    357 |          3601.02 | 7561500 |   331.75 |
+| PPO_BreakoutNoFrameskip-v4_926d2_00004    | TERMINATED |       |    692 |          3600.28 | 3460000 |    28.44 |
+| PPO_BreakoutNoFrameskip-v4_926d2_00005    | TERMINATED |       |    946 |          3600.06 | 4730000 |    53.61 |
+| PPO_BreakoutNoFrameskip-v4_926d2_00006    | TERMINATED |       |    943 |          3602.39 | 4715000 |    37.03 |
+| PPO_BreakoutNoFrameskip-v4_926d2_00007    | TERMINATED |       |    944 |          3602.4  | 4720000 |    36.35 |
+| PPO_BreakoutNoFrameskip-v4_926d2_00008    | TERMINATED |       |     27 |          3600.14 |  135000 |     2.02 |
+| PPO_BreakoutNoFrameskip-v4_926d2_00009    | TERMINATED |       |     27 |          3723.31 |  135000 |     5.37 |
+| PPO_BreakoutNoFrameskip-v4_926d2_00010    | TERMINATED |       |     21 |          3704.86 |  105000 |     1.5  |
+| PPO_BreakoutNoFrameskip-v4_926d2_00011    | TERMINATED |       |     22 |          3630.76 |  110000 |     1.98 |
+| APEX_BreakoutNoFrameskip-v4_926d2_00012   | TERMINATED |       |    106 |          3607.17 | 5372960 |    74.44 |
+| APEX_BreakoutNoFrameskip-v4_926d2_00013   | TERMINATED |       |    107 |          3603.72 | 8074080 |    87.35 |
+| APEX_BreakoutNoFrameskip-v4_926d2_00014   | TERMINATED |       |     90 |          3633.64 | 6787680 |    59.02 |
+| APEX_BreakoutNoFrameskip-v4_926d2_00015   | TERMINATED |       |     90 |          3616.73 | 6784160 |    61.9  |
+| A2C_BreakoutNoFrameskip-v4_926d2_00016    | TERMINATED |       |    354 |          3603.13 | 3347000 |    74.53 |
+| A2C_BreakoutNoFrameskip-v4_926d2_00017    | TERMINATED |       |    353 |          3601.95 | 3374000 |   174.69 |
+| A2C_BreakoutNoFrameskip-v4_926d2_00018    | TERMINATED |       |    353 |          3601.02 | 3377000 |   149.69 |
+| A2C_BreakoutNoFrameskip-v4_926d2_00019    | TERMINATED |       |    353 |          3605.72 | 3732000 |   203.06 |
+| DQN_BreakoutNoFrameskip-v4_926d2_00020    | TERMINATED |       |     27 |          3695.93 |  280000 |    14.71 |
+| DQN_BreakoutNoFrameskip-v4_926d2_00021    | TERMINATED |       |     26 |          3613.54 |  270000 |    16.37 |
+| DQN_BreakoutNoFrameskip-v4_926d2_00022    | TERMINATED |       |     27 |          3661.86 |  280000 |    15.43 |
+| DQN_BreakoutNoFrameskip-v4_926d2_00023    | TERMINATED |       |     27 |          3680.85 |  280000 |    15.62 |
+-------------------------------------------+------------+-------+--------+------------------+---------+----------+
--- a/doc/dev/release_logs/0.8.6/stress_tests/application_stress_test.txt
+++ b/doc/dev/release_logs/0.8.6/stress_tests/application_stress_test.txt
@ -0,0 +1,14 @@
+== Status ==
+Memory usage on this node: 19.9/480.3 GiB
+Using FIFO scheduling algorithm.
+Resources requested: 0/640 CPUs, 0/8 GPUs, 0.0/1906.49 GiB heap, 0.0/566.21 GiB objects
+Result logdir: /home/ubuntu/ray_results/atari-impala
+Number of trials: 4 (4 TERMINATED)
+------------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------+
+| Trial name                                     | status     | loc   | env                         |   iter |   total time (s) |       ts |   reward |
+|------------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------|
+| IMPALA_BreakoutNoFrameskip-v4_a4e31_00000      | TERMINATED |       | BreakoutNoFrameskip-v4      |    680 |          6847.72 | 30047500 |   358.37 |
+| IMPALA_BeamRiderNoFrameskip-v4_a4e31_00001     | TERMINATED |       | BeamRiderNoFrameskip-v4     |    694 |          6987.42 | 30001500 |  2657.04 |
+| IMPALA_QbertNoFrameskip-v4_a4e31_00002         | TERMINATED |       | QbertNoFrameskip-v4         |    676 |          6821.02 | 30016500 |  3606.75 |
+| IMPALA_SpaceInvadersNoFrameskip-v4_a4e31_00003 | TERMINATED |       | SpaceInvadersNoFrameskip-v4 |    684 |          6889.83 | 30025000 |   653.4  |
+------------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------+
--- a/doc/dev/release_logs/0.8.6/stress_tests/test_dead_actors.txt
+++ b/doc/dev/release_logs/0.8.6/stress_tests/test_dead_actors.txt
@ -0,0 +1,4 @@
+Finished in: 457.8229465484619s
+Average iteration time: 4.578227622509003s
+Max iteration time: 28.17513918876648s
+Min iteration time: 0.31386494636535645s
--- a/doc/dev/release_logs/0.8.6/stress_tests/test_many_tasks.txt
+++ b/doc/dev/release_logs/0.8.6/stress_tests/test_many_tasks.txt
@ -0,0 +1,16 @@
+Stage 0 results:
+	Total time: 21.636702299118042
+Stage 1 results:
+	Total time: 150.684504032135
+	Average iteration time: 15.068440675735474
+	Max iteration time: 16.537473440170288
+	Min iteration time: 14.317448616027832
+Stage 2 results:
+	Total time: 1103.2014956474304
+	Average iteration time: 220.63988456726074
+	Max iteration time: 229.98777270317078
+	Min iteration time: 203.2540054321289
+Stage 3 results:
+	Actor creation time: 0.09029483795166016
+	Total time: 3138.855129480362
+