ray/release/rllib_tests
Kai Fricke 6c5229295e
[ci/release] Support running tests with different python versions (#24843)
OSS release tests currently run with hardcoded Python 3.7 base. In the future we will want to run tests on different python versions. 
This PR adds support for a new `python` field in the test configuration. The python field will determine both the base image used in the Buildkite runner docker container (for Ray client compatibility) and the base image for the Anyscale cluster environments. 

Note that in Buildkite, we will still only wait for the python 3.7 base image before kicking off tests. That is acceptable, as we can assume that most wheels finish in a similar time, so even if we wait for the 3.7 image and kick off a 3.8 test, that runner will wait maybe for 5-10 more minutes.
2022-05-17 17:03:12 +01:00
..
learning_tests [RLlib] Replay Buffer API and Ape-X. (#24506) 2022-05-17 13:43:49 +02:00
multi_gpu_learning_tests [RLlib] Replay Buffer API and Ape-X. (#24506) 2022-05-17 13:43:49 +02:00
multi_gpu_with_attention_learning_tests [CI] Format Python code with Black (#21975) 2022-01-29 18:41:57 -08:00
multi_gpu_with_lstm_learning_tests [CI] Format Python code with Black (#21975) 2022-01-29 18:41:57 -08:00
stress_tests [CI] Format Python code with Black (#21975) 2022-01-29 18:41:57 -08:00
unit_gpu_tests [ci] Clean up ci/ directory (refactor ci/travis) (#23866) 2022-04-13 18:11:30 +01:00
1gpu_4cpus.yaml [RLlib] Add an RLlib Tune experiment to UserTest suite. (#19807) 2021-11-03 17:04:27 -07:00
2gpus_32cpus.yaml [RLlib] Add an RLlib Tune experiment to UserTest suite. (#19807) 2021-11-03 17:04:27 -07:00
4gpus_64cpus.yaml [RLlib] Add an RLlib Tune experiment to UserTest suite. (#19807) 2021-11-03 17:04:27 -07:00
4gpus_544_cpus.yaml [Tune release test] Set 500G disk space for rllib_tests. (#19730) 2021-10-26 10:12:03 -07:00
8gpus_64cpus.yaml [Tune release test] Set 500G disk space for rllib_tests. (#19730) 2021-10-26 10:12:03 -07:00
8gpus_96cpus.yaml [Tune release test] Set 500G disk space for rllib_tests. (#19730) 2021-10-26 10:12:03 -07:00
12gpus_192cpus.yaml [RLlib] Create a set of performance benchmark tests to run nightly. (#19945) 2021-11-08 18:15:13 +01:00
app_config.yaml [ci/release] Support running tests with different python versions (#24843) 2022-05-17 17:03:12 +01:00
README.rst [RLlib] Upgrade RLlib regression test scripts to new testing tool - RLlib release logs for 1.4. (#16080) 2021-06-01 17:39:18 +02:00

RLlib Tests
===========

This directory contains various RLlib release tests.

You should run these tests with the `releaser <https://github.com/ray-project/releaser>`_ tool.

Overview
--------
Currently, there are 3 RLlib tests:

1. ``learning_tests`` - Tests, whether major algos (tf+torch) can learn in Atari or PyBullet envs in ~30-60min.
1. ``stress_tests`` - Runs 4 IMPALA Atari jobs, each one using 1GPU and 128CPUs (needs autoscaling to succeed).
1. ``unit_gpu_tests`` - Tests, whether all of RLlib's example scripts can be run on a GPU.

Generally the releaser tool will run all tests in parallel.

Acceptance criteria
-------------------
These tests are considered passing when they throw no error at the end of the output log.