ray/release
Kai Fricke 331b71ea8d
[ci/release] Refactor release test e2e into package (#22351)
Adds a unit-tested and restructured ray_release package for running release tests.

Relevant changes in behavior:

Per default, Buildkite will wait for the wheels of the current commit to be available. Alternatively, users can a) specify a different commit hash, b) a wheels URL (which we will also wait for to be available) or c) specify a branch (or user/branch combination), in which case the latest available wheels will be used (e.g. if master is passed, behavior matches old default behavior).

The main subpackages are:

    Cluster manager: Creates cluster envs/computes, starts cluster, terminates cluster
    Command runner: Runs commands, e.g. as client command or sdk command
    File manager: Uploads/downloads files to/from session
    Reporter: Reports results (e.g. to database)

Much of the code base is unit tested, but there are probably some pieces missing.

Example build (waited for wheels to be built): https://buildkite.com/ray-project/kf-dev/builds/51#_
Wheel build: https://buildkite.com/ray-project/ray-builders-branch/builds/6023
2022-02-16 17:35:02 +00:00
..
.buildkite Add comment to periodic CI pipeline to update release process doc when updating test suites (#22037) 2022-02-11 11:14:24 -06:00
alerts Revert "[Nightly Test] Add more metadata to test result (#21990)" (#22052) 2022-02-02 12:56:42 -08:00
golden_notebook_tests [CI] Format Python code with Black (#21975) 2022-01-29 18:41:57 -08:00
horovod_tests [CI] Format Python code with Black (#21975) 2022-01-29 18:41:57 -08:00
kubernetes_manual_tests [test][k8s] Restore kubernetes test directory, adds some info (#18982) 2021-10-01 11:23:22 +01:00
lightgbm_tests [ci/release] Refactor release test e2e into package (#22351) 2022-02-16 17:35:02 +00:00
long_running_distributed_tests [CI] Format Python code with Black (#21975) 2022-01-29 18:41:57 -08:00
long_running_tests Fix a bug from many drivers. (#22248) 2022-02-09 15:17:15 -08:00
microbenchmark [CI] Format Python code with Black (#21975) 2022-01-29 18:41:57 -08:00
ml_user_tests [CI] Format Python code with Black (#21975) 2022-01-29 18:41:57 -08:00
nightly_tests It seems like the S3 read sometimes fails; #22214. I found out the file actually does exist in S3, so it is highly likely a transient error. This PR adds a retry mechanism to avoid the issue. 2022-02-12 11:58:58 +09:00
ray_release [ci/release] Refactor release test e2e into package (#22351) 2022-02-16 17:35:02 +00:00
release_logs [Release] Add release logs for 1.11.0rc0 (GCS KV & pubsub not enabled) (#22041) 2022-02-10 00:03:31 -08:00
rllib_tests [Release tests] Disable A3C CI tests on torch for now. Also extend performance_test deadline to 3hrs. (#22426) 2022-02-16 13:06:09 +01:00
runtime_env_tests [CI] Format Python code with Black (#21975) 2022-01-29 18:41:57 -08:00
serve_tests [CI] Format Python code with Black (#21975) 2022-01-29 18:41:57 -08:00
sgd_tests [CI] Format Python code with Black (#21975) 2022-01-29 18:41:57 -08:00
tune_tests [tune] Retry cloud sync up/down/delete on fail (#22029) 2022-02-15 12:27:29 +00:00
util [CI] Format Python code with Black (#21975) 2022-01-29 18:41:57 -08:00
xgboost_tests [ci/release] Refactor release test e2e into package (#22351) 2022-02-16 17:35:02 +00:00
__init__.py [release] move release testing end to end script to main ray repo (#17070) 2021-07-14 12:39:07 -07:00
alert.py [CI] Format Python code with Black (#21975) 2022-01-29 18:41:57 -08:00
BUILD [ci/release] Refactor release test e2e into package (#22351) 2022-02-16 17:35:02 +00:00
config_generator.html [release] move release testing end to end script to main ray repo (#17070) 2021-07-14 12:39:07 -07:00
e2e.py [Nightly Test] Addressed client failure properly (#22438) 2022-02-16 09:03:17 -08:00
README.md [Release] Remove release process doc (#19312) 2021-10-18 11:24:03 -07:00
release_tests.yaml [ci/release] Refactor release test e2e into package (#22351) 2022-02-16 17:35:02 +00:00
requirements.txt [ci/release] Refactor release test e2e into package (#22351) 2022-02-16 17:35:02 +00:00
requirements_buildkite.txt [ci/release] Refactor release test e2e into package (#22351) 2022-02-16 17:35:02 +00:00
run_e2e.sh [ci/release] Increase long running timeout, fix artifacts copy (#21905) 2022-01-26 21:25:03 +00:00
run_release_test.sh [ci/release] Refactor release test e2e into package (#22351) 2022-02-16 17:35:02 +00:00
setup.py [ci/release] Refactor release test e2e into package (#22351) 2022-02-16 17:35:02 +00:00
test_owners.yaml [ci] Add release test owner overview file (#18590) 2021-09-14 11:00:31 -07:00

Release Tests

While the exact release process relies on Anyscale internal tooling, the tests we run during the releases are located at https://github.com/ray-project/ray/blob/master/release/.buildkite/build_pipeline.py