Clark Zinzow
2cd3045b16
[Test Infra] Fix e2e.py help info for --report ( #21757 )
...
This momentarily confused me as to whether --report would enable or disable reporting.
2022-01-21 03:29:50 -08:00
SangBin Cho
b1308b1c8c
[Test Infra] Unrevert team col ( #21700 )
...
This fixes the previous problems from team column revert.
This has 2 additional changes;
alert handler receives the team argument, which was the root cause of breakage; https://github.com/ray-project/ray/pull/21289
Previously, tests without a team column were raising an exception, but I made the condition weaker (warning logs). I will eventually change it to raise an exception, but for smoother transition, we will log warning instead for a short time
2022-01-19 13:29:53 -08:00
Kai Fricke
e233f8172d
[ci/release] Terminate session on session startup timeout ( #21703 )
...
When a session startup times out due to resources not being available, the session may still come up after that timeout. At that time the control script (e2e.py) is already terminated, so the session runs until the autosuspend limit is hit, incurring unnecessary costs. Instead, we should always trigger session termination on session timeout.
2022-01-19 10:01:03 -08:00
Kai Fricke
0e9e8824e4
[ci/release] use s3 sync ( #21626 )
...
Previous changes failed because a) permission errors b) unzip being unavailable at remote nodes. Instead we are using tar gzip archives now.
This reverts commit 42bcab27e8
2022-01-15 17:53:19 -08:00
Kai Fricke
42bcab27e8
Revert "[Release Test] Opt-in tests to use K8s based cloud. ( #21583 )" ( #21605 )
...
This reverts commit 0d5fbcc7bb
2022-01-14 11:46:52 -08:00
Simon Mo
0d5fbcc7bb
[Release Test] Opt-in tests to use K8s based cloud. ( #21583 )
2022-01-13 17:20:36 -08:00
Kai Fricke
aa35045b6f
[ci/release] Update to recent anyscale API changes ( #21149 )
...
Recent changes in the anyscale API rendered the current e2e script incompatible. This PR resolves these subtle API changes.
2022-01-04 11:21:47 +00:00
mwtian
0b3fed5ef3
Revert "[Nightly Test] Add a team column to each test config. ( #21198 )" ( #21289 )
...
This reverts commit b5b11b2d06
2021-12-30 06:44:51 +09:00
SangBin Cho
b5b11b2d06
[Nightly Test] Add a team column to each test config. ( #21198 )
...
Please review **e2e.py and test_suite belonging to your team**!
This is the first part of https://docs.google.com/document/d/16IrwerYi2oJugnRf5hvzukgpJ6FAVEpB6stH_CiNMjY/edit#
This PR adds a team name to each test suite.
If the name is not specified, it will be reported as unspecified.
If you are running a local test, and if the new test suite doesn't have a team name specified, it will raise an exception (in this way, we can avoid missing team names in the future).
Note that we will aggregate all of test config into a single file, nightly_test.yaml.
2021-12-27 14:42:41 -08:00
architkulkarni
2489b17634
[release] Uninstall old ray in all release test app configs to fix commit mismatch error ( #21175 )
...
* uninstall old ray in all release test app configs
* add instruction to e2e.py dosctring
2021-12-18 16:58:49 -08:00
Yi Cheng
4e0de0053d
[nightly] Add staging nightly test for gcs ha ( #21004 )
...
This PR adds four staging nightly tests for gcs :
- many_actors
- many_tasks
- many_pgs
- many_nodes
These are benchmark tests that are highly related to gcs ha.
To make it easier to add tests, this PR also change e2e.py a little bit to include testing flags to app config.
2021-12-09 23:07:23 -08:00
Kai Fricke
b3a9d4d87d
[ci/release] Remove quotation marks from pip installs ( #20638 )
...
Quotation marks were needed in Anyscale app configs to avoid install errors when # were used e.g. in URLs.
Since this has been fixed on the Anyscale side, we can get rid of these.
2021-12-05 17:57:08 -08:00
Kai Fricke
6b683ec8dc
[ci] Retry release tests on infra error ( #20478 )
...
This PR introduces proper exit codes for release tests. These are used to restart a certain set of infrastructure related failures automatically.
2021-12-02 10:34:40 -08:00
Simon Mo
d7f208dea4
[Releaes] Make e2e.py link clickable on buildkite ( #20436 )
...
Adds log formatting to output clickable links to buildkite console logs
2021-11-18 12:45:59 +00:00
Kai Fricke
693063d6f8
[ci/release] fix exit code (use value, not object) ( #20427 )
2021-11-16 15:15:39 +00:00
Kai Fricke
d191ad2de8
[ci/release] Return exit codes based on different errors ( #20289 )
2021-11-15 19:41:00 +00:00
Jiajun Yao
992ab3e098
[Release] Commit sanity check when a url is provided ( #20255 )
2021-11-11 13:33:58 -08:00
SangBin Cho
f3e3c04469
[Nightly test] Make report False by default. ( #20238 )
...
* Make report False by default.
* fix
2021-11-11 04:58:23 -08:00
Jiajun Yao
e110d958a1
Support different s3 url formats ( #20133 )
2021-11-07 14:58:51 -08:00
Amog Kamsetty
3408b60d2b
[Release] Refactor User Tests ( #20028 )
...
* wip
* add directory
* wip
* try again
* Revert "try again"
This reverts commit 82d33ccea6f92848df025e019b87df73cea49e5d.
* finish
* formatting
* fix merge
* fix path
* chmod
* check
* sudo
* wip
* update
* fix horovod
* try
* typo
* reduce num workers
2021-11-05 17:28:37 -07:00
Kai Fricke
a13f738a10
[ci/release] Fix cloud search query ( #19876 )
2021-10-29 11:30:34 +02:00
Kai Fricke
564d8551ed
[ci/release] only check alert if test succeeded before ( #19857 )
2021-10-28 16:09:10 -07:00
Simon Mo
3e038aebb2
[CI] Allow release tests infra to accept buildkite artifacts ( #19803 )
2021-10-27 13:04:01 -07:00
Kai Fricke
98244ad130
[ci/release] Report error to database on alert ( #19743 )
2021-10-26 10:48:02 +01:00
Kai Fricke
96ddf5b9ac
[ci/release] Choose cloud by name or ID ( #19742 )
2021-10-26 10:21:54 +01:00
Kai Fricke
71564040ec
[ci/release] Unwrap after installing pip packages ( #19552 )
2021-10-20 13:41:16 +01:00
Kai Fricke
3e8587644b
[ci/release] wrap all release test pip github installs in quotation marks ( #19521 )
2021-10-19 20:55:02 +01:00
Kai Fricke
eee05505b1
[ci/release] Add separate timeout parameter for prepare commands ( #19459 )
2021-10-18 16:29:25 +01:00
Kai Fricke
c10d434713
[release] Allow commit hashes instead of URLs, add bisection utility ( #19398 )
2021-10-18 10:44:29 +01:00
Kai Fricke
e17b23fa5b
[ci/release] Add support for RAY_WHEELS url ( #19364 )
2021-10-14 21:40:01 +01:00
Carlo Grisetti
5cee8a1985
[release tests] Switch from yaml.load to yaml.safe_load ( #19365 )
2021-10-13 17:27:25 -07:00
Kai Fricke
42116badba
[ci/release] Check test result alerts after test finished ( #19105 )
2021-10-05 21:27:27 +01:00
Jiajun Yao
b8ef4f0a34
[CI] Add a retry helper to e2e.py ( #19045 )
2021-10-02 09:54:41 -07:00
SangBin Cho
55227a15b9
Handle retry to avoid statement timeout exception/ ( #18968 )
2021-09-29 23:04:35 -07:00
Jiajun Yao
35774fd399
[CI] Print out the mismatched commit in ci ( #18956 )
2021-09-29 15:48:57 +01:00
Kai Fricke
e08d4253cf
[ci/release] Start cluster before connecting via anyscale connect ( #18878 )
2021-09-24 16:17:06 +01:00
Kai Fricke
2cbf326410
[ci/release] store buildkite artifacts on buildkite ( #18712 )
2021-09-22 11:35:59 +01:00
gjoliver
5b6d69d61a
Minor change to switch result checking order so there is no artificial delay. ( #18555 )
...
Co-authored-by: Jun Gong <jungong@mbpro.local>
2021-09-20 22:49:17 +01:00
SangBin Cho
b8c361d3fb
[Test] Mark app config failure as a infra failure ( #18614 )
2021-09-14 17:20:05 -07:00
Kai Fricke
7d1e6d3129
[ci/release] Add sanity check for ray wheels hash to release tests ( #18489 )
2021-09-10 17:50:31 +01:00
Kai Fricke
6aa8a4eddc
[release] prettier output of release test results and artifacts ( #18337 )
2021-09-03 14:00:55 +01:00
SangBin Cho
dfbad8668a
Support better infra failure detection + stable flag ( #18202 )
2021-08-30 10:51:03 -07:00
Kai Fricke
fca8af88d2
[release] Fix e2e environment variable passing from pipeline ( #18000 )
2021-08-23 09:26:37 +02:00
Antoni Baum
0a1228ef6e
Add configurable autosuspend for connect tests ( #17958 )
2021-08-20 10:57:41 +02:00
architkulkarni
36c26578a7
[runtime env] [test] Add nightly test to verify Ray wheel URLs are valid ( #17938 )
2021-08-19 15:48:37 -07:00
Kai Fricke
0eee355d2e
Terminate session instead of stop ( #17946 )
2021-08-19 10:26:59 +02:00
Clark Zinzow
e2c7706f76
Add support for an app config override to the release test script, allowing better integration with compile-on-product. ( #17913 )
2021-08-18 13:35:27 -07:00
Antoni Baum
2b7d907762
Print description in --help ( #17871 )
2021-08-17 17:29:01 +02:00
architkulkarni
b173b33934
[tests] Add runtime envs release test to nightly build script ( #17638 )
2021-08-06 13:18:25 -05:00
Kai Fricke
86fcf076d6
[release] use job timeout with ray/anyscale connect ( #17614 )
2021-08-05 17:53:14 +01:00
