Commit graph

5278 commits

Author SHA1 Message Date
Kai Yang
bdc005a4d4
[Java] Use test groups to filter tests of different run modes (#9703) 2020-07-29 11:18:45 +08:00
Simon Mo
9fbfee2424
Pin pytest version (#9767) 2020-07-28 19:54:48 -07:00
mehrdadn
fb5280f21b
Fix some Windows CI issues (#9708)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-28 18:10:23 -07:00
SangBin Cho
423dc96cc4
Revert "[dist] swap mac/linux wheel build order (#9746)" and "Fix package and upload ray jar (#9742)" (#9758)
* Revert "[dist] swap mac/linux wheel build order (#9746)"

This reverts commit a9340565ff.

* Revert "Fix package and upload ray jar (#9742)"

This reverts commit c290c308fe.
2020-07-28 15:34:29 -07:00
Alex Wu
21af0ceb0c
Register function race (#9346) 2020-07-28 13:51:34 -07:00
SangBin Cho
c00742f103
[Release] Fix release tests (#9733) 2020-07-28 10:44:06 -07:00
SangBin Cho
7e3ba289dc
[Stats] Basic Metrics Infrastructure (Metrics Agent + Prometheus Exporter) (#9607) 2020-07-28 10:28:01 -07:00
Alex Wu
feb3751824
[New scheduler] First unit test for task manager (#9696)
* .

* .

* refactor WorkerInterface

* .

* Basic unit test structure complete?

* .

* bad git >:-(

* small clean up

* CR

* .

* .

* One more fixture

* One more fixture

* .

* .

* bazel-format

* .
2020-07-28 09:44:58 -07:00
Ian Rodney
b1c2983c97
Run _with_interactive in Docker (#9747) 2020-07-28 08:57:04 -07:00
fangfengbin
bd18e975c0
fix windows compile bug (#9741)
Co-authored-by: 灵洵 <fengbin.ffb@antfin.com>
2020-07-28 21:30:31 +08:00
bermaker
6e23aff723
[Metrics]Ray java worker metric registry (#9636)
* ray worker metrics gauge init

* ray java metric mapping

* add jni source files for gauge and tagkey

* mapping all metric classes to stats object

* check non-null for tags and name

* lint

* add symbol for native metric JNI

* extern c for symbol

* add tests for all metrics

* Update Metric.java

use metricNativePointer instead.

* unify metric native stuff to one class

* fix jni file

* add comments for metric transform function in jni utils

* move metric function to native metric file

* remove unused disconnect jni

* Add a metric registry for java metircs

* Restore install-bazel.sh

* Add some comments for metric registry

* Fix thread safe problem of metrics

* Fix metric tests and remove sleep code from tests

* Fix comments of metrics

Co-authored-by: lingxuan.zlx <skyzlxuan@gmail.com>
2020-07-28 21:29:33 +08:00
Sven Mika
ff9c1dac88
[RLlib] Issue 9667 DDPG Torch bugs and enhancements. (#9680) 2020-07-28 14:15:03 +02:00
Sven Mika
e6ea33a03c
[RLlib] Enhance reward clipping test; add action_clipping tests. (#9684) 2020-07-28 10:44:54 +02:00
chaokunyang
a9340565ff
[dist] swap mac/linux wheel build order (#9746) 2020-07-28 16:44:19 +08:00
Alan Guo
5831737287
Introduce file_mounts_sync_continuously cluster option (#9544)
* Separate out file_mounts contents hashing into its own separate hash

Add an option to continuously sync file_mounts from head node to worker nodes:
monitor.py will re-sync file mounts whenver contents change but will only run setup_commands if the config also changes

* add test and default value for file_mounts_sync_continuously

* format code

* Update comments

* Add param to skip setup commands when only file_mounts content changed during monitor.py's update tick

Fixed so setup commands run when ray up is run and file_mounts content changes

* Refactor so that runtime_hash retains previous behavior

runtime_hash is almost identical as before this PR. It is used to determine if setup_commands need to run
file_mounts_contents_hash is an additional hash of the file_mounts content that is used to detect when only file syncing has to occur.

Note: runtime_hash value will have changed from before the PR because we hash the hash of the contents of the file_mounts as a performance optimization

* fix issue with hashing a hash

* fix bug where trying to set contents hash when it wasn't generated

* Fix lint error

Fix bug in command_runner where check_output was no longer returning the output of the command

* clear out provider between tests to get rid of flakyness

* reduce chance of race condition from node_launcher launching a node in the middle of an autoscaler.update call
2020-07-28 00:02:08 -07:00
chaokunyang
c290c308fe
Fix package and upload ray jar (#9742) 2020-07-28 11:53:25 +08:00
Lingxuan Zuo
1049c9e53b
[Stats] fix stats shutdown crash if opencensus exporter not initialized (#9727) 2020-07-28 11:21:10 +08:00
SangBin Cho
914cc96c91
Fix broken actor failure tests. (#9737) 2020-07-27 18:59:44 -07:00
Ian Rodney
ebcfef012f
[docker] Uses Latest Conda & Py 3.7 (#9732) 2020-07-27 16:18:12 -07:00
mehrdadn
2949c09ee8
Fix remote-watch.py (#9625)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-27 15:54:23 -07:00
Ian Rodney
78c34ae35e
Include open-ssh-client for transparency (#9693) 2020-07-27 15:31:35 -07:00
Alisa
51e12ee97c
Python api of placement group (#9243) 2020-07-27 14:57:05 -07:00
Michael Luo
b51ab2af66
[RLlib] Offline Type Annotations (#9676)
* Offline Annotations

* Modifications

* Fixed circular dependencies

* Linter fix
2020-07-27 14:01:17 -07:00
Bill Chambers
2e9d748100
[Cluster Launcher] Re Org the cluster launcher pages. (#9687) 2020-07-27 13:47:06 -07:00
Ian Rodney
d35605079e
[core] Removes Error when Internal Config is not set (#9700) 2020-07-27 11:47:54 -07:00
Robert Nishihara
5d89aedd40
Keep build-autoscaler-images.sh alive in CI (#9720) 2020-07-27 10:43:14 -07:00
Simon Mo
9213a81734
Only build docker wheels in LINUX_WHEELS env (#9729) 2020-07-27 10:42:40 -07:00
Simon Mo
7740136b93
Revert "Package and upload ray cross-platform jar (#9540)" (#9730)
This reverts commit 881032593d.
2020-07-27 10:40:21 -07:00
chaokunyang
881032593d
Package and upload ray cross-platform jar (#9540) 2020-07-27 17:20:20 +08:00
fangfengbin
2790818c53
[GCS]GCS client support multi-thread subscribe&resubscribe&unsubscribe (#9718) 2020-07-27 13:58:39 +08:00
Robert Nishihara
db0d6e8efa
Make wait_for_condition raise exception when timing out. (#9710) 2020-07-26 22:56:32 -07:00
fyrestone
4d08ddbf24
[Dashboard] New dashboard skeleton (#9099) 2020-07-27 11:34:47 +08:00
Tao Wang
44ccca1acb
Only update raylet map when autoscaler configured (#9435) 2020-07-27 11:23:06 +08:00
ZhuSenlin
a269ae9bc4
[GCS] Fix actor task hang when its owner exits before local dependencies resolved (#8045) 2020-07-27 10:56:52 +08:00
Richard Liaw
f3fdb5c5db
[tune] distributed torch wrapper (#9550)
* changes

* add-working

* checkpoint

* ccleanu

* fix

* ok

* formatting

* ok

* tests

* some-good-stuff

* fix-torch

* ddp-torch

* torch-test

* sessions

* add-small-test

* fix

* remove

* gpu-working

* update-tests

* ok

* try-test

* formgat

* ok

* ok
2020-07-26 09:37:22 -07:00
Tao Wang
c6a7b3ac68
[Tests]lock vector to avoid potential flaky test (#9656) 2020-07-26 11:57:28 +08:00
fangfengbin
ff1d7a6b36
[GCS]Open test_gcs_fault_tolerance testcase (#9677)
* enable test_gcs_fault_tolerance

* fix lint error

Co-authored-by: 灵洵 <fengbin.ffb@antfin.com>
2020-07-26 11:26:45 +08:00
Robert Nishihara
a8efb214de
Fix a few flaky tests (#9709)
Fix test_custom_resources, Remove test_pandas_parquet_serialization, Better error message for test_output.py, Potentially fix test_dynres::test_dynamic_res_creation_scheduler_consistency
2020-07-25 17:11:38 -07:00
Siyuan (Ryans) Zhuang
54a0d8b69e
[Core] Try remove all windows compat shims (#9671)
* try remove compat for arrow

* remove unistd.h

* remove socket compat

* delete arrow windows patch
2020-07-25 12:00:36 -07:00
SangBin Cho
d49b19c24c
[Stats] Improve Stats::Init & Add it to GCS server (#9563) 2020-07-25 10:42:08 -07:00
fangfengbin
28d5f9696d
Add placement group java api (#9611)
* add part code

* add part code

* add part code

* fix code style

* fix review comment

* fix review comment

* add part code

* add part code

* add part code

* add part code

* fix review comment

* fix review comment

* fix code style

* fix review comment

* fix lint error

* fix lint error

Co-authored-by: 灵洵 <fengbin.ffb@antfin.com>
2020-07-25 15:39:05 +08:00
Sven Mika
5dc4b6686e
[RLlib] Implement DQN PyTorch distributional head. (#9589) 2020-07-25 09:29:24 +02:00
krfricke
9f3570828a
[tune] move jenkins tests to travis (#9609)
Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
Co-authored-by: Kai Fricke <kai@anyscale.com>
2020-07-24 21:22:54 -07:00
Justin Terry
0d67602051
Update rllib-algorithms.rst (#9640) 2020-07-24 19:35:28 -07:00
Maksim Smolin
d364afbd31
[Autoscaler] CLI Logger docs (#9690)
Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
2020-07-24 16:59:25 -07:00
mehrdadn
56d2cf6479
Shellcheck rewrites (#9597)
* Fix SC2001: See if you can use ${variable//search/replace} instead.

* Fix SC2010: Don't use ls | grep. Use a glob or a for loop with a condition to allow non-alphanumeric filenames.

* Fix SC2012: Use find instead of ls to better handle non-alphanumeric filenames.

* Fix SC2015: Note that A && B || C is not if-then-else. C may run when A is true.

* Fix SC2028: echo may not expand escape sequences. Use printf.

* Fix SC2034: variable appears unused. Verify use (or export if used externally).

* Fix SC2035: Use ./*glob* or -- *glob* so names with dashes won't become options.

* Fix SC2071: > is for string comparisons. Use -gt instead.

* Fix SC2154: variable is referenced but not assigned

* Fix SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

* Fix SC2188: This redirection doesn't have a command. Move to its command (or use 'true' as no-op).

* Fix SC2236: Use -n instead of ! -z.

* Fix SC2242: Can only exit with status 0-255. Other data should be written to stdout/stderr.

* Fix SC2086: Double quote to prevent globbing and word splitting.

Co-authored-by: Mehrdad <noreply@github.com>
2020-07-24 17:24:19 -05:00
Simon Mo
678e238784
Add Ray Serve to README.rst (#9688) 2020-07-24 14:53:21 -07:00
mehrdadn
7344fb9858
Switch from GitHub checkout@v2 to checkout@v1 due to bugs in checkout (#9697)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-24 14:25:58 -07:00
Petros Christodoulou
46c64c90d0
fixed simplex initialisation seeding bug (#9660)
Co-authored-by: Petros Christodoulou <petrochr@amazon.com>
2020-07-24 14:22:41 -07:00
Bill Chambers
22d446bc2b
[Serve] Fix Formatting, stale docs (#9617) 2020-07-24 13:34:32 -07:00