Qing Wang
048e7f7d5d
[Core] Port concurrency groups with asyncio ( #18567 )
...
## Why are these changes needed?
This PR aims to port concurrency groups functionality with asyncio for Python.
### API
```python
@ray.remote(concurrency_groups={"io": 2, "compute": 4})
class AsyncActor:
def __init__(self):
pass
@ray.method(concurrency_group="io")
async def f1(self):
pass
@ray.method(concurrency_group="io")
def f2(self):
pass
@ray.method(concurrency_group="compute")
def f3(self):
pass
@ray.method(concurrency_group="compute")
def f4(self):
pass
def f5(self):
pass
```
The annotation above the actor class `AsyncActor` defines this actor will have 2 concurrency groups and defines their max concurrencies, and it has a default concurrency group. Every concurrency group has an async eventloop and a pythread to execute the methods which is defined on them.
Method `f1` will be invoked in the `io` concurrency group. `f2` in `io`, `f3` in `compute` and etc.
TO BE NOTICED, `f5` and `__init__` will be invoked in the default concurrency.
The following method `f2` will be invoked in the concurrency group `compute` since the dynamic specifying has a higher priority.
```python
a.f2.options(concurrency_group="compute").remote()
```
### Implementation
The straightforward implementation details are:
- Before we only have 1 eventloop binding 1 pythread for an asyncio actor. Now we create 1 eventloop binding 1 pythread for every concurrency group of the asyncio actor.
- Before we have 1 fiber state for every caller in the asyncio actor. Now we create a FiberStateManager for every caller in the asyncio actor. And the FiberStateManager manages the fiber states for concurrency groups.
## Related issue number
#16047
2021-10-21 21:46:56 +08:00
Antoni Baum
a04b02e2e8
[tune] Better bad Stopper type message ( #19496 )
2021-10-21 14:31:27 +01:00
Kai Fricke
44fb7d09df
[tune] sync_client: Fix delete template formatting ( #19553 )
2021-10-21 10:59:54 +01:00
Patrick Ames
20d47873c9
[data] Add pickle support for PyArrow CSV WriteOptions ( #19378 )
2021-10-21 00:46:52 -07:00
Matti Picus
bacd5f92e2
MAINT: cleanups for windows ( #19430 )
...
* dead processes should increment total_stopped
* use psutil in testing to check pid
* remove unneeded repititions
2021-10-20 23:32:35 -07:00
Yi Cheng
cba8480616
[dashboard] Fix the wrong metrics for grpc query execution time in server side ( #19500 )
...
## Why are these changes needed?
It looks like the metrics set on server side are wrong. The time the query is constructed sometimes is not the time we get the query. This PR fixed this.
## Related issue number
2021-10-20 23:06:35 -07:00
Oscar Knagg
5a05e89267
[Core] Add TLS/SSL support to gRPC channels ( #18631 )
2021-10-20 22:39:11 -07:00
heng2j
6d23fb1ff1
[Tune] Support custom tags in MLflow logger callback ( #19532 )
...
* Added Food Collector support to rllib/env/unity3d_env.py
* feat(mlflow): added parameter tags to MLflowLoggerCallback
* fix(unit_test): added tags tests in test_integration_mlflow.MLflowTest()
* chore: lint the changes in this PR
* update
* Update python/ray/tune/integration/mlflow.py
* fix
* copy
* fix
Co-authored-by: zla0368 <zhongheng.li@stresearch.com>
Co-authored-by: Li, Zhongheng <zhongheng.li@str.us>
Co-authored-by: Amog Kamsetty <amogkamsetty@yahoo.com>
Co-authored-by: Amog Kamsetty <amogkam@users.noreply.github.com>
2021-10-20 22:31:33 -07:00
SangBin Cho
085162c68e
[Log] Print actor & task name upon stderr messages. ( #19542 )
2021-10-20 22:16:32 -07:00
Eric Liang
699c5aeac6
Revert "[Dashboard] Disable unnecessary event messages. ( #19490 )" ( #19574 )
...
This reverts commit 7fb681a35d
2021-10-20 20:17:57 -07:00
Eric Liang
48ecb1f88a
[data] Fix O(n^2) issues in simple_block sort ( #19543 )
2021-10-20 18:26:20 -07:00
Philipp Moritz
45f1ff0fa9
[Windows] Update react-scripts dependency for dashboard ( #19489 )
2021-10-20 17:57:30 -07:00
SangBin Cho
7fb681a35d
[Dashboard] Disable unnecessary event messages. ( #19490 )
...
* Disable unnecessary event messages.
* use warning
* Fix tests
2021-10-20 17:40:25 -07:00
Edward Oakes
bcf584294f
[runtime_env] Refactor working dir packaging code into runtime_env.packaging module ( #19112 )
2021-10-20 18:38:50 -05:00
gjoliver
44a4e42172
[rllib] Add entropy_coeff_schedule support for APPO. ( #19544 )
...
* Add entropy_coeff_schedule support for APPO.
* lint
2021-10-20 14:18:01 -07:00
Eric Liang
7daf28f348
Revert "[Test] Fix flaky test_gpu test ( #19524 )" ( #19562 )
...
This reverts commit 39e54cd276
2021-10-20 12:21:19 -07:00
Clark Zinzow
88c5fcde8c
[Datasets] Unrevert Arrow table copy method change. ( #19534 )
2021-10-20 11:57:36 -07:00
Jiao
c51f79bca6
[runtime_env] Support remote s3 package in runtime env ( #19315 )
2021-10-20 10:41:54 -05:00
mwtian
aaff6901dd
[Pubsub] refactor pubsub to support different channel types ( #19498 )
...
* refactor pubsub to support different channel types
* fix
* use std::string for key id
* fix mock
* fix
2021-10-20 07:02:55 -07:00
Kai Fricke
71564040ec
[ci/release] Unwrap after installing pip packages ( #19552 )
2021-10-20 13:41:16 +01:00
Jiajun Yao
39e54cd276
[Test] Fix flaky test_gpu test ( #19524 )
2021-10-19 22:36:34 -07:00
Yi Cheng
01b899dafb
[nightly] Fix broken test due to bad syntax #19536 ( #19536 )
2021-10-19 21:43:46 -07:00
Simon Mo
59eef6521b
[Serve] Use regular dict for handle caching ( #19162 )
2021-10-19 21:27:01 -07:00
Yi Cheng
7a9cedfc5c
[nightly] Add grpc based broadcasting into nightly test for decision_tree ( #19531 )
...
* dbg
* up
* check
* up
* up
* put grpc based one into nightly test
* up
2021-10-19 19:59:39 -07:00
Jiajun Yao
4fc5b11c68
Simple block dataset groupBy ( #19435 )
2021-10-19 19:53:13 -07:00
Eric Liang
eacfbf8be2
[data] Don't shuffle during repartition by default ( #19379 )
2021-10-19 19:46:22 -07:00
SangBin Cho
3222d39fb8
[Dashboard] Dashboard memory improvement ( #19385 )
...
* many ppo profiling
* completed
* improve memory usage lint
* revert temporarily
* Addressed code review
* Fix a test
2021-10-19 19:34:42 -07:00
Simon Mo
30c8c073a2
[Doc] Generate sitemap ( #19375 )
2021-10-19 14:14:17 -07:00
Simon Mo
48cf366dca
[Hotfix] Pin node version to 14 ( #19522 )
2021-10-19 14:13:06 -07:00
Ian Rodney
02090afc26
[Docker] Re-Tag Docker Images with a lambda ( #19081 )
...
* lil lambda
* Better Credential Handling
* use a script for this :)
* better timeout and link & echo messages
2021-10-19 14:06:31 -07:00
matthewdeng
19eabd7a55
[train] remove default num_workers ( #19518 )
...
* [train] remove default num_workers
* fix tests
2021-10-19 13:53:23 -07:00
Akash Patel
7e10f6a876
add missing <limits> header for prometheus_cpp ( #19108 )
2021-10-19 13:33:31 -07:00
gjoliver
2bd7932830
Add a script to analyze python module dependencies using static analysis ( #18965 )
2021-10-19 13:33:02 -07:00
mwtian
098ff36faa
[CI] Remove config that disables Bazel test result cache ( #18701 )
2021-10-19 13:31:42 -07:00
Edward Oakes
a596d59863
[serve] Modify serve debugger example to use current APIs ( #19513 )
2021-10-19 13:21:56 -07:00
matthewdeng
56e46c3c23
[train] add callbacks package compatibility ( #19519 )
2021-10-19 12:56:49 -07:00
Kai Fricke
3e8587644b
[ci/release] wrap all release test pip github installs in quotation marks ( #19521 )
2021-10-19 20:55:02 +01:00
Edward Oakes
4645893a5f
Add prototype of ray.serve.pipeline ( #19278 )
2021-10-19 11:36:49 -07:00
xwjiang2010
a6f9c93db0
Revert "[Datasets] Add support for slicing Arrow blocks that contain tensor columns. ( #19494 )" ( #19517 )
...
This reverts commit ad03917b8f
2021-10-19 11:35:04 -07:00
Duarte OC
5af6152e76
[Serve] [Doc] Update docs with import missing ( #19469 )
2021-10-19 11:23:50 -07:00
Tao He
1dde588702
[Dataset] Support dataset from a single dataframe/table. ( #18205 )
2021-10-19 10:27:43 -07:00
Alex Wu
a819e417ac
Revert "[Hotfix] Revert "[Workflow] workflow.delete"" ( #19248 )
...
* Revert "Revert "[Workflow] workflow.delete (#19178 )" (#19247 )"
This reverts commit b59317520d
2021-10-19 09:47:56 -07:00
Gagandeep Singh
cc00ab74da
[Windows] Fix test_fair_queuing and test_wait_timing ( #19456 )
...
* modified timeout in test_fair_qeueing
* bump bounds to pass the tests
2021-10-19 09:27:04 -07:00
mwtian
3260330e45
Disable clang-tidy until ergonomic issues are resolved ( #19499 )
...
Why are these changes needed?
Currently clang-tidy does not run inside scripts/format.sh. Also clang-tidy can produce false positive warnings. Maybe we can disable clang-tidy until ergonomic issues are resolved.
2021-10-19 08:45:25 -07:00
architkulkarni
b8941338d3
[runtime env] Raise error when creating runtime env when ray[default] is not installed ( #19491 )
2021-10-19 09:16:04 -05:00
Jiajun Yao
805ce453dd
[Java] Remove auto-generated pom.xml files. ( #19475 )
2021-10-19 17:35:37 +08:00
matthewdeng
4674c78050
[Train] Rename Ray SGD v2 to Ray Train ( #19436 )
2021-10-18 22:27:46 -07:00
Guyang Song
46b4c7464d
runtime env eager install by default ( #19449 )
2021-10-19 11:31:14 +08:00
Clark Zinzow
ad03917b8f
[Datasets] Add support for slicing Arrow blocks that contain tensor columns. ( #19494 )
2021-10-18 20:07:06 -07:00
Simon Mo
6f2eb1f9fa
[Serve] Use ray core metrics for autoscaling ( #19038 )
2021-10-18 19:32:49 -07:00
