matthewdeng
b3b739266e
[docs] add dask compatibility for 1.8.0 ( #19578 )
2021-10-21 07:26:07 -07:00
Qing Wang
048e7f7d5d
[Core] Port concurrency groups with asyncio ( #18567 )
...
## Why are these changes needed?
This PR aims to port concurrency groups functionality with asyncio for Python.
### API
```python
@ray.remote(concurrency_groups={"io": 2, "compute": 4})
class AsyncActor:
def __init__(self):
pass
@ray.method(concurrency_group="io")
async def f1(self):
pass
@ray.method(concurrency_group="io")
def f2(self):
pass
@ray.method(concurrency_group="compute")
def f3(self):
pass
@ray.method(concurrency_group="compute")
def f4(self):
pass
def f5(self):
pass
```
The annotation above the actor class `AsyncActor` defines this actor will have 2 concurrency groups and defines their max concurrencies, and it has a default concurrency group. Every concurrency group has an async eventloop and a pythread to execute the methods which is defined on them.
Method `f1` will be invoked in the `io` concurrency group. `f2` in `io`, `f3` in `compute` and etc.
TO BE NOTICED, `f5` and `__init__` will be invoked in the default concurrency.
The following method `f2` will be invoked in the concurrency group `compute` since the dynamic specifying has a higher priority.
```python
a.f2.options(concurrency_group="compute").remote()
```
### Implementation
The straightforward implementation details are:
- Before we only have 1 eventloop binding 1 pythread for an asyncio actor. Now we create 1 eventloop binding 1 pythread for every concurrency group of the asyncio actor.
- Before we have 1 fiber state for every caller in the asyncio actor. Now we create a FiberStateManager for every caller in the asyncio actor. And the FiberStateManager manages the fiber states for concurrency groups.
## Related issue number
#16047
2021-10-21 21:46:56 +08:00
Oscar Knagg
5a05e89267
[Core] Add TLS/SSL support to gRPC channels ( #18631 )
2021-10-20 22:39:11 -07:00
Jiajun Yao
4fc5b11c68
Simple block dataset groupBy ( #19435 )
2021-10-19 19:53:13 -07:00
Simon Mo
30c8c073a2
[Doc] Generate sitemap ( #19375 )
2021-10-19 14:14:17 -07:00
Edward Oakes
a596d59863
[serve] Modify serve debugger example to use current APIs ( #19513 )
2021-10-19 13:21:56 -07:00
Duarte OC
5af6152e76
[Serve] [Doc] Update docs with import missing ( #19469 )
2021-10-19 11:23:50 -07:00
Alex Wu
a819e417ac
Revert "[Hotfix] Revert "[Workflow] workflow.delete"" ( #19248 )
...
* Revert "Revert "[Workflow] workflow.delete (#19178 )" (#19247 )"
This reverts commit b59317520d
2021-10-19 09:47:56 -07:00
matthewdeng
4674c78050
[Train] Rename Ray SGD v2 to Ray Train ( #19436 )
2021-10-18 22:27:46 -07:00
Guyang Song
46b4c7464d
runtime env eager install by default ( #19449 )
2021-10-19 11:31:14 +08:00
Jiajun Yao
4d9585773f
[Release] Remove release process doc ( #19312 )
2021-10-18 11:24:03 -07:00
Eric Liang
13d4ad6100
[data] Preserve epoch by default when using rewindow() ( #19359 )
2021-10-14 09:17:36 -07:00
Antoni Baum
e9df253f5d
[CI/docs] Remove [default] from xgboost-ray ( #19186 )
...
Co-authored-by: Kai Fricke <kai@anyscale.com>
2021-10-14 16:29:55 +01:00
Eric Liang
430a5f4a21
[doc] Bump dataset to beta for 1.8 and add backlink to SGD ( #19332 )
2021-10-12 18:32:29 -07:00
Jasha10
53e791d136
[Docs] Fix Typo in walkthrough ( #19335 )
...
There is one backtick too many in walkthrough.rst, it's causing a formatting issue.
2021-10-12 17:47:28 -07:00
Eric Liang
9f1cd9e867
[docs] Document fake multi-node autoscaler ( #19329 )
2021-10-12 15:59:07 -07:00
Amog Kamsetty
f6f2435b91
[SGD] Sgd v2 Dataset Integration ( #17626 )
...
* wip
* wip
* wip
* draft
* disable tf autosharding
* wip
* wip
* wip
* wip
* add example
* wip
* wip
* wip
* use dataset.split
* add unit tests
* add linear example
* concatenate tensors and fix example
* WIP tune example
* add tensorflow example
* wip
* random_shuffle_each_window
* fault tolerance test
* GPU, examples, CI
* formatting
* fix
* Update python/ray/util/sgd/v2/tests/test_trainer.py
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
* wip
* type hints
* wip
* update user guide
* fix
* fix immediate issues
* update example
* update
* fix tune gpu test
* fix resources for smoke test - 1 CPU for dataset tasks
* update tests, docs, examples
* Apply suggestions from code review
Co-authored-by: Clark Zinzow <clarkzinzow@gmail.com>
* address comments
* add warning
* fix tests
* minor doc updates
* update example in doc
* configure tests
* Update doc/source/raysgd/v2/user_guide.rst
Co-authored-by: Clark Zinzow <clarkzinzow@gmail.com>
* Update python/ray/data/dataset.py
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
* fix docstring
Co-authored-by: Matthew Deng <matthew.j.deng@gmail.com>
Co-authored-by: matthewdeng <matt@anyscale.com>
Co-authored-by: Clark Zinzow <clarkzinzow@gmail.com>
2021-10-12 14:03:10 -07:00
Eric Liang
0ab6749602
Support iter_epochs for Datasets ( #19217 )
2021-10-12 11:05:00 -07:00
architkulkarni
1ee3b4136c
[Serve] [Doc] Serve fix tracing snippet ( #19296 )
2021-10-11 16:59:04 -07:00
Chen Shen
c740aae54c
[Core][Dataset] adding example for large scale data ingestion ( #18998 )
2021-10-11 15:37:09 -07:00
Matti Picus
9ca34c7192
add dependencies to BUILD.bazel and update windows bazel to 4.2.1 ( #19132 )
...
* add dependencies to BUILD.bazel and update windows bazel to 4.2.1
* fixes from review
2021-10-11 10:25:19 -07:00
Guyang Song
bae543c956
[runtime env] support eager_install in runtime env ( #17949 )
2021-10-09 17:59:57 +08:00
Eric Liang
b59317520d
Revert "[Workflow] workflow.delete ( #19178 )" ( #19247 )
...
This reverts commit 7ea512f343
2021-10-08 19:12:55 -07:00
Alex Wu
7ea512f343
[Workflow] workflow.delete ( #19178 )
...
Why are these changes needed?
This PR implements workflow.delete which allows users to delete the information in storage related to a workflow. (This assumes the workflow isn't currently running).
Related issue number
Closes #18848
2021-10-08 16:11:59 -07:00
Antoni Baum
c7d6f838f6
[tune] Optional forcible trial cleanup, return default autofilled metrics even if Trainable doesn't report at least once ( #19144 )
2021-10-08 18:16:26 +01:00
xwjiang2010
7ffd9cbed1
[Tune] Fix column width in doc. ( #19159 )
2021-10-07 18:16:21 +01:00
Antoni Baum
27b8633198
[docs] Remove outdated note in Tune docs ( #19110 )
2021-10-07 15:42:11 +01:00
Edward Oakes
0f33aaf933
Revert "[Doc] Document existing runtime env's container support ( #19076 )" ( #19160 )
...
This reverts commit 4beba3f727
2021-10-07 08:55:30 -05:00
Eric Liang
86cbe3e833
[data] Add support for repeating and re-windowing a DatasetPipeline ( #19091 )
2021-10-06 20:13:43 -07:00
Simon Mo
4beba3f727
[Doc] Document existing runtime env's container support ( #19076 )
2021-10-06 10:25:57 -05:00
architkulkarni
281fcaa91a
[Serve] [Doc] Add note about serving multiple deployments defined by the same class ( #19118 )
2021-10-06 10:24:42 -05:00
Amog Kamsetty
db0483a29a
[SGD] SGD Namespace Consistency ( #19048 )
...
* wip
* update
* add callbacks
* fix
* fix
* update
* add
* address comments
2021-10-05 15:56:42 -07:00
Jiajun Yao
7ccf737f97
Add compatible dask version for ray 1.6.0 and 1.7.0 ( #19080 )
2021-10-05 10:23:06 +09:00
Eric Liang
032a420ee6
Rename Dataset.pipeline to Dataset.window ( #19050 )
2021-10-01 19:55:29 -07:00
Clark Zinzow
d22f838795
[Datasets] Delineate between ref and raw APIs for the Pandas/Arrow integrations. ( #18992 )
2021-10-01 13:08:25 -07:00
Antoni Baum
cc3199b814
[docs] Provide information about resource deadlocks, early stopping in Tune docs ( #18947 )
...
Co-authored-by: Kai Fricke <krfricke@users.noreply.github.com>
2021-10-01 13:52:47 +01:00
architkulkarni
8af9646cb0
[Doc] [runtime env] Remove delta caching remark and state Client+@remote limitation ( #19010 )
2021-09-30 13:29:50 -05:00
architkulkarni
0f0b161ea1
Revert "Revert "[Serve] [doc] Improve runtime env doc"" ( #18943 )
...
* Revert "Revert "[Serve] [doc] Improve runtime env doc (#18782 )" (#18935 )"
This reverts commit e4f4c79252
2021-09-30 13:28:44 -05:00
Amog Kamsetty
98ac3f601c
[SGD] v1 to v2 Migration Guide ( #18887 )
...
* wip
* add guide
* fix test
* address comments
* add to docs
* fix
* remove markdown
* add warning to all pages
* formatting
* fix
* links
* Update doc/source/raysgd/v2/migration-guide.rst
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
* Update doc/source/raysgd/v2/migration-guide.rst
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
* Update doc/source/raysgd/v2/migration-guide.rst
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
* Update doc/source/raysgd/v2/migration-guide.rst
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
* Update doc/source/raysgd/v2/migration-guide.rst
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
* address comments
* address comments
* fix
* address comments
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
2021-09-30 09:15:21 -07:00
architkulkarni
bf6e50813c
[runtime env] Parse local pip/conda requirements files locally upon task/actor definition ( #18988 )
2021-09-30 09:47:15 -05:00
Alex Wu
5709c6501b
[dataset][usability] Dataset dependencies ( #18346 )
2021-09-29 17:29:31 -07:00
matthewdeng
83a28cc901
[client] update documentation with AWS security_group ( #18905 )
...
* [client] update documentation with AWS security_group
* Update doc/source/cluster/ray-client.rst
Co-authored-by: Ian Rodney <ian.rodney@gmail.com>
* lint
Co-authored-by: Ian Rodney <ian.rodney@gmail.com>
2021-09-28 16:47:46 -07:00
Richard Liaw
227aa9e89b
[tune] change delimiter for results ( #16573 )
...
Signed-off-by: Richard Liaw <rliaw@berkeley.edu>
Co-authored-by: Kai Fricke <kai@anyscale.com>
2021-09-28 10:03:00 +01:00
Yi Cheng
e4f4c79252
Revert "[Serve] [doc] Improve runtime env doc ( #18782 )" ( #18935 )
...
This reverts commit d4d71985d5
2021-09-27 21:52:13 -07:00
matthewdeng
d2caa00be8
[SGD] add SGDv2 survey link to docs ( #18934 )
2021-09-27 19:15:37 -07:00
Eric Liang
caf34a452c
Unify ArrowTensorType tables and Tensor blocks ( #18867 )
2021-09-27 16:24:09 -07:00
architkulkarni
d4d71985d5
[Serve] [doc] Improve runtime env doc ( #18782 )
2021-09-27 16:12:03 -05:00
Yi Cheng
994d156a0d
[doc] Mark multi-client as an experimental feature
2021-09-27 14:08:49 -07:00
Edward Oakes
627e475cc1
[docs] Don't recommend SignalActor from test_utils
2021-09-27 12:08:37 -07:00
Antoni Baum
72cc0c9bda
[SGDv2] Add Tune-Cifar-PyTorch-PBT example ( #18860 )
...
* [SGDv2] Add Tune-Cifar-PyTorch-PBT example
* Update python/ray/util/sgd/v2/BUILD
* Lint
* Update example
* Update docs
2021-09-27 09:22:40 -07:00
