Edward Oakes
48b87d5830
[serve] Fix actor resources error in failure test ( #20400 )
2021-11-16 12:24:54 -08:00
Eric Liang
12a4489e30
Revert "[core] Nested task support via task depth + backpressure" ( #20438 )
...
Reverts ray-project/ray#17887
This causing several tests to be flaky (test_multinode_failures, test_virtual_actor, test_component_failures_2).
2021-11-16 11:14:45 -08:00
Simon Mo
5f2b035bba
Pin Redis version to < 4.0.0 ( #20430 )
...
<!-- Please add a reviewer to the assignee section when you create a PR. If you don't have the access to it, we will shortly find a reviewer and assign them to your PR. -->
## Why are these changes needed?
This pin is needed to fix `test_output` on master, which broke when 4.0.0 was released.
It may also fix the windows build (unsure).
## Related issue number
<!-- For example: "Closes #1234 " -->
## Checks
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for https://docs.ray.io/en/master/ .
- [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
- [ ] Unit tests
- [ ] Release tests
- [ ] This PR is not tested :(
2021-11-16 10:48:36 -08:00
Alex Wu
8f21cdbddb
Revert "[dependencies] Use redis[hiredis] in setup.py" ( #20435 )
...
Reverts ray-project/ray#20423
`hiredis` will break our M1 support right now.
2021-11-16 10:46:22 -08:00
Kai Fricke
6ec256122c
[dependencies] Use redis[hiredis] in setup.py ( #20423 )
...
This is recommended by `redis-py` and as a side effect gets rid of a current error in `test_output` for the minimal dependency test (e.g. https://buildkite.com/ray-project/ray-builders-branch/builds/4746#7444b5d0-87c3-4998-b722-1cbc2d9fe7e3 )
2021-11-16 10:25:36 -08:00
Antoni Baum
3f9ded55f7
[tune] Merge Analysis into ExperimentAnalysis ( #20197 )
...
Co-authored-by: Kai Fricke <kai@anyscale.com>
2021-11-16 16:47:12 +00:00
Antoni Baum
c097f64c79
[tune] Drop 0 value keys from PGF ( #20279 )
...
Co-authored-by: Kai Fricke <krfricke@users.noreply.github.com>
Co-authored-by: Kai Fricke <kai@anyscale.com>
2021-11-16 16:44:09 +00:00
Amog Kamsetty
4f88796d5a
[Train] Move to beta ( #20378 )
2021-11-16 08:19:30 -08:00
Kai Fricke
8a6c936aa8
[tune] Fix syncer=None not disabling trial-to-driver syncing ( #20418 )
2021-11-16 14:36:23 +00:00
Sven Mika
f82880eda1
Revert "Revert [RLlib] POC: Deprecate build_policy (policy template) for torch only; PPOTorchPolicy ( #20061 ) ( #20399 )" ( #20417 )
...
This reverts commit 90dc5460d4
2021-11-16 14:49:41 +01:00
Eric Liang
a1d78088e6
[Hotfix] Fix flaky test_basic_workflows_2
...
This test seems very flaky after the nested tasks PR merged. I think it's since the test is broken, and we made one of the branches more likely.
2021-11-16 01:11:36 -08:00
Yi Cheng
a4e187c0e7
[gcs] Update function table to use internal kv ( #20152 )
...
## Why are these changes needed?
This is a part of redis removal. This PR remove redis kv in function table.
rpush related code is not updated in this PR.
## Related issue number
2021-11-15 23:34:41 -08:00
Eric Liang
460cf86858
Split blocks automatically into 500MB chunks on file read and transformation ( #20235 )
...
This PR adds support for automatic block splitting on read and map transforms, to keep block size bounded to ~500MiB. This avoids potential OOM situations where a map task may consume too much intermediate Python heap memory, or too much object store shared memory for one block.
2021-11-15 22:25:11 -08:00
Siyuan (Ryans) Zhuang
3e9cd4248e
[workflow] Refactoring workflow to make it easier to follow the logic ( #20349 )
...
* update
* cleanup
2021-11-15 21:02:33 -08:00
Stephanie Wang
31eb385426
Revert "Revert "Revert "[core] Fail objects when pull/reconstruction hangs ( #19789 )" ( #19904 )" ( #20120 )" ( #20406 )
...
This reverts commit 0f57a9a105
2021-11-15 20:36:22 -08:00
Alex Wu
75f421a3fd
[core] Nested task support via task depth + backpressure ( #17887 )
...
* needs depth
* depth
* .
* .
* .
* lint
* .
* lint
* fix tests
* .
* .
* .
* .
* cleanup
* .
* tests
* .
* more tests
* fix rest(?) of tests
* cleanup
* .
* .
* .
* .
* lint
* fix test basic
* fix ref counting?
* cleanup
* lint
* .
* pass dataset pipeline test
* .
* stephanie's comments + fix tests
* cleanup
* cleanup
* minor cleanup, then fix merge conflict
* lint
* cast
* feature flag
* lint
* lint
* refactor
* needs cleanup
* should pass
* lint
* .
* .
* .
* work?
* .
* works?
* lint
* work?
* .
* fix cpp tests
* .
* .
* split test
* fix windows?
* fix windows?
* fix test + check
* .
* all passing
* tests
* lint
* cleanup
* .
* most stephanie ocmments
* lint
* remove timer
* .
* allowed - capacity
* .
* everything except barrier
* addd guard
* works
* lint
* works?
* debug string
* last comment?
* short comments
* most comments
* lint
* done?
* done?
* .
* .
* .
* .
* done?
* done?
* update
* lint
* fix last test
* .
* .
* .
* .
* .
* .
* debug
* .
* .
* .
* .
* fix type
* .
* .
* cleanup
Co-authored-by: Alex Wu <alex@anyscale.com>
2021-11-15 17:39:50 -08:00
Amog Kamsetty
90dc5460d4
Revert "[RLlib] POC: Deprecate build_policy (policy template) for torch only; PPOTorchPolicy ( #20061 )" ( #20399 )
...
This reverts commit 5b1c8e46e1
2021-11-15 16:11:35 -08:00
matthewdeng
35dc3cf21b
[train] fix Train/Tune integration on Client ( #20351 )
...
* [train] fix Train/Tune integration on Client
* remove force_on_current_node
2021-11-15 14:36:33 -08:00
Alex Wu
884bb3de33
[Dataset] Bump numpy >=1.20 dependency ( #20374 )
...
* done?
* .
Co-authored-by: Alex Wu <alex@anyscale.com>
2021-11-15 14:10:00 -08:00
Amog Kamsetty
ef7967476c
[Train] Torch data transfer automatic conversion ( #20333 )
...
* update
* formatting
* fix failures
* fix session tests
* address comments
* add to api docs
* package refactor
* wip
* wip
* wip
* finish
* finish
* fix
* comment
* fix
* install horovod for docs
* address comment
* Update python/ray/train/session.py
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
* Update python/ray/train/torch.py
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
* address comments
* try fix docs
* fix doc build failure
* wip
* fix
* fix
* fix
* try fix doc highlighting
* fix docs
* finish
* formatting
* address comments and fix tests
* address comments and fix test
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
2021-11-15 09:14:12 -08:00
Lixin Wei
b7e35acf14
[RuntimeEnv] Raise RuntimeEnvSetupError when Actor Creation Failed due to It ( #19888 )
...
* ray_pkg passed
* fix
* fix typo
* fix test
* fix test
* fix test
* fix
* draft
* compile OK
* lint
* fix
* lint
* fix ci
* Update src/ray/gcs/gcs_server/gcs_actor_manager.cc
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
* remove comment
* rename
* resolve conflict
* use unique ownership
* use DestroyActor instead of ReconstructActor
* fix sigment fault
* fix crash in debug log
* Revert "fix crash in debug log"
This reverts commit 8f0e3d37f062b664d8d0e07c6c1a9a715b8ba1ee.
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
2021-11-15 07:43:35 -08:00
Will Drevo
fa878e2d4d
Added example to user guide for cloud checkpointing ( #20045 )
...
Co-authored-by: will <will@anyscale.com>
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
Co-authored-by: Kai Fricke <kai@anyscale.com>
2021-11-15 15:43:06 +00:00
Amog Kamsetty
a74cf7ff1c
[Train] Torch Prepare utilities ( #20254 )
...
* update
* formatting
* fix failures
* fix session tests
* address comments
* add to api docs
* package refactor
* wip
* wip
* wip
* finish
* finish
* fix
* comment
* fix
* install horovod for docs
* address comment
* Update python/ray/train/session.py
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
* Update python/ray/train/torch.py
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
* address comments
* try fix docs
* fix doc build failure
* fix
* fix
* fix
* try fix doc highlighting
* fix docs
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
2021-11-15 07:34:17 -08:00
Sven Mika
5b1c8e46e1
[RLlib] POC: Deprecate build_policy (policy template) for torch only; PPOTorchPolicy ( #20061 )
2021-11-15 10:41:54 +01:00
SangBin Cho
477b6265d9
[Core] Fix get_actor consistency on ray.kill ( #20178 )
...
* Improve race condition on ray.kill
* Fix a bug.
* Fix a bug
* fix core worker test
* done
2021-11-14 23:29:47 -08:00
Jiajun Yao
61778a952d
Only grant or reject spillback lease request ( #20050 )
2021-11-14 21:34:28 -08:00
SangBin Cho
475e4dbf76
[Core] Fix pg stats not imported ( #20288 )
...
* Fix pg stats not imported
* Fix metrics are not exported to the test
2021-11-14 19:05:57 -08:00
Stephanie Wang
0f57a9a105
Revert "Revert "[core] Fail objects when pull/reconstruction hangs ( #19789 )" ( #19904 )" ( #20120 )
...
* Revert "Revert "[core] Fail objects when pull/reconstruction hangs (#19789 )" (#19904 )"
This reverts commit 630a8cacb3
2021-11-14 14:24:02 -08:00
Yi Cheng
87fa56def4
[gcs] Make gcs client in python able to auto reconnect ( #20299 )
...
## Why are these changes needed?
Since we are using gcs client as kv backend, we need to make it auto-reconnect in case of a failure. This PR adds this feature.
This PR adds auto_reconnect decorator to gcs-utils and in case of a failure it'll try to reconnect to gcs until it succeeds.
This feature right now support redis which should be deleted later once we finished bootstrap since kv will always go to gcs.
## Related issue number
2021-11-14 11:27:49 -08:00
iasoon
4cbe8f4c9c
Remove confusing max_calls examples from documentation ( #19395 )
2021-11-14 10:16:41 -08:00
shrekris-anyscale
c0aeb4a236
[runtime_env] Support working_dir and py_modules from HTTPS and Google Cloud Storage ( #20280 )
2021-11-14 02:16:45 -08:00
SangBin Cho
6cc493079b
[Core] Add Placement group performance test ( #20218 )
...
* in progress
* ip
* Fix issues
* done
* Address code review.
2021-11-14 09:17:54 +09:00
matthewdeng
e22632dabc
[train] wrap BackendExecutor in ray.remote() ( #20123 )
...
* [train] wrap BackendExecutor in ray.remote()
* wip
* fix trainer tests
* move CheckpointManager to Trainer
* [tune] move force_on_current_node to ml_utils
* fix import
* force on head node
* init ray
* split test files
* update example
* move tests to ray client
* address comments
* move comment
* address comments
2021-11-13 15:30:44 -08:00
Stephanie Wang
9e2bd508d7
[core] Move test_reconstruction to large test suite ( #20306 )
2021-11-13 13:51:18 -08:00
mwtian
df8042c576
[Client] connect to localhost via 127.0.0.1 ( #20274 )
...
Some Ray client users are likely seeing an issue similar to #7084 . Inside a container, connecting to localhost: fails but connecting to 127.0.0.1: succeeds. Changing Ray client to use 127.0.0.1 for localhost connection / serving should fix the issue.
2021-11-13 11:12:55 -08:00
architkulkarni
96de740cd2
[runtime env] Enable multinode tests ( #20264 )
2021-11-13 11:08:29 -08:00
Amog Kamsetty
65a17da2ec
[Train] Refactor Backends ( #20312 )
...
* wip
* finish
* comment
* fix
* install horovod for docs
* address comment
* fix doc build failure
2021-11-13 11:05:53 -08:00
xwjiang2010
f13c2a5350
[Tune] Revert "remove pg caching" ( #20308 )
...
This reverts commit 5f14eb3ee4
2021-11-13 16:25:22 +00:00
mwtian
875b0aea0a
fallback to grpc.experimental.aio when importing grpc.aio ( #20287 )
2021-11-13 15:59:57 +09:00
mwtian
cdadc2b7d2
Change owner ( #20313 )
2021-11-12 21:23:36 -08:00
Eric Liang
567e955810
Revert "[job submission] Use ray.init format addresses for JobSubmissionClient ( #20245 )" ( #20314 )
...
This reverts commit adc15a0fb0
2021-11-12 21:11:24 -08:00
Clark Zinzow
918a215442
[Datasets] Multi-aggregations [2/3]: Add groupby multi-column/multi-lambda aggregation ( #20074 )
2021-11-12 15:53:58 -08:00
Nikita Vemuri
adc15a0fb0
[job submission] Use ray.init format addresses for JobSubmissionClient ( #20245 )
2021-11-12 13:52:43 -08:00
xwjiang2010
cdf70c2900
[Tune] Remove legacy resources implementations in Runner and Executor. ( #19773 )
2021-11-12 12:33:39 -08:00
architkulkarni
138ec75246
[runtime env] Revert reference counting for per-actor URIs ( #20281 )
2021-11-12 11:09:38 -08:00
Matti Picus
1e80a2a83a
[WINDOWS] unskip tests ( #20212 )
2021-11-12 10:11:11 -08:00
Siyuan (Ryans) Zhuang
3b62388a9a
[Workflow] Workflow tail recursion optimization ( #19928 )
...
* tail recursion optimization
2021-11-12 09:13:40 -08:00
Simon Mo
b6bd4fd5f3
[Serve] Don't recover from current state checkpoint ( #19998 )
2021-11-12 09:02:27 -08:00
xwjiang2010
5f14eb3ee4
[Tune] Remove PG caching. ( #19515 )
...
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
2021-11-12 14:36:04 +00:00
Kai Fricke
d88fdd6e38
[tune] refactor SyncConfig ( #20155 )
2021-11-12 09:36:15 +00:00
