matthewdeng
35dc3cf21b
[train] fix Train/Tune integration on Client ( #20351 )
...
* [train] fix Train/Tune integration on Client
* remove force_on_current_node
2021-11-15 14:36:33 -08:00
Alex Wu
884bb3de33
[Dataset] Bump numpy >=1.20 dependency ( #20374 )
...
* done?
* .
Co-authored-by: Alex Wu <alex@anyscale.com>
2021-11-15 14:10:00 -08:00
Amog Kamsetty
ef7967476c
[Train] Torch data transfer automatic conversion ( #20333 )
...
* update
* formatting
* fix failures
* fix session tests
* address comments
* add to api docs
* package refactor
* wip
* wip
* wip
* finish
* finish
* fix
* comment
* fix
* install horovod for docs
* address comment
* Update python/ray/train/session.py
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
* Update python/ray/train/torch.py
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
* address comments
* try fix docs
* fix doc build failure
* wip
* fix
* fix
* fix
* try fix doc highlighting
* fix docs
* finish
* formatting
* address comments and fix tests
* address comments and fix test
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
2021-11-15 09:14:12 -08:00
Lixin Wei
b7e35acf14
[RuntimeEnv] Raise RuntimeEnvSetupError when Actor Creation Failed due to It ( #19888 )
...
* ray_pkg passed
* fix
* fix typo
* fix test
* fix test
* fix test
* fix
* draft
* compile OK
* lint
* fix
* lint
* fix ci
* Update src/ray/gcs/gcs_server/gcs_actor_manager.cc
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
* remove comment
* rename
* resolve conflict
* use unique ownership
* use DestroyActor instead of ReconstructActor
* fix sigment fault
* fix crash in debug log
* Revert "fix crash in debug log"
This reverts commit 8f0e3d37f062b664d8d0e07c6c1a9a715b8ba1ee.
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
2021-11-15 07:43:35 -08:00
Will Drevo
fa878e2d4d
Added example to user guide for cloud checkpointing ( #20045 )
...
Co-authored-by: will <will@anyscale.com>
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
Co-authored-by: Kai Fricke <kai@anyscale.com>
2021-11-15 15:43:06 +00:00
Amog Kamsetty
a74cf7ff1c
[Train] Torch Prepare utilities ( #20254 )
...
* update
* formatting
* fix failures
* fix session tests
* address comments
* add to api docs
* package refactor
* wip
* wip
* wip
* finish
* finish
* fix
* comment
* fix
* install horovod for docs
* address comment
* Update python/ray/train/session.py
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
* Update python/ray/train/torch.py
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
* address comments
* try fix docs
* fix doc build failure
* fix
* fix
* fix
* try fix doc highlighting
* fix docs
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
2021-11-15 07:34:17 -08:00
Sven Mika
5b1c8e46e1
[RLlib] POC: Deprecate build_policy (policy template) for torch only; PPOTorchPolicy ( #20061 )
2021-11-15 10:41:54 +01:00
SangBin Cho
477b6265d9
[Core] Fix get_actor consistency on ray.kill ( #20178 )
...
* Improve race condition on ray.kill
* Fix a bug.
* Fix a bug
* fix core worker test
* done
2021-11-14 23:29:47 -08:00
Jiajun Yao
61778a952d
Only grant or reject spillback lease request ( #20050 )
2021-11-14 21:34:28 -08:00
SangBin Cho
475e4dbf76
[Core] Fix pg stats not imported ( #20288 )
...
* Fix pg stats not imported
* Fix metrics are not exported to the test
2021-11-14 19:05:57 -08:00
Stephanie Wang
0f57a9a105
Revert "Revert "[core] Fail objects when pull/reconstruction hangs ( #19789 )" ( #19904 )" ( #20120 )
...
* Revert "Revert "[core] Fail objects when pull/reconstruction hangs (#19789 )" (#19904 )"
This reverts commit 630a8cacb3
2021-11-14 14:24:02 -08:00
Yi Cheng
87fa56def4
[gcs] Make gcs client in python able to auto reconnect ( #20299 )
...
## Why are these changes needed?
Since we are using gcs client as kv backend, we need to make it auto-reconnect in case of a failure. This PR adds this feature.
This PR adds auto_reconnect decorator to gcs-utils and in case of a failure it'll try to reconnect to gcs until it succeeds.
This feature right now support redis which should be deleted later once we finished bootstrap since kv will always go to gcs.
## Related issue number
2021-11-14 11:27:49 -08:00
iasoon
4cbe8f4c9c
Remove confusing max_calls examples from documentation ( #19395 )
2021-11-14 10:16:41 -08:00
shrekris-anyscale
c0aeb4a236
[runtime_env] Support working_dir and py_modules from HTTPS and Google Cloud Storage ( #20280 )
2021-11-14 02:16:45 -08:00
SangBin Cho
6cc493079b
[Core] Add Placement group performance test ( #20218 )
...
* in progress
* ip
* Fix issues
* done
* Address code review.
2021-11-14 09:17:54 +09:00
matthewdeng
e22632dabc
[train] wrap BackendExecutor in ray.remote() ( #20123 )
...
* [train] wrap BackendExecutor in ray.remote()
* wip
* fix trainer tests
* move CheckpointManager to Trainer
* [tune] move force_on_current_node to ml_utils
* fix import
* force on head node
* init ray
* split test files
* update example
* move tests to ray client
* address comments
* move comment
* address comments
2021-11-13 15:30:44 -08:00
Stephanie Wang
9e2bd508d7
[core] Move test_reconstruction to large test suite ( #20306 )
2021-11-13 13:51:18 -08:00
mwtian
df8042c576
[Client] connect to localhost via 127.0.0.1 ( #20274 )
...
Some Ray client users are likely seeing an issue similar to #7084 . Inside a container, connecting to localhost: fails but connecting to 127.0.0.1: succeeds. Changing Ray client to use 127.0.0.1 for localhost connection / serving should fix the issue.
2021-11-13 11:12:55 -08:00
architkulkarni
96de740cd2
[runtime env] Enable multinode tests ( #20264 )
2021-11-13 11:08:29 -08:00
Amog Kamsetty
65a17da2ec
[Train] Refactor Backends ( #20312 )
...
* wip
* finish
* comment
* fix
* install horovod for docs
* address comment
* fix doc build failure
2021-11-13 11:05:53 -08:00
xwjiang2010
f13c2a5350
[Tune] Revert "remove pg caching" ( #20308 )
...
This reverts commit 5f14eb3ee4
2021-11-13 16:25:22 +00:00
mwtian
875b0aea0a
fallback to grpc.experimental.aio when importing grpc.aio ( #20287 )
2021-11-13 15:59:57 +09:00
mwtian
cdadc2b7d2
Change owner ( #20313 )
2021-11-12 21:23:36 -08:00
Eric Liang
567e955810
Revert "[job submission] Use ray.init format addresses for JobSubmissionClient ( #20245 )" ( #20314 )
...
This reverts commit adc15a0fb0
2021-11-12 21:11:24 -08:00
Clark Zinzow
918a215442
[Datasets] Multi-aggregations [2/3]: Add groupby multi-column/multi-lambda aggregation ( #20074 )
2021-11-12 15:53:58 -08:00
Nikita Vemuri
adc15a0fb0
[job submission] Use ray.init format addresses for JobSubmissionClient ( #20245 )
2021-11-12 13:52:43 -08:00
xwjiang2010
cdf70c2900
[Tune] Remove legacy resources implementations in Runner and Executor. ( #19773 )
2021-11-12 12:33:39 -08:00
architkulkarni
138ec75246
[runtime env] Revert reference counting for per-actor URIs ( #20281 )
2021-11-12 11:09:38 -08:00
Matti Picus
1e80a2a83a
[WINDOWS] unskip tests ( #20212 )
2021-11-12 10:11:11 -08:00
Siyuan (Ryans) Zhuang
3b62388a9a
[Workflow] Workflow tail recursion optimization ( #19928 )
...
* tail recursion optimization
2021-11-12 09:13:40 -08:00
Simon Mo
b6bd4fd5f3
[Serve] Don't recover from current state checkpoint ( #19998 )
2021-11-12 09:02:27 -08:00
xwjiang2010
5f14eb3ee4
[Tune] Remove PG caching. ( #19515 )
...
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
2021-11-12 14:36:04 +00:00
Kai Fricke
d88fdd6e38
[tune] refactor SyncConfig ( #20155 )
2021-11-12 09:36:15 +00:00
Edward Oakes
ee4e4f4036
[runtime_env] Support specifying the runtime_resources directory for testing ( #20257 )
2021-11-11 21:50:42 -08:00
Edward Oakes
7c9881b73d
[serve] Fix serve_failure test ( #20268 )
2021-11-11 19:19:34 -08:00
Edward Oakes
eb6449b21b
[serve] Remove 5s halt from controller startup ( #20262 )
2021-11-11 19:18:43 -08:00
SangBin Cho
e901180a55
Do not import pytest in test util ( #20252 )
2021-11-12 12:09:28 +09:00
Qing Wang
5d773e75e6
Fix idle worker leak issue if it received a SIGTERM when DrainAndShutdown. ( #19877 )
...
This PR fixes the issue that worker might be leaked if task finished with some errors.
See #19639 for more details.
2021-11-12 09:26:46 +08:00
chenk008
74fa267c72
Enable worker in container CI test ( #20174 )
2021-11-11 16:11:06 -08:00
Edward Oakes
5ae5c1ba28
[job submission] Basic CLI prototype ( #20204 )
2021-11-11 15:59:13 -08:00
Teofilo Zosa
abf0eb53cc
Fix aiohttp 3.8.0 breaking changes (and unpin from 3.7) ( #20261 )
2021-11-11 15:35:20 -08:00
mwtian
0330852baf
[Core][Pubsub] Implement Python GCS publisher and subscriber ( #20111 )
...
## Why are these changes needed?
This change adds Python publisher and subscriber in `gcs_utils.py`, and GRPC handler on GCS for publishing iva GCS. Error info is migrated to use the GCS-based pubsub, if feature flag `RAY_gcs_grpc_based_pubsub=true`.
Also, add a `--gcs-address` flag to some Python processes. It is not set anywhere yet, but will be set aftering Redis-less bootstrapping work.
Unit tests are added for the Python publisher and subscriber. Migrated error info publishers and subscribers are tested with existing unit tests, e.g. tests calling `ray._private.test_utils.get_error_message()` to ensure error info is published.
GCS based pubsub has gaps in handling deadline, cancelled requests and GCS restarts. So 3 more unit tests are disabled in the `HA GCS` mode. They will be addressed in a separate change.
## Related issue number
2021-11-11 14:59:57 -08:00
Simon Mo
fca851eef5
[Serve] Change ReplicaName to use internal prefix ( #20067 )
2021-11-11 14:21:34 -08:00
Dmitri Gekhtman
8971422d8f
[autoscaler] Use drain node api in autoscaler before terminating nodes ( #20013 )
...
* wip
* Draft
* Use bytest for node id
* remove stray helm change
* fix autoscaler init arg
* don't forget to instantiate new load metrics dict
* remove extraneous diff
* Timeout, comments, function signature.
* typo
* another comment
* tweak
* docstring
* shorter timeout
* Use a better error code
* missing self
* Dedent example
* Add drain node prometheus metric.
* comment
* Update tests part 1: test_autoscaler.py
* Update tests part 2: test_resource_demand_scheduler
* lint
* Update tests part 3: test_autoscaling_policy
* Unit tests for new Prometheus metric and DrainNode error handling.
* comment
* removed unused function
* Try adding ability to mock out process termination to fake node provider
* Add integration test.
* fix
* fix
* lint
* Improve log message
* fix
* Simplify test
* Fix doc example
* remove unused dict
* Mock out process termination in a subclass
* Add add doc string and comment explaining prune active ips.
* Comment: wtf is use_node_id_as_ip
* one more comment
* more explanation
* period
* tweak
2021-11-11 08:31:40 -08:00
xwjiang2010
883fbd003c
[CI; Tune] Split Tune tests and examples ( #20210 )
...
* Split Tune tests and examples part 1 into tests and examples separate.
* fix typo.
* fix typo.
* Add docs.
2021-11-11 10:50:51 +01:00
Siyuan (Ryans) Zhuang
8adcca54e8
[tune] Fix type error ( #19872 )
2021-11-10 21:35:38 -08:00
Yi Cheng
e54d3117a4
[gcs] Update all redis kv usage in python except function table ( #20014 )
...
## Why are these changes needed?
This is part of redis removal project. In this PR all direct usage of redis got removed except function table.
Function table will be migrated in the next PR
## Related issue number
#19443
2021-11-10 20:24:53 -08:00
Tobias Kaymak
893f57591d
[serve] Add Google Cloud Storage as a backend ( #20104 )
2021-11-10 19:45:19 -08:00
Edward Oakes
082a4af3e6
[serve] Remove lingering backend/endpoint wording in docs ( #20229 )
2021-11-10 16:49:29 -08:00
Edward Oakes
81f036d078
[job submission] Move job_manager to dashboard module, common parts to common.py ( #20209 )
2021-11-10 14:14:55 -08:00
