shrekris-anyscale
c0aeb4a236
[runtime_env] Support working_dir and py_modules from HTTPS and Google Cloud Storage ( #20280 )
2021-11-14 02:16:45 -08:00
SangBin Cho
6cc493079b
[Core] Add Placement group performance test ( #20218 )
...
* in progress
* ip
* Fix issues
* done
* Address code review.
2021-11-14 09:17:54 +09:00
matthewdeng
e22632dabc
[train] wrap BackendExecutor in ray.remote() ( #20123 )
...
* [train] wrap BackendExecutor in ray.remote()
* wip
* fix trainer tests
* move CheckpointManager to Trainer
* [tune] move force_on_current_node to ml_utils
* fix import
* force on head node
* init ray
* split test files
* update example
* move tests to ray client
* address comments
* move comment
* address comments
2021-11-13 15:30:44 -08:00
Stephanie Wang
9e2bd508d7
[core] Move test_reconstruction to large test suite ( #20306 )
2021-11-13 13:51:18 -08:00
mwtian
df8042c576
[Client] connect to localhost via 127.0.0.1 ( #20274 )
...
Some Ray client users are likely seeing an issue similar to #7084 . Inside a container, connecting to localhost: fails but connecting to 127.0.0.1: succeeds. Changing Ray client to use 127.0.0.1 for localhost connection / serving should fix the issue.
2021-11-13 11:12:55 -08:00
architkulkarni
96de740cd2
[runtime env] Enable multinode tests ( #20264 )
2021-11-13 11:08:29 -08:00
Amog Kamsetty
65a17da2ec
[Train] Refactor Backends ( #20312 )
...
* wip
* finish
* comment
* fix
* install horovod for docs
* address comment
* fix doc build failure
2021-11-13 11:05:53 -08:00
xwjiang2010
f13c2a5350
[Tune] Revert "remove pg caching" ( #20308 )
...
This reverts commit 5f14eb3ee4
2021-11-13 16:25:22 +00:00
mwtian
875b0aea0a
fallback to grpc.experimental.aio when importing grpc.aio ( #20287 )
2021-11-13 15:59:57 +09:00
mwtian
cdadc2b7d2
Change owner ( #20313 )
2021-11-12 21:23:36 -08:00
Eric Liang
567e955810
Revert "[job submission] Use ray.init format addresses for JobSubmissionClient ( #20245 )" ( #20314 )
...
This reverts commit adc15a0fb0
2021-11-12 21:11:24 -08:00
Clark Zinzow
918a215442
[Datasets] Multi-aggregations [2/3]: Add groupby multi-column/multi-lambda aggregation ( #20074 )
2021-11-12 15:53:58 -08:00
Nikita Vemuri
adc15a0fb0
[job submission] Use ray.init format addresses for JobSubmissionClient ( #20245 )
2021-11-12 13:52:43 -08:00
xwjiang2010
cdf70c2900
[Tune] Remove legacy resources implementations in Runner and Executor. ( #19773 )
2021-11-12 12:33:39 -08:00
architkulkarni
138ec75246
[runtime env] Revert reference counting for per-actor URIs ( #20281 )
2021-11-12 11:09:38 -08:00
Matti Picus
1e80a2a83a
[WINDOWS] unskip tests ( #20212 )
2021-11-12 10:11:11 -08:00
Siyuan (Ryans) Zhuang
3b62388a9a
[Workflow] Workflow tail recursion optimization ( #19928 )
...
* tail recursion optimization
2021-11-12 09:13:40 -08:00
Simon Mo
b6bd4fd5f3
[Serve] Don't recover from current state checkpoint ( #19998 )
2021-11-12 09:02:27 -08:00
xwjiang2010
5f14eb3ee4
[Tune] Remove PG caching. ( #19515 )
...
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
2021-11-12 14:36:04 +00:00
Kai Fricke
d88fdd6e38
[tune] refactor SyncConfig ( #20155 )
2021-11-12 09:36:15 +00:00
Edward Oakes
ee4e4f4036
[runtime_env] Support specifying the runtime_resources directory for testing ( #20257 )
2021-11-11 21:50:42 -08:00
Edward Oakes
7c9881b73d
[serve] Fix serve_failure test ( #20268 )
2021-11-11 19:19:34 -08:00
Edward Oakes
eb6449b21b
[serve] Remove 5s halt from controller startup ( #20262 )
2021-11-11 19:18:43 -08:00
SangBin Cho
e901180a55
Do not import pytest in test util ( #20252 )
2021-11-12 12:09:28 +09:00
Qing Wang
5d773e75e6
Fix idle worker leak issue if it received a SIGTERM when DrainAndShutdown. ( #19877 )
...
This PR fixes the issue that worker might be leaked if task finished with some errors.
See #19639 for more details.
2021-11-12 09:26:46 +08:00
chenk008
74fa267c72
Enable worker in container CI test ( #20174 )
2021-11-11 16:11:06 -08:00
Edward Oakes
5ae5c1ba28
[job submission] Basic CLI prototype ( #20204 )
2021-11-11 15:59:13 -08:00
Teofilo Zosa
abf0eb53cc
Fix aiohttp 3.8.0 breaking changes (and unpin from 3.7) ( #20261 )
2021-11-11 15:35:20 -08:00
mwtian
0330852baf
[Core][Pubsub] Implement Python GCS publisher and subscriber ( #20111 )
...
## Why are these changes needed?
This change adds Python publisher and subscriber in `gcs_utils.py`, and GRPC handler on GCS for publishing iva GCS. Error info is migrated to use the GCS-based pubsub, if feature flag `RAY_gcs_grpc_based_pubsub=true`.
Also, add a `--gcs-address` flag to some Python processes. It is not set anywhere yet, but will be set aftering Redis-less bootstrapping work.
Unit tests are added for the Python publisher and subscriber. Migrated error info publishers and subscribers are tested with existing unit tests, e.g. tests calling `ray._private.test_utils.get_error_message()` to ensure error info is published.
GCS based pubsub has gaps in handling deadline, cancelled requests and GCS restarts. So 3 more unit tests are disabled in the `HA GCS` mode. They will be addressed in a separate change.
## Related issue number
2021-11-11 14:59:57 -08:00
Simon Mo
fca851eef5
[Serve] Change ReplicaName to use internal prefix ( #20067 )
2021-11-11 14:21:34 -08:00
Dmitri Gekhtman
8971422d8f
[autoscaler] Use drain node api in autoscaler before terminating nodes ( #20013 )
...
* wip
* Draft
* Use bytest for node id
* remove stray helm change
* fix autoscaler init arg
* don't forget to instantiate new load metrics dict
* remove extraneous diff
* Timeout, comments, function signature.
* typo
* another comment
* tweak
* docstring
* shorter timeout
* Use a better error code
* missing self
* Dedent example
* Add drain node prometheus metric.
* comment
* Update tests part 1: test_autoscaler.py
* Update tests part 2: test_resource_demand_scheduler
* lint
* Update tests part 3: test_autoscaling_policy
* Unit tests for new Prometheus metric and DrainNode error handling.
* comment
* removed unused function
* Try adding ability to mock out process termination to fake node provider
* Add integration test.
* fix
* fix
* lint
* Improve log message
* fix
* Simplify test
* Fix doc example
* remove unused dict
* Mock out process termination in a subclass
* Add add doc string and comment explaining prune active ips.
* Comment: wtf is use_node_id_as_ip
* one more comment
* more explanation
* period
* tweak
2021-11-11 08:31:40 -08:00
xwjiang2010
883fbd003c
[CI; Tune] Split Tune tests and examples ( #20210 )
...
* Split Tune tests and examples part 1 into tests and examples separate.
* fix typo.
* fix typo.
* Add docs.
2021-11-11 10:50:51 +01:00
Siyuan (Ryans) Zhuang
8adcca54e8
[tune] Fix type error ( #19872 )
2021-11-10 21:35:38 -08:00
Yi Cheng
e54d3117a4
[gcs] Update all redis kv usage in python except function table ( #20014 )
...
## Why are these changes needed?
This is part of redis removal project. In this PR all direct usage of redis got removed except function table.
Function table will be migrated in the next PR
## Related issue number
#19443
2021-11-10 20:24:53 -08:00
Tobias Kaymak
893f57591d
[serve] Add Google Cloud Storage as a backend ( #20104 )
2021-11-10 19:45:19 -08:00
Edward Oakes
082a4af3e6
[serve] Remove lingering backend/endpoint wording in docs ( #20229 )
2021-11-10 16:49:29 -08:00
Edward Oakes
81f036d078
[job submission] Move job_manager to dashboard module, common parts to common.py ( #20209 )
2021-11-10 14:14:55 -08:00
architkulkarni
923131ba37
[runtime env] Enable reference counting for URIs for actors ( #20165 )
2021-11-10 10:52:03 -08:00
matthewdeng
790e22f9ad
[tune] move force_on_current_node to ml_utils ( #20211 )
2021-11-10 10:21:24 -08:00
DK.Pino
20f126896e
[Placement Group] [Test] Add fractional resources test for placement group ( #20185 )
...
* add fractional resources test
* lint
2021-11-10 07:25:49 -08:00
Kai Fricke
4e3e213549
[tune] Allow more versatile experiment analysis loading ( #20181 )
2021-11-10 11:46:27 +00:00
DK.Pino
2c41936a39
[Placement Group] [Test] Fix pg.ready hang forever when gcs restarting ( #20063 )
...
* fixed
* lint
* fix comment
* revert previous fix code
2021-11-10 00:53:42 -08:00
SangBin Cho
3bae6b94b3
[test] Fix flaky chaos_test.py ( #20202 )
...
* Fix
* fix lint
2021-11-10 00:23:55 -08:00
Edward Oakes
5475bb054c
[job submission] Redirect stdout + stderr to a single log file ( #20208 )
2021-11-09 22:34:12 -08:00
Jiajun Yao
5ffa0bb01f
Listen on 127.0.0.1 if node ip is 127.0.0.1 ( #20190 )
2021-11-09 20:24:05 -08:00
Sungho Joo
dc51af798c
[RLlib] Minor fix on json encoding during worker sampling ( #20134 )
...
* import custom json encoder from util and improve encoder default function
* linting
2021-11-09 16:46:41 -08:00
matthewdeng
33af739bf2
[train] add placement group support ( #20091 )
...
* [train] add placement group support
* fix additional resources
* fix tests
* add comment to add_workers
2021-11-09 16:36:07 -08:00
Edward Oakes
f6399e3389
[job submission] Remove jobs intermediate directory for logs ( #20192 )
2021-11-09 16:20:40 -08:00
Edward Oakes
39b3eb9763
[serve] Don't halt main control loop due to exceptions in snapshot logic ( #20151 )
2021-11-09 14:46:15 -08:00
Zyiqin-Miranda
333d0b43fd
[autoscaler] AWS Autoscaler CloudWatch Integration ( #18619 )
2021-11-09 11:48:55 -08:00
