Commit graph

5465 commits

Author SHA1 Message Date
Yi Cheng
99034f5af5
Revert "Revert "[core] Fix wrong local resource view in raylet (#1991… (#19996)
This reverts commit f1eedb15b6.

## Why are these changes needed?
Self node should avoid reading any updates from gcs for node resource change since it'll maintain local view by itself.

## Related issue number
#19438
2021-11-03 00:11:40 -07:00
Eric Liang
398d4cbf34
[data] Skip tests locally if moto server is not installed 2021-11-02 23:56:32 -07:00
Eric Liang
9e448db731
[RFC] Add tsan build mode (#19971) 2021-11-02 22:29:51 -07:00
Jiajun Yao
6acf276959
Listen to 127.0.0.1 if node ip is 127.0.0.1 (#19918)
* Listen to 127.0.0.1 if node ip is 127.0.0.1

* Listen to 127.0.0.1 if node ip is 127.0.0.1

* Listen to 127.0.0.1 if node ip is 127.0.0.1
2021-11-03 12:17:55 +09:00
mwtian
ef4b6e4648
[Core][GCS] remove gcs object manager (#19963) 2021-11-02 16:20:53 -07:00
Edward Oakes
14d0889fbc
[serve] Rename BackendInfo -> DeploymentInfo (#19947) 2021-11-02 17:09:15 -05:00
iasoon
7e6ea9e3df
[serve] split ReplicaStartupState.PENDING into PENDING_ALLOCATION and PENDING_INITIALIZATION (#19431) 2021-11-02 17:08:52 -05:00
SangBin Cho
f1eedb15b6
Revert "[core] Fix wrong local resource view in raylet (#19911)" (#19992)
This reverts commit a907168184.

## Why are these changes needed?

This PR seems to have some huge perf regression on `placement_group_test_2.py`. It took 128s before, and after this PR was merged, it took 315 seconds. 

## Related issue number
2021-11-02 14:27:05 -07:00
Edward Oakes
f8a6cad0b7
[job submission] SDK prototype w/ dynamic working_dir uploads (#19843) 2021-11-02 16:01:54 -05:00
Siyuan (Ryans) Zhuang
3c9f91bd1d
[Workflow] Group options into a single workflow step options dataclass (#19654)
* group options into workflow step options

* fix comments

* cleanup virtual actor call options

* fix default value

* step_options.make()

* rename
2021-11-02 12:25:30 -07:00
SangBin Cho
857f23652f
Add more shuffle tests to CI (#17684)
* IP

* done

* done
2021-11-02 08:07:59 -07:00
SangBin Cho
563eb0bca2
[Runtime env] Add a test to make sure resource deadlock message is not printed when waiting for workers (#19870)
* ip

* Add a runtime env resource deadlock msg test

* Fix a bug

* Skip on windows
2021-11-02 07:48:55 -07:00
Sven Mika
2d24ef0d32
[RLlib] Add all simple learning tests as framework=tf2. (#19273)
* Unpin gym and deprecate pendulum v0

Many tests in rllib depended on pendulum v0,
however in gym 0.21, pendulum v0 was deprecated
in favor of pendulum v1. This may change reward
thresholds, so will have to potentially rerun
all of the pendulum v1 benchmarks, or use another
environment in favor. The same applies to frozen
lake v0 and frozen lake v1

Lastly, all of the RLlib tests and Tune tests have
been moved to python 3.7

* fix tune test_sampler::testSampleBoundsAx

* fix re-install ray for py3.7 tests

Co-authored-by: avnishn <avnishn@uw.edu>
2021-11-02 12:10:17 +01:00
Simon Mo
6040319d02
[CI] Pin aiohttp version to fix master branch (#19948) 2021-11-01 23:00:08 -07:00
Kai Yang
a33466e905
[Core] Fail inflight tasks on actor restarting (#19354)
## Why are these changes needed?

If an actor failover is triggered, but the RPC connection between the caller and the crashed actor instance is not disconnected automatically, subsequent tasks to the new actor instance may not be executed. The root cause is that the sequence numbers of tasks sent to the new actor instance is not starting from 0. Details can be found in #14727.

This PR fixes it by ensuring all inflight actor tasks fail immediately when actor failover is detected (via actor state notifications).

## Related issue number

closes #14727
2021-11-02 11:03:12 +08:00
Yi Cheng
a907168184
[core] Fix wrong local resource view in raylet (#19911)
## Why are these changes needed?
When gcs broad cast node resource change, raylet will use that to update local node as well which will lead to local node instance and nodes_ inconsistent.

1. local node has used all some pg resource
2. gcs broadcast node resources
3. local node now have resources
4. scheduler picks local node
5. local node can't schedule the task
6. since there is only one type of job and local nodes hasn't finished any tasks so it'll go to step 4 ==> hangs

## Related issue number
#19438
2021-11-01 19:52:03 -07:00
Amog Kamsetty
3a52187da8
[Release/Lightning] Add Ray lightning user test (#19812)
* wip

* wip

* add ray lightning test

* fix

* update

* merge and add

* fix

* fix

* rename

* autoscale

* add tblib

* gloo backend

* typo

* upgrade torch

* latest and master
2021-11-01 18:29:48 -07:00
Amog Kamsetty
474e44f7e0
[Release/Horovod] Add user test for Horovod (#19661)
* infra

* wip

* add test

* typo

* typo

* update

* rename

* fix

* full path

* formatting

* reorder

* update

* update

* Update release/horovod_tests/workloads/horovod_user_test.py

Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>

* bump num_workers

* update installs

* try

* add pip_packages

* min_workers

* fix

* bump pg timeout

* Fix symlink

* fix

* fix

* cmake

* fix

* pin filelock

* final

* update

* fix

* Update release/horovod_tests/workloads/horovod_user_test.py

* fix

* fix

* separate compute template

* test latest and master

Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
2021-11-01 18:28:07 -07:00
matthewdeng
e1e4a45b8d
[train] add simple Ray Train release tests (#19817)
* [train] add simple Ray Train release tests

* simplify tests

* update

* driver requirements

* move to test

* remove connect

* fix

* fix

* fix torch

* gpu

* add assert

* remove assert

* use gloo backend

* fix

* finish

Co-authored-by: Amog Kamsetty <amogkamsetty@yahoo.com>
2021-11-01 18:25:19 -07:00
Jiajun Yao
05c63f0208
[workflow] Mark workflow test_recovery as large test (#19950)
## Why are these changes needed?
move test_recovery to large test
## Related issue number
2021-11-01 15:50:38 -07:00
Sven Mika
bab9c0f670
[RLlib; Docs overhaul] Redo: Docstring cleanup: Trainer, trainer_template, Callbacks."" (#19830) 2021-11-01 21:45:11 +01:00
Alex Wu
80fb3f10ae
[ci] Script for building M1 wheels (#19925)
This PR includes a script for building wheels for Macs with M1 processors. It roughly follows the pattern of the other scripts with a few differences.

Manually installs nvm
Uses miniforge conda to install python/pip instead of python foundation .pkgs
Doesn't pin numpy (we probably shouldn't be pinning it in the other scripts either...)
Commit detection falls back to git instead of erroring
All of these changes were made so that the script works on a laptop, which comes with a subset of the dependencies that the x86 buildkite image comes with.
2021-11-01 11:44:59 -07:00
Hao Zhang
a03c4363b5
[Collective] Allow send/recv partial tensors in Send/Recv primitives (#19921) 2021-11-01 10:25:43 -07:00
Edward Oakes
ee57025be6
[serve] Rename BackendConfig -> DeploymentConfig (#19923) 2021-11-01 10:24:02 -07:00
architkulkarni
702bffe072
[runtime env] [test] Enable runtime env nightly test with working_dir reconnection (#19906) 2021-10-31 10:48:48 -05:00
architkulkarni
de8a9b5151
[runtime env] Always print package pushing logs regardless of size (#19897) 2021-10-31 10:47:37 -05:00
Edward Oakes
e507b7ba6e
[serve] Rename BackendVersion -> DeploymentVersion (#19798) 2021-10-31 10:27:19 -05:00
Chen Shen
961742f8e7
[Core] deflake windows test failure (test_task_retry_mini_integration) #19916 2021-10-30 15:13:38 -07:00
Sven Mika
4d945fe651
[RLlib] Issue 19878: Re-instate bare_metal_policy example script (#19881) 2021-10-30 12:50:39 -07:00
Stephanie Wang
630a8cacb3
Revert "[core] Fail objects when pull/reconstruction hangs (#19789)" (#19904)
This reverts commit e6d60d7376.
2021-10-30 10:54:39 -07:00
chenk008
57363995f3
[runtime env] Move container related code to runtime env (#19067) 2021-10-29 16:31:11 -07:00
Jiao
bb0ebb7903
[job submission] Temporarily make pydantic imports conditional (#19827) 2021-10-29 18:09:18 -05:00
Gagandeep Singh
f549e528c7
Bumped time limit in test_cancel::test_comprehensive (#19871) 2021-10-29 15:51:49 -07:00
SangBin Cho
99b5932d06
Add a simple node failure integration test + clean up spammy logs upon node failures (#19695)
* .

* Done

* clean up

* lint

* fix a bug

* lint

* fix issue

* Remove no-op from StartRayLog

* Addressed code review.
2021-10-29 18:42:35 -04:00
architkulkarni
16d3afc665
[serve] Base autoscaling decisions on target num replicas, not current num replicas (#19869) 2021-10-29 17:03:53 -05:00
Eric Liang
456d73754a
[data] Initial pass at support multiple-block returns for read and transform tasks (#19660) 2021-10-29 14:21:56 -07:00
Philipp Moritz
0a5942d8b0
[Documentation] Fix quotes for windows installations (#19859)
* [Documentation] Fix quotes for windows installations

* update

* formatting
2021-10-29 10:54:38 -07:00
Lixin Wei
56301e34b2
[Refactor] Remove ServiceBased Abstraction (#19694)
## Why are these changes needed?

Prior to this PR, we have:
```cpp
class XxxAccessor {}
class ServiceBasedXxxAccessor : public XxxAccessor{}

class GcsClient {}
class ServiceBasedGcsClient : public GcsClient{}
```

However, XxxAccessor has only one implementation: ServiceBasedXxxAccessor. And GcsClient has only one implementation: ServiceBasedGcsClient.

I think this abstraction is not necessary and will make development hard(I have to modify two files every time).

This PR removes all ServiceBasedXxx and moves its implementations to the base class.

Now we only have:
```cpp
class XxxAccessor {}
class GcsClient {}
```
2021-10-29 10:16:14 -07:00
Gagandeep Singh
9460a5375b
Added retry logic in test_basic::test_ray_options (#19832)
* Added retry logic in test_ray_options

* Applied linting format

* Made test consistent
2021-10-29 10:15:12 -07:00
Edward Oakes
bf23a31017
[job submission] Always generate and return job_id (#19851) 2021-10-29 09:09:54 -05:00
SangBin Cho
16dcff4091
[Core/RuntimeEnv] Fix runtime environment hanging issues. (#19823)
* done

* Add a right test

* Fix unit tests

* fix issues
2021-10-29 07:01:56 -07:00
Antoni Baum
f2773267c7
[docs] Tune doc fixes (#19791) 2021-10-29 11:45:29 +02:00
Sven Mika
902e854af2
[RLlib; Docs overhaul] Docstring cleanup: Environments. (#19784)
* wip.

* Test: Make a change in tune to trigger tune tests, which are not run otherwise, but seem to fail nevertheless with this PR's changes.

* remove bare_metal_policy_with_custom_view_reqs from tests
2021-10-29 10:46:52 +02:00
Stephanie Wang
e6d60d7376
[core] Fail objects when pull/reconstruction hangs (#19789) 2021-10-28 23:34:51 -07:00
Chris K. W
bd4ad84ead
[Client] Add deprecation warnings for direct ray.client().connect() calls (#18783)
* add deprecation warning

* Update wording

* add test

* actually connect

* add env var tests

* fix message and test

* skip on windows

* add _LocalBuilder case, update test_namespace

* better variable name
2021-10-28 22:06:11 -07:00
Jiajun Yao
760878f950
Handle empty dataset for sort and groupby (#19849) 2021-10-28 18:49:33 -07:00
Simon Mo
0433281ec8
[CI] Bump Serve test_regression to medium for windows (#19844) 2021-10-28 17:49:50 -07:00
Edward Oakes
42ac906313
[job submission] Support passing metadata to the JobConfig (#19845) 2021-10-28 16:40:03 -05:00
SangBin Cho
9126810c41
[Usabiilty] Improve the serialization failure message (#19691)
* Done

* done

* Done

* fix test

* Adressed code review.

* done

* done

* fix mistake

* Skip tests on windows
2021-10-28 14:25:51 -07:00
matthewdeng
bfb0ef1b08
move jsonschema to core dependencies and update default AutoscalerPrometheusMetrics (#19831) 2021-10-28 13:04:22 -07:00