This reverts commit f1eedb15b6.
## Why are these changes needed?
Self node should avoid reading any updates from gcs for node resource change since it'll maintain local view by itself.
## Related issue number
#19438
## Why are these changes needed?
This is the first step in migrating Redis pubsub to be GCS pubsub based. Changes include:
- Remove `SubscribeAll()` API for Actor pubsub since it is only used in tests. Supporting both `Subscribe()` and `SubscribeAll()` APIs would be too complex without much return.
- Update `Subscribe()` API to accept a done status callback.
- Implement `SubscribeAll()` / `Unsubscribe()`(from channel) API in Ray pubsub.
- Implement using Ray pubsub for Actor, Job, Node info and Node resource publishing / subscribing.
GCS changes are tested with GCS server test in GCS pubsub mode.
## Related issue number
This reverts commit a907168184.
## Why are these changes needed?
This PR seems to have some huge perf regression on `placement_group_test_2.py`. It took 128s before, and after this PR was merged, it took 315 seconds.
## Related issue number
* [xgboost/release] Add GPU connect user test
* Use scaling cluster
* typo
* Increase xgboost placement group timeout
* Much higher timeout
* Move os environment timeout
* Move os environ
* [dev] install xgboost-ray from master
* GPU xgboost master
* Remove master install after new xgboost release
* Install latest
* Add master test
* Unpin gym and deprecate pendulum v0
Many tests in rllib depended on pendulum v0,
however in gym 0.21, pendulum v0 was deprecated
in favor of pendulum v1. This may change reward
thresholds, so will have to potentially rerun
all of the pendulum v1 benchmarks, or use another
environment in favor. The same applies to frozen
lake v0 and frozen lake v1
Lastly, all of the RLlib tests and Tune tests have
been moved to python 3.7
* fix tune test_sampler::testSampleBoundsAx
* fix re-install ray for py3.7 tests
Co-authored-by: avnishn <avnishn@uw.edu>
* Added fixes to CartPole example
* Apply suggestions from code review
Co-authored-by: will <will@anyscale.com>
Co-authored-by: Sven Mika <sven@anyscale.io>
## Why are these changes needed?
If an actor failover is triggered, but the RPC connection between the caller and the crashed actor instance is not disconnected automatically, subsequent tasks to the new actor instance may not be executed. The root cause is that the sequence numbers of tasks sent to the new actor instance is not starting from 0. Details can be found in #14727.
This PR fixes it by ensuring all inflight actor tasks fail immediately when actor failover is detected (via actor state notifications).
## Related issue number
closes#14727
## Why are these changes needed?
When gcs broad cast node resource change, raylet will use that to update local node as well which will lead to local node instance and nodes_ inconsistent.
1. local node has used all some pg resource
2. gcs broadcast node resources
3. local node now have resources
4. scheduler picks local node
5. local node can't schedule the task
6. since there is only one type of job and local nodes hasn't finished any tasks so it'll go to step 4 ==> hangs
## Related issue number
#19438
This PR includes a script for building wheels for Macs with M1 processors. It roughly follows the pattern of the other scripts with a few differences.
Manually installs nvm
Uses miniforge conda to install python/pip instead of python foundation .pkgs
Doesn't pin numpy (we probably shouldn't be pinning it in the other scripts either...)
Commit detection falls back to git instead of erroring
All of these changes were made so that the script works on a laptop, which comes with a subset of the dependencies that the x86 buildkite image comes with.
