Why are these changes needed?
Together with ray-project/kuberay#391, this should address #26064
Have the autoscaler try to make the Ray log directory before setting up logging.
ray-project/kuberay#391 should be good enough for this, but this PR makes things safer in case the KubeRay user overrides log mounts or something like that.
I am surprised by the fact that `GetTimeoutError` is not a subclass of `TimeoutError`, which is counter-intuitive and may discourage users from trying the timeout feature in `ray.get`, because you have to "guess" the correct error type. For most people, I believe the first error type in their mind would be `TimeoutError`.
This PR fixes this.
✨ Add extra types for datasets
Why are these changes needed?
By using Protocols for the definition of callable classes, the type parameter in datasets can be preserved for the methods that work on rows.
This allows getting typing information, editor support, in transformed datasets. It also enables editor support, including autocompletion, inside of the parameters inside of lambdas passed to things like ds.map() and ds.filter().
For example, after several transformations, ds.filter() still gets autocompletion for the x parameter in the lambda, the editor knows it's an int.
https://user-images.githubusercontent.com/1326112/179423609-6d77da23-5f5e-47ce-a17f-6eb0d06d82d0.png
I see there was a TODO comment to do this trick with the protocol, so Clark already had this idea. 💡 The good news is that it's not necessary to wait until only Python 3.8 is supported by using typing_extensions.
We previously found all differences between origin/master and the PR branch, which includes forward changes if origin/master is ahead of the branch. By finding file differences with respect to the merge base we will only include actually changed files.
See https://matthew-brett.github.io/pydagogue/git_diff_dots.html
Signed-off-by: Kai Fricke <kai@anyscale.com>
Redo the agent-id changes from #24968. The original PR is in the first commit, the second commit fixes a fatal flaw when using RAY_BACKEND_LOG_LEVEL=debug, which caused the "Ray C++, Java" tests to fail on macOS.
Currently replicas for each deployment are packed based on Ray's default scheduling policy. This is problematic when node failures occur because a given deployment may have all of its replicas crash at once.
This PR changes the default behavior to a soft SPREAD. If the replicas can be spread based on current resources, they will be, else they will still be placed.
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
We've supported namespace in c++ worker in https://github.com/ray-project/ray/pull/26327. Here we add doc for usage and also reinforce the documents of Java and Python, like adding explanation of specifying namespace while creating named actors.
- [x] add doc for basic c++ worker namespace usage
- [x] add explanation for specifying namespace while creating named actors, in Python, Java and C++
- Add chaos tests for dataset random shuffle 1tb: both simple shuffle and push-based shuffle
- Mark dataset_shuffle_push_based_random_shuffle_1tb as stable
Signed-off-by: avnish avnish@anyscale.com
This change should fix issues with IMPALA and potentially APEX that stem from the various learner threads
Signed-off-by: avnish <avnish@anyscale.com>
Each node is supposed to report bytes used by its primary in-memory object copies. During spilling, this number is set incorrectly to 1 (see #26639 for more details).
This PR changes the reporting to include objects that are currently being spilled. It also splits out the "return nonzero number if there are local spilled objects" logic into a separate method.
Related issue number
Closes#26639.
Signed-off-by: Stephanie Wang <swang@cs.berkeley.edu>
Making sure that tuning multiple trials in parallel is not significantly slower than training each individual trials.
Some overhead is expected.
Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>
Signed-off-by: Richard Liaw <rliaw@berkeley.edu>
Signed-off-by: Kai Fricke <kai@anyscale.com>
Co-authored-by: Jimmy Yao <jiahaoyao.math@gmail.com>
Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
Co-authored-by: Kai Fricke <kai@anyscale.com>
Add benchmark data for 4x4 GPU setup.
Signed-off-by: Richard Liaw <rliaw@berkeley.edu>
Co-authored-by: Jimmy Yao <jiahaoyao.math@gmail.com>
Co-authored-by: Kai Fricke <kai@anyscale.com>
NOTE: tabulate is copied/pasted to the codebase for table formatting.
This PR changes the default layout to be the table format for both summary and list APIs.
* Revert "Revert "Bump pytest from 5.4.3 to 7.0.1""
This reverts commit ab10890e90.
Signed-off-by: Riatre Foo <foo@riat.re>
* Fix missing test data files dependency in rllib/BUILD
See # 26334 and # 26517 for context.
Once this is in, it should be good to roll-forwrad again.
Signed-off-by: Riatre Foo <foo@riat.re>
* debug: run all tests
Signed-off-by: Riatre Foo <foo@riat.re>
* Revert "debug: run all tests"
This reverts commit 0c5e796b0eb437d64922f66749c61b0412486970.
Signed-off-by: Riatre Foo <foo@riat.re>
* fix new tests since last rebase
Signed-off-by: Riatre Foo <foo@riat.re>
As a followup of #26619 (comment) and #26619 (comment), here we change from PermissionError to OSError, to be consistent as original error, and also change function name from _handle_read_s3_files_error to _handle_read_os_error, which is more general that we can handle other file systems such as GCS in the future.
Also change to hanlde any error message with pattern AWS Error [code xxx]: No response body as new issue with error code 100 is raised in #26672 .
This introduces an easy interface to retrieve the number of errored and terminated (non-errored) trials from the result grid.
Previously `tune.run(raise_on_failed_trial)` could be used to raise a TuneError if at least one trial failed. We've removed this option to make sure we always get a return value. `ResultGrid.num_errored` will make it easy for users to identify if trials failed and react to it instead of the old try-catch loop.
Signed-off-by: Kai Fricke <kai@anyscale.com>
