This was a holdover from when local resources for URIs were deleted directly from the runtime env agent, and the URI name itself needed to store the information of which plugin it corresponded to so the appropriate plugin's `delete()` function could be called. After the URI reference refactor, I don't think this is needed anymore.
Adds `get_dataframe()` method to pass through experiment analysis to result grid. Also fixes config dictionary returns which previously did not flatten the dict (even though the docstring suggested it would)
This PR supports setting the jars for an actor in Ray API. The API looks like:
```java
class A {
public boolean findClass(String className) {
try {
Class.forName(className);
} catch (ClassNotFoundException e) {
return false;
}
return true;
}
}
RuntimeEnv runtimeEnv = new RuntimeEnv.Builder()
.addJars(ImmutableList.of("https://github.com/ray-project/test_packages/raw/main/raw_resources/java-1.0-SNAPSHOT.jar"))
.build();
ActorHandle<A> actor1 = Ray.actor(A::new).setRuntimeEnv(runtimeEnv).remote();
boolean ret = actor1.task(A::findClass, "io.testpackages.Foo").remote().get();
System.out.println(ret); // true
```
This PR support reconnection of the GCS client in gRPC channel layer. Previously this is implemented in the application layer:
- Health check is in the application layer by starting a new channel.
- Monitor the GCS address change and do resubscribe.
- Always retry the failed request and do reconnection in case of a failure.
However, there are several issues with this approach:
- We need service to discover for GCS address change.
- Monitor is too heavy since it always creates a channel.
- Reconnection is a blocking call that prevents the code from running.
This new approach moves the reconnection to gRPC layer directly to fix these issues.
- DNS name resolution is done by gRPC so we don't need to write this.
- Health check is done by checking the channel state.
- Queue the failed call and retry once GCS is up so that it's not a blocking call.
Under heap memory pressure, the raylet is often killed by the OS OOM killer. This is bad because it can cause whole system crashes and it is difficult to find the error afterwards. This PR adjusts the OOM score for any workers that the raylet spawns so that the OOM killer will hopefully prioritize killing those instead of the raylet.
Adds support for chunking large schedule calls. Needed to support ray.remote calls with more than 2GiB of arguments.
Deprecates the args and kwargs fields of ClientTask and replaces them with a data field that contains a tuple of the serialized args and kwargs fields, which can be chunked and reassembled more easily using the same logic as PutRequest's.
A simple way to redirect remote job output to a custom file is extremely useful.
Especially if it does not require any code changes for the user, and especially if it captures all output and not limited only to python logging.
When the PR will be merged, the user will be able to do the following (for example):
ray submit some_cluster_config.yaml example_runnable_script.py --screen --extra-screen-args "-Logfile /gpfs/usr/someone/ray/output_log.txt"
which will, in addition to creating the screen session, also output (continuously) to custom text file.
It allows additional flexibility, for example, the user will be able to choose custom screen session name etc.
Closes#24514 by filtering out pods with metadata.deletionTimestamp set in the KuberayNodeProvider.
Adds some e2e test logic to confirm reasonable behavior when Ray worker pod termination hangs.
Also, a bit of code cleanup -- defining constants, adding to doc strings, etc.
HEBOSearch algorithm currently fails if the config search space contains a categorical parameter where each category is an iterable.
For instance, choosing the hidden layers of a NN:
` hyperparam_search_space = {'hidden_sizes': tune.choice([[512, 256, 128], [1024, 512, 256]])}`
This is due to the creation of the Pandas DataFrame with HEBO suggested parameters, without explicitly telling Pandas that the hyper-parameter suggestion is a single row of data while the index is being defined as a single row. This results in an exception such as "ValueError: Length of values (3) does not match length of index (1)".
Co-authored-by: Makan Arastuie <makan.arastuie@seagate.com>
Logs a warning when a user sets max_workers for local node provider less than the number of available ips.
Also removes defaults of 0 for min_workers and max_workers from example configs to help prevent users inadvertantly setting max_workers=0 again.
Currently job drivers cannot use GPUs due to `CUDA_VISIBLE_DEVICES` being set (no resource request for job driver's supervisor actor). This is a regression from `ray submit`.
This is a temporary workaround -- in the future we should support a resource request for the job supervisor actor.
Especially when using datasets, we sometimes run into very long string representations. Tune should make sure to cut these according to a specified maximum length.
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
This PR includes the changes from #24172
This PR adds an end-to-end training and serving example for the RLTrainer/RLPredictor. It also adds an `RLServeEnv` that can be used as an external env for rllib inference, querying the served policy from the RLPredictor.
This draft PR runs end to end, but I'd like to gather some initial feedback before promoting it to a full PR.
[User complains](https://discuss.ray.io/t/which-attributes-can-be-used-in-checkpoint-score-attr-when-using-tune-run/5826) about logging on failure of locating `checkpoint_score_attr ` in results dict not being informative.
I propose that we log the actual results dict keys and extended stopping criteria, which imho should not log the whole result dict as this might contain tensors.
Maybe there are other similar cases in tune library, in which I don't know my way around that good.
This PR fixes a typo in the KubeRay example config in Ray's docs.
Specifics:
Ray versions in the Ray repo's example KubeRay CR were recently updated from 1.11.0 to 1.12.0.
However, the worker group's Ray version was accidentally left at 1.11.0. This leads to alarming crash-looping when deploying the example in the docs.
This PR matches up the Ray images by setting the worker group to rayproject/ray:1.12.0.
* fix init() requires hardcoded storage path when connecting to existing cluster
* update tests with new init(storage) behavior
* update tests with latest api behavior
Make sure users can read csv with columns types specified.
Users may want to do this because sometimes PyArrow's type inference doesn't work as intended, in which case users can step in and work around the type inference.
Adds a from_huggingface method to Datasets, which allows the conversion of a Hugging Face Dataset to a Ray Dataset. As a Hugging Face Dataset is backed by an Arrow table, the conversion is trivial.
The documentation says that @ray.remote can take fractional num_gpus which is true, but the documentation lists it as an integer. I think this is strictly a problem in the docs.
This PR
- adds an example on how to run Ray Train and log results to weights & biases
- adds functionality to the W&B plugin to store checkpoints
- fixes a bug introduced in #24017
- Adds a CI utility script to setup credentials
- Adds a CI utility script to remove test state from external services cc @simon-mo
