Logs a warning when a user sets max_workers for local node provider less than the number of available ips.
Also removes defaults of 0 for min_workers and max_workers from example configs to help prevent users inadvertantly setting max_workers=0 again.
Currently job drivers cannot use GPUs due to `CUDA_VISIBLE_DEVICES` being set (no resource request for job driver's supervisor actor). This is a regression from `ray submit`.
This is a temporary workaround -- in the future we should support a resource request for the job supervisor actor.
Especially when using datasets, we sometimes run into very long string representations. Tune should make sure to cut these according to a specified maximum length.
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
This PR includes the changes from #24172
This PR adds an end-to-end training and serving example for the RLTrainer/RLPredictor. It also adds an `RLServeEnv` that can be used as an external env for rllib inference, querying the served policy from the RLPredictor.
This draft PR runs end to end, but I'd like to gather some initial feedback before promoting it to a full PR.
[User complains](https://discuss.ray.io/t/which-attributes-can-be-used-in-checkpoint-score-attr-when-using-tune-run/5826) about logging on failure of locating `checkpoint_score_attr ` in results dict not being informative.
I propose that we log the actual results dict keys and extended stopping criteria, which imho should not log the whole result dict as this might contain tensors.
Maybe there are other similar cases in tune library, in which I don't know my way around that good.
This PR fixes a typo in the KubeRay example config in Ray's docs.
Specifics:
Ray versions in the Ray repo's example KubeRay CR were recently updated from 1.11.0 to 1.12.0.
However, the worker group's Ray version was accidentally left at 1.11.0. This leads to alarming crash-looping when deploying the example in the docs.
This PR matches up the Ray images by setting the worker group to rayproject/ray:1.12.0.
* fix init() requires hardcoded storage path when connecting to existing cluster
* update tests with new init(storage) behavior
* update tests with latest api behavior
Make sure users can read csv with columns types specified.
Users may want to do this because sometimes PyArrow's type inference doesn't work as intended, in which case users can step in and work around the type inference.
Jackson is a widely-used utility. User from Ant reports the jackson class is conflicted between Ray jar and user's jar.
This PR shade the jackson in Ray jar to avoid the conflict.
Co-authored-by: Kai Yang <kfstorm@outlook.com>
Adds a from_huggingface method to Datasets, which allows the conversion of a Hugging Face Dataset to a Ray Dataset. As a Hugging Face Dataset is backed by an Arrow table, the conversion is trivial.
The documentation says that @ray.remote can take fractional num_gpus which is true, but the documentation lists it as an integer. I think this is strictly a problem in the docs.
This PR
- adds an example on how to run Ray Train and log results to weights & biases
- adds functionality to the W&B plugin to store checkpoints
- fixes a bug introduced in #24017
- Adds a CI utility script to setup credentials
- Adds a CI utility script to remove test state from external services cc @simon-mo
