mirror of
https://github.com/vale981/ray
synced 2025-03-05 18:11:42 -05:00
784a6399b0
This PR introduces single-node fault tolerance for Tune. ## Previous behavior: - Actors will be restarted without checking if resources are available. This can lead to problems if we lose resources. ## New behavior: - RUNNING trials will be resumed on another node on a best effort basis (meaning they will run if resources available). - If the cluster is saturated, RUNNING trials on that failed node will become PENDING and queued. - During recovery, TrialSchedulers and SearchAlgorithms should receive notification of this (via `trial_runner.stop_trial`) so that they don’t wait/block for a trial that isn’t running. Remaining questions: - Should `last_result` be consistent during restore? Yes; but not for earlier trials (trials that are yet to be checkpointed). - Waiting for some PRs to merge first (#3239) Closes #2851.
11 lines
487 B
Docker
11 lines
487 B
Docker
# The examples Docker image adds dependencies needed to run the examples
|
|
|
|
FROM ray-project/deploy
|
|
|
|
# This updates numpy to 1.14 and mutes errors from other libraries
|
|
RUN conda install -y numpy
|
|
RUN apt-get install -y zlib1g-dev
|
|
RUN pip install gym[atari] opencv-python==3.2.0.8 tensorflow lz4 keras pytest-timeout
|
|
RUN pip install -U h5py # Mutes FutureWarnings
|
|
RUN pip install --upgrade git+git://github.com/hyperopt/hyperopt.git
|
|
RUN conda install pytorch-cpu torchvision-cpu -c pytorch
|