Richard Liaw 784a6399b0 [tune] Node Fault Tolerance (#3238) ... This PR introduces single-node fault tolerance for Tune. ## Previous behavior: - Actors will be restarted without checking if resources are available. This can lead to problems if we lose resources. ## New behavior: - RUNNING trials will be resumed on another node on a best effort basis (meaning they will run if resources available). - If the cluster is saturated, RUNNING trials on that failed node will become PENDING and queued. - During recovery, TrialSchedulers and SearchAlgorithms should receive notification of this (via `trial_runner.stop_trial`) so that they don’t wait/block for a trial that isn’t running. Remaining questions: - Should `last_result` be consistent during restore? Yes; but not for earlier trials (trials that are yet to be checkpointed). - Waiting for some PRs to merge first (#3239) Closes #2851.		2018-11-21 12:38:16 -08:00
..
multi_node_tests	Deprecate num_workers argument to ray.init and ray start. (#3114)	2018-10-28 20:12:49 -07:00
multi_node_docker_test.py	Remove legacy Ray code. (#3121)	2018-10-26 13:36:58 -07:00
run_asv.sh	[asv] Pushing to s3 (#2246)	2018-06-20 10:43:44 -07:00
run_multi_node_tests.sh	[tune] Node Fault Tolerance (#3238)	2018-11-21 12:38:16 -08:00
run_rllib_asv.sh	[rllib] Add more regression tests and autogenerate (#2324)	2018-07-02 08:20:53 -07:00