mirror of
https://github.com/vale981/ray
synced 2025-03-06 10:31:39 -05:00
d49b4bef0a
## What do these changes do? This implements basic task reconstruction in raylet. There are two parts to this PR: 1. Reconstruction suppression through the `TaskReconstructionLog`. This prevents two raylets from reconstructing the same task if they decide simultaneously (via the logic in #2497) that reconstruction is necessary. 2. Task resubmission once a raylet becomes responsible for reconstructing a task. Reconstruction is quite slow in this PR, especially for long chains of dependent tasks. This is mainly due to the lease table mechanism, where nodes may wait too long before trying to reconstruct a task. There are two ways to improve this: 1. Expire entries in the lease table using Redis `PEXPIRE`. This is a WIP and I may include it in this PR. 2. Introduce a "fast path" for reconstructing dependencies of a re-executed task. Normally, we wait for an initial timeout before checking whether a task requires reconstruction. However, if a task requires reconstruction, then it's likely that its dependencies also require reconstruction. In this case, we could skip the initial timeout before checking the GCS to see whether reconstruction is necessary (e.g., if the object has been evicted). Since handling failures of other raylets is probably not yet complete in master, this only turns back on Python tests for reconstructing evicted objects.
33 lines
876 B
Bash
Executable file
33 lines
876 B
Bash
Executable file
#!/usr/bin/env bash
|
|
|
|
# Cause the script to exit if a single command fails.
|
|
set -e
|
|
|
|
# Show explicitly which commands are currently running.
|
|
set -x
|
|
|
|
ROOT_DIR=$(cd "$(dirname "${BASH_SOURCE:-$0}")"; pwd)
|
|
$ROOT_DIR/../build.sh -l java
|
|
|
|
pushd $ROOT_DIR/../thirdparty/build/arrow/java
|
|
mvn clean install -pl plasma -am -Dmaven.test.skip
|
|
popd
|
|
pushd $ROOT_DIR/../java
|
|
mvn clean install -Dmaven.test.skip
|
|
check_style=$(mvn checkstyle:check)
|
|
echo "${check_style}"
|
|
[[ ${check_style} =~ "BUILD FAILURE" ]] && exit 1
|
|
|
|
# test non-raylet
|
|
sed -i 's/^use_raylet.*$/use_raylet = false/g' $ROOT_DIR/../java/ray.config.ini
|
|
mvn_test=$(mvn test)
|
|
echo "${mvn_test}"
|
|
[[ ${mvn_test} =~ "BUILD SUCCESS" ]] || exit 1
|
|
|
|
# test raylet
|
|
sed -i 's/^use_raylet.*$/use_raylet = true/g' $ROOT_DIR/../java/ray.config.ini
|
|
mvn_test=$(mvn test)
|
|
echo "${mvn_test}"
|
|
[[ ${mvn_test} =~ "BUILD SUCCESS" ]] || exit 1
|
|
|
|
popd
|