mirror of
https://github.com/vale981/ray
synced 2025-03-08 19:41:38 -05:00
ee08c8274b
* Implement sharding in the Ray core * Single node Python modifications to do sharding * Do the sharding in redis.cc * Pipe num_redis_shards through start_ray.py and worker.py. * Use multiple redis shards in multinode tests. * first steps for sharding ray.global_state * Fix problem in multinode docker test. * fix runtest.py * fix some tests * fix redis shard startup * fix redis sharding * fix * fix bug introduced by the map-iterator being consumed * fix sharding bug * shard event table * update number of Redis clients to be 64K * Fix object table tests by flushing shards in between unit tests * Fix local scheduler tests * Documentation * Register shard locations in the primary shard * Add plasma unit tests back to build * lint * lint and fix build * Fix * Address Robert's comments * Refactor start_ray_processes to start Redis shard * lint * Fix global scheduler python tests * Fix redis module test * Fix plasma test * Fix component failure test * Fix local scheduler test * Fix runtest.py * Fix global scheduler test for python3 * Fix task_table_test_and_update bug, from actor task table submission race * Fix jenkins tests. * Retry Redis shard connections * Fix test cases * Convert database clients to DBClient struct * Fix race condition when subscribing to db client table * Remove unused lines, add APITest for sharded Ray * Fix * Fix memory leak * Suppress ReconstructionTests output * Suppress output for APITestSharded * Reissue task table add/update commands if initial command does not publish to any subscribers. * fix * Fix linting. * fix tests * fix linting * fix python test * fix linting
42 lines
1.3 KiB
Bash
Executable file
42 lines
1.3 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
|
|
# Cause the script to exit if a single command fails.
|
|
set -e
|
|
|
|
# show explicitly which commands are currently running
|
|
set -x
|
|
|
|
ROOT_DIR=$(cd "$(dirname "${BASH_SOURCE:-$0}")"; pwd)
|
|
|
|
DOCKER_SHA=$($ROOT_DIR/../../build-docker.sh --output-sha --no-cache --skip-examples)
|
|
echo "Using Docker image" $DOCKER_SHA
|
|
|
|
python $ROOT_DIR/multi_node_docker_test.py \
|
|
--docker-image=$DOCKER_SHA \
|
|
--num-nodes=5 \
|
|
--num-redis-shards=10 \
|
|
--test-script=/ray/test/jenkins_tests/multi_node_tests/test_0.py
|
|
|
|
python $ROOT_DIR/multi_node_docker_test.py \
|
|
--docker-image=$DOCKER_SHA \
|
|
--num-nodes=5 \
|
|
--num-redis-shards=5 \
|
|
--num-gpus=0,1,2,3,4 \
|
|
--num-drivers=7 \
|
|
--driver-locations=0,1,0,1,2,3,4 \
|
|
--test-script=/ray/test/jenkins_tests/multi_node_tests/remove_driver_test.py
|
|
|
|
python $ROOT_DIR/multi_node_docker_test.py \
|
|
--docker-image=$DOCKER_SHA \
|
|
--num-nodes=5 \
|
|
--num-redis-shards=2 \
|
|
--num-gpus=0,0,5,6,50 \
|
|
--num-drivers=100 \
|
|
--test-script=/ray/test/jenkins_tests/multi_node_tests/many_drivers_test.py
|
|
|
|
python $ROOT_DIR/multi_node_docker_test.py \
|
|
--docker-image=$DOCKER_SHA \
|
|
--num-nodes=1 \
|
|
--mem-size=60G \
|
|
--shm-size=60G \
|
|
--test-script=/ray/test/jenkins_tests/multi_node_tests/large_memory_test.py
|