ray/.buildkite/pipeline.gpu.large.yml

- label: ":tv: :steam_locomotive: Train GPU tests "
  conditions: ["RAY_CI_TRAIN_AFFECTED"]
  commands:
    - cleanup() { if [ "${BUILDKITE_PULL_REQUEST}" = "false" ]; then ./ci/travis/upload_build_info.sh; fi }; trap cleanup EXIT
    - PYTHON=3.6 TRAIN_TESTING=1 TUNE_TESTING=1 INSTALL_HOROVOD=1 ./ci/travis/install-dependencies.sh
    # Because Python version changed, we need to re-install Ray here
    - rm -rf ./python/ray/thirdparty_files; rm -rf ./python/ray/pickle5_files; ./ci/travis/ci.sh build
    - pip install -Ur ./python/requirements_ml_docker.txt
    - ./ci/travis/env_info.sh
    - bazel test --config=ci $(./scripts/bazel_export_options) --build_tests_only --test_tag_filters=gpu,gpu_only python/ray/train/...

- label: ":tv: :database: :steam_locomotive: Datasets Train Integration GPU Tests and Examples (Python 3.7)"
  conditions: ["RAY_CI_TRAIN_AFFECTED"]
  commands:
    - cleanup() { if [ "${BUILDKITE_PULL_REQUEST}" = "false" ]; then ./ci/travis/upload_build_info.sh; fi }; trap cleanup EXIT
    - PYTHON=3.7 TRAIN_TESTING=1 DATA_PROCESSING_TESTING=1 ./ci/travis/install-dependencies.sh
    - pip install -Ur ./python/requirements_ml_docker.txt
    - ./ci/travis/env_info.sh
    - bazel test --config=ci $(./scripts/bazel_export_options) --build_tests_only --test_tag_filters=datasets_train doc/...

- label: ":tv: :brain: RLlib: Multi-GPU Tests"
  conditions: ["RAY_CI_RLLIB_AFFECTED"]
  commands:
    - cleanup() { if [ "${BUILDKITE_PULL_REQUEST}" = "false" ]; then ./ci/travis/upload_build_info.sh; fi }; trap cleanup EXIT
    - PYTHON=3.7 RLLIB_TESTING=1 ./ci/travis/install-dependencies.sh
    - pip install -Ur ./python/requirements_ml_docker.txt
    - ./ci/travis/env_info.sh
    # --jobs 2 is necessary as we only need to have at least 2 gpus on the machine
    # and running tests in parallel would cause timeouts as the other scripts would
    # wait for the GPU to become available.
    - bazel test --config=ci $(./scripts/bazel_export_options) --build_tests_only --jobs 2
      --test_tag_filters=multi_gpu --test_env=RAY_USE_MULTIPROCESSING_CPU_COUNT=1 rllib/...
[Train] Rename Ray SGD v2 to Ray Train (#19436) 2021-10-18 22:27:46 -07:00			`- label: ":tv: :steam_locomotive: Train GPU tests "`
			`conditions: ["RAY_CI_TRAIN_AFFECTED"]`
[ci/tune] Add SGD and Tune GPU pipeline step to CI (#18469) * [ci/tune] Add Tune GPU pipeline step to CI * cont. * add sgd gpu tests * format yaml, fix imports * install horovod; fix line wrapping * set GPU per worker to 0.5 * fix import * move test to 4gpu machine * fix lint * lint * set visible devices * pull in tf gpu fix * Fix Tune GPU pipeline step * nit * Disable GPU tests until we have some * Re-add empty rllib tests Co-authored-by: Matthew Deng <matthew.j.deng@gmail.com> 2021-10-02 02:34:05 +01:00			`commands:`
			`- cleanup() { if [ "${BUILDKITE_PULL_REQUEST}" = "false" ]; then ./ci/travis/upload_build_info.sh; fi }; trap cleanup EXIT`
[ci/gpu] Move ML dependency install to Dockerfile (#21711) Instead of installing dependencies in each Buildkite job, let's move this to the Dockerfile instead. This will update GPU tests to always use Python 3.7. 2022-02-01 12:04:55 +00:00			`- PYTHON=3.6 TRAIN_TESTING=1 TUNE_TESTING=1 INSTALL_HOROVOD=1 ./ci/travis/install-dependencies.sh`
			`# Because Python version changed, we need to re-install Ray here`
			`- rm -rf ./python/ray/thirdparty_files; rm -rf ./python/ray/pickle5_files; ./ci/travis/ci.sh build`
[ci/tune] Add SGD and Tune GPU pipeline step to CI (#18469) * [ci/tune] Add Tune GPU pipeline step to CI * cont. * add sgd gpu tests * format yaml, fix imports * install horovod; fix line wrapping * set GPU per worker to 0.5 * fix import * move test to 4gpu machine * fix lint * lint * set visible devices * pull in tf gpu fix * Fix Tune GPU pipeline step * nit * Disable GPU tests until we have some * Re-add empty rllib tests Co-authored-by: Matthew Deng <matthew.j.deng@gmail.com> 2021-10-02 02:34:05 +01:00			`- pip install -Ur ./python/requirements_ml_docker.txt`
			`- ./ci/travis/env_info.sh`
[Train] Rename Ray SGD v2 to Ray Train (#19436) 2021-10-18 22:27:46 -07:00			`- bazel test --config=ci $(./scripts/bazel_export_options) --build_tests_only --test_tag_filters=gpu,gpu_only python/ray/train/...`
[rllib] Add deterministic test to gpu (#19306) Co-authored-by: sven1977 <svenmika1977@gmail.com> 2021-10-26 10:11:39 -07:00
[train][datasets] add example for big data training (#20042) * [train][datasets] add example for big data training * add title docstring * lint and dependencies * add dask_ml requirement 2021-11-05 09:28:48 -07:00			`- label: ":tv: :database: :steam_locomotive: Datasets Train Integration GPU Tests and Examples (Python 3.7)"`
			`conditions: ["RAY_CI_TRAIN_AFFECTED"]`
			`commands:`
			`- cleanup() { if [ "${BUILDKITE_PULL_REQUEST}" = "false" ]; then ./ci/travis/upload_build_info.sh; fi }; trap cleanup EXIT`
[ci/gpu] Move ML dependency install to Dockerfile (#21711) Instead of installing dependencies in each Buildkite job, let's move this to the Dockerfile instead. This will update GPU tests to always use Python 3.7. 2022-02-01 12:04:55 +00:00			`- PYTHON=3.7 TRAIN_TESTING=1 DATA_PROCESSING_TESTING=1 ./ci/travis/install-dependencies.sh`
[train][datasets] add example for big data training (#20042) * [train][datasets] add example for big data training * add title docstring * lint and dependencies * add dask_ml requirement 2021-11-05 09:28:48 -07:00			`- pip install -Ur ./python/requirements_ml_docker.txt`
			`- ./ci/travis/env_info.sh`
			`- bazel test --config=ci $(./scripts/bazel_export_options) --build_tests_only --test_tag_filters=datasets_train doc/...`

[rllib] Add deterministic test to gpu (#19306) Co-authored-by: sven1977 <svenmika1977@gmail.com> 2021-10-26 10:11:39 -07:00			`- label: ":tv: :brain: RLlib: Multi-GPU Tests"`
			`conditions: ["RAY_CI_RLLIB_AFFECTED"]`
			`commands:`
			`- cleanup() { if [ "${BUILDKITE_PULL_REQUEST}" = "false" ]; then ./ci/travis/upload_build_info.sh; fi }; trap cleanup EXIT`
[ci/gpu] Move ML dependency install to Dockerfile (#21711) Instead of installing dependencies in each Buildkite job, let's move this to the Dockerfile instead. This will update GPU tests to always use Python 3.7. 2022-02-01 12:04:55 +00:00			`- PYTHON=3.7 RLLIB_TESTING=1 ./ci/travis/install-dependencies.sh`
[rllib] Add deterministic test to gpu (#19306) Co-authored-by: sven1977 <svenmika1977@gmail.com> 2021-10-26 10:11:39 -07:00			`- pip install -Ur ./python/requirements_ml_docker.txt`
			`- ./ci/travis/env_info.sh`
			`# --jobs 2 is necessary as we only need to have at least 2 gpus on the machine`
			`# and running tests in parallel would cause timeouts as the other scripts would`
			`# wait for the GPU to become available.`
			`- bazel test --config=ci $(./scripts/bazel_export_options) --build_tests_only --jobs 2`
Remove the flaky test tag (#21006) 2021-12-11 01:03:17 -08:00			`--test_tag_filters=multi_gpu --test_env=RAY_USE_MULTIPROCESSING_CPU_COUNT=1 rllib/...`