2cdb76789e
See #23676 for context. This is another attempt at that as I figured out what's going wrong in `bazel test`. Supersedes #24828. Now that there are Python 3.10 wheels for Ray 1.13 and this is no longer a blocker for supporting Python 3.10, I still want to make `bazel test //python/ray/tests/...` work for developing in a 3.10 env, and make it easier to add Python 3.10 tests to CI in future. The change contains three commits with rather descriptive commit message, which I repeat here: Pass deps to py_test in py_test_module_list Bazel macro py_test_module_list takes a `deps` argument, but completely ignores it instead of passes it to `native.py_test`. Fixing that as we are going to use deps of py_test_module_list in BUILD in later changes. cpp/BUILD.bazel depends on the broken behaviour: it deps-on a cc_library from a py_test, which isn't working, see upstream issue: https://github.com/bazelbuild/bazel/issues/701. This is fixed by simply removing the (non-working) deps. Depend on conftest and data files in Python tests BUILD files Bazel requires that all the files used in a test run should be represented in the transitive dependencies specified for the test target. For py_test, it means srcs, deps and data. Bazel enforces this constraint by creating a "runfiles" directory, symbolic links files in the dependency closure and run the test in the "runfiles" directory, so that the test shouldn't see files not in the dependency graph. Unfortunately, the constraint does not apply for a large number of Python tests, due to pytest (>=3.9.0, <6.0) resolving these symbolic links during test collection and effectively "breaks out" of the runfiles tree. pytest >= 6.0 introduces a breaking change and removed the symbolic link resolving behaviour, see pytest pull request https://github.com/pytest-dev/pytest/pull/6523 for more context. Currently, we are underspecifying dependencies in a lot of BUILD files and thus blocking us from updating to newer pytest (for Python 3.10 support). This change hopefully fixes all of them, and at least those in CI, by adding data or source dependencies (mostly for conftest.py-s) where needed. Bump pytest version from 5.4.3 to 7.0.1 We want at least pytest 6.2.5 for Python 3.10 support, but not past 7.1.0 since it drops Python 3.6 support (which Ray still supports), thus the version constraint is set to <7.1. Updating pytest, combined with earlier BUILD fixes, changed the ground truth of a few error message based unit test, these tests are updated to reflect the change. There are also two small drive-by changes for making test_traceback and test_cli pass under Python 3.10. These are discovered while debugging CI failures (on earlier Python) with a Python 3.10 install locally. Expect more such issues when adding Python 3.10 to CI. |
||
|---|---|---|
| .. | ||
| build | ||
| env | ||
| k8s | ||
| lint | ||
| pipeline | ||
| run | ||
| ci.sh | ||
| keep_alive | ||
| README.md | ||
| remote-watch.py | ||
| repro-ci.py | ||
| suppress_output | ||
CI process
This document is a work-in-progress. Please double-check file/function/etc. names for changes, as this document may be out of sync.
Dependencies
All dependencies (e.g. apt, pip) should be installed in install_dependencies(), following the same pattern as
those that already exist.
Once a dependency is added/removed, please ensure that shell environment variables are persisted appropriately, as CI
systems differ on when ~/.bashrc et al. are reloaded, if at all. (And they are not necessarily idempotent.)
Bazel, environment variables, and caching
Any environment variables passed to Bazel actions (e.g. PATH) should be idempotent to hit the Bazel cache.
If a different PATH gets passed to a Bazel action, Bazel will not hit the cache, and you might trigger a full rebuild
when you really expect an incremental (or no-op) build for an option (say pip install -e . after bazel build //...).
Invocation
The CI system (such as Travis) must source (not execute) ci/ci.sh and pass the action(s) to execute.
The script either handles the work or dispatches it to other script(s) as it deems appropriate.
This helps ensure any environment setup/teardown is handled appropriately.
Development best practices & pitfalls (read before adding a new script)
Before adding new scripts, please read this section.
First, please consider modifying an existing script instead (e.g. add your code as a separate function). Adding new scripts has a number of pitfalls that easily take hours (even days) to track down and fix:
-
When calling other scripts (as executables), environment variables (like
PATH) cannot propagate back up to the caller. Often, the caller expects such variables to be updated. -
When sourcing other scripts, global state (
ROOT_DIR,main,set -e, etc.) may be overwritten silently, causing unexpected behavior.
The following practices can avoid such pitfalls while maintaining intuitive control flow:
-
Put all environment-modifying functions in the top-level shell script, so that their invocation behaves intuitively. (The sheer length of the script is a secondary concern and can be mitigated by keeping functions modular.)
-
Avoid adding new scripts if possible. If it's necessary that you do so, call them instead of sourcing them. Note that this implies new scripts should not modify the environment, or the caller will not see such changes!
-
Always add code inside a function, not at global scope. Use
localfor variables where it makes sense. However, be careful and know the shell rules: for example, e.g.local x=$(false)succeeds even underset -e.
Ultimately, it's best to only add new scripts if they might need to be executed directly by non-CI code, as in that case, they should probably not use CI entrypoints (which assume exclusive control over the machine).