We want to use `clangd` as the language server.
`clangd` is an awesome language server that has many features and is very accurate.
But it needs a `compile_commands.json` to work.
This PR adds a popular bazel rule to generate this file.
Builds are currently failing because `mirror.bazel.build`'s SSL certificate expired. This PR adds another bazel mirror to avoid this problem.
Builds are still failing because https://github.com/jupp0r/prometheus-cpp explicitly lists `mirror.bazel.build`.
Since ray supports Redis as a storage backend, we should ensure the code path with Redis as storage is still being covered e2e.
The tests don't run for a while after we switch to memory mode by default. This PR tries to fix this and make it run with every commit.
In the future, if we support more and more storage backends, this should be revised to be more efficient and selective. But now I think the cost should be ok.
This PR is part of GCS HA testing-related work.
Currently, we are not running doc notebooks in CI due to a bazel misconfiguration - we are using `glob` in a top level package in order to get the paths for the notebooks, but those are contained inside subpackages, which glob purposefully ignores. Therefore, the lists of notebooks to run are empty. This PR fixes that by:
* Running the `py_test_run_all_notebooks` macro inside the relevant subpackages
* Editing the `test_myst_doc.py` script to allow for recursive search for the target file, allowing to deal with mismatches between `name` and `data` arguments in `py_test_run_all_notebooks`
* Setting the `allow_empty=False` flag inside `glob` calls in our macros to ensure that this oversight is caught early
* Enabling detection of changes in doc folder for `*.ipynb` and `BUILD` files
This PR also adds a GPU runner for doc tests, allowing one of our examples to pass - and setting the infra for more to come. Finally, a misconfigured path for one set of doc tests is also fixed.
Add bazel platform plugin for ray setup deps.
It will fail to build java related package on ubuntu lastest (ubuntu 20)/mac lastest 11.x version since bazel tools put a wrong platform verion in its deps, so all of users might get such exception
```
ERROR: /github/home/.cache/bazel/_bazel_root/fa5a074cd6f1[25](https://github.com/ray-project/mobius/runs/5273958213?check_suite_focus=true#step:5:25)5c5f5cefe240bb7613/external/bazel_tools/src/conditions/BUILD:61:15: no such target '@platforms//cpu:riscv64': target 'riscv64' not declared in package 'cpu' defined by /github/home/.cache/bazel/_bazel_root/fa5a074cd6f1255c5f5cefe240bb7613/external/platforms/cpu/BUILD and referenced by '@bazel_tools//src/conditions:linux_riscv64'
INFO: Repository remote_coverage_tools instantiated at:
/DEFAULT.WORKSPACE.SUFFIX:3:13: in <toplevel>
Repository rule http_archive defined at:
/github/home/.cache/bazel/_bazel_root/fa5a074cd6f1255c5f5cefe240bb7613/external/bazel_tools/tools/build_defs/repo/http.bzl:364:[31](https://github.com/ray-project/mobius/runs/5273958213?check_suite_focus=true#step:5:31): in <toplevel>
INFO: Repository com_google_absl instantiated at:
/__w/mobius/mobius/streaming/WORKSPACE:16:15: in <toplevel>
/github/home/.cache/bazel/_bazel_root/fa5a074cd6f1255c5f5cefe240bb7613/external/com_github_ray_project_ray/bazel/ray_deps_setup.bzl:217:22: in ray_deps_setup
/github/home/.cache/bazel/_bazel_root/fa5a074cd6f1255c5f5cefe240bb7613/external/com_github_ray_project_ray/bazel/ray_deps_setup.bzl:76:24: in auto_http_archive
Repository rule http_archive defined at:
/github/home/.cache/bazel/_bazel_root/fa5a074cd6f1255c5f5cefe240bb7613/external/bazel_tools/tools/build_defs/repo/http.bzl:364:31: in <toplevel>
ERROR: /github/home/.cache/bazel/_bazel_root/fa5a074cd6f1255c5f5cefe240bb7613/external/bazel_tools/tools/jdk/BUILD:90:11: errors encountered resolving select() keys for @bazel_tools//tools/jdk:jni
```
The bazel dev suggests us to update platform mannually in this issue : https://github.com/bazelbuild/bazel/issues/14097.
It's to say that we reuse the old platforms plugin then fail to select a true jni setting on mips64 or riscv64 instruction if we don't download the new platform.
Co-authored-by: lingxuan.zlx <lingxuan.zlx@antgroup.com>
Updates grpc to 1.44.0 to remove local patch needed for grpc to build.
EDIT: there have been changes to how python is found (mostly removing python2 support) and as such the local python-patch we have for grpc needs to be modified.
This time contributing it to upstream (grpc/grpc#28895) so that it'll get added in a newer version!
For anyone that comes across this:
Here is the error itself for why we need the grpc-python.patch file: https://buildkite.com/ray-project/ray-builders-pr/builds/24659#d293616f-225d-41f9-8de2-03780f12b13f/2386-2416
add patch for newer setuptools, can be removed once grpc 1.44 is release
Why are these changes needed?
With grpc updated to 1.43, one of the patches is not needed.
Patch needed when building locally for newer setuptools version. See grpc/grpc#28392 for more details.
Also needed as a prereq to #21221
To make other system or internal project reuse ray deps bazel function, we need change this local accessing style to global accessing with ray-project namespace.
Co-authored-by: 林濯 <lingxuzn.zlx@antgroup.com>
These dependencies are widely used:
- com.google.common
- com.google.protobuf
- com.google.thirdparty
So that we need to shade them to avoid being conflict with jars introduced by user.
In this PR, we introduce a `bazel_jar_jar` rule for doing these and also shade them in maven pom files.
* Update build rules and patches for darwin_arm64 platform.
Changes include:
Update nelhage/rules_boost package from current version (08/5/2020) to 5/27/2021 version.
Remove rules_boost-undefine-boost_fallthrough.patch, since BOOST_FALLTHROUGH seems to be defined now.
Minor changes to rules_boost-windows-linkopts.patch to use default condition to add -lpthread flag for all platforms.
Add darwin_arm64 config to BUILD files for lib civetweb pulled in via prometheu dependency.
* upgrade boost to 1.74.0 from 1.71.0 to match the udpated build file for windows.
* Fix ray_cpp_pkg
* Use boost/bind/bind.hpp
boost/bind.hpp and global namespace placeholders are deprecated.
* lint
* Use absl::bind_front when possible. Otherwise, NOLINT
* lint
* lint
* lint
* lint
* more lint
* final lint
* trigger build
* Add Bazel config for building with llvm. Upgrade C++ std to 17.
* Fix redis. Try fixing asan and tsan
* Fix asan and format
* Update comments.
Co-authored-by: Chen Shen <scv119@gmail.com>
* clang-tidy
* fix
* fix script
* test clang compiler
* fix clang-tidy rules
* Fix windows and other issues.
* Fix
* Improve information when running check-git-clang-tidy-output.sh on different OS
* Update Bazel (to 3.4.1), gRPC, boringssl, absl.
* Always reinstall Bazel if needing to upgrade to a new Bazel version.
* Add patch for properly detecting Windows Python headers when building gRPC.
* Add minimum Bazel version check.
* Update docs with new Bazel version.
* spdlog support
* fatal abort for spdlog
* print all logs in stderr if no logger given
* fix log test
* install signal handler for spdlog by reusing glog lib
* fix lint
* Avoid duplicated dump
* log rotation and fmt comments
* fix
* make glog flush and RAY_LOG thread-safe
* dump error log to console
* mapping all levels to destination
* hack glog for exporting message to stdout if no base name given
* patch lint
* use stdout logger by default
* add raylet std/err pytest checker
* add worker logs file check
* fix asan check
* loop in glog enums
* fix python lint
* lint for autoindent
* fix indent lint
* make raylet.err is not empty
* Improve reporter module
* Add test_node_physical_stats to test_reporter.py
* Add test_class_method_route_table to test_dashboard.py
* Add stats_collector module for dashboard
* Subscribe actor table data
* Add log module for dashboard
* Only enable test module in some test cases
* CI run all dashboard tests
* Reduce test timeout to 10s
* Use fstring
* Remove unused code
* Remove blank line
* Fix dashboard tests
* Fix asyncio.create_task not available in py36; Fix lint
* Add format_web_url to ray.test_utils
* Update dashboard/modules/reporter/reporter_head.py
Co-authored-by: Max Fitton <mfitton@berkeley.edu>
* Add DictChangeItem type for Dict change
* Refine logger.exception
* Refine GET /api/launch_profiling
* Remove disable_test_module fixture
* Fix test_basic may fail
Co-authored-by: 刘宝 <po.lb@antfin.com>
Co-authored-by: Max Fitton <mfitton@berkeley.edu>
* Revert "[dist] swap mac/linux wheel build order (#9746)"
This reverts commit a9340565ff.
* Revert "Fix package and upload ray jar (#9742)"
This reverts commit c290c308fe.
