ray/python
mwtian 0330852baf
[Core][Pubsub] Implement Python GCS publisher and subscriber (#20111)
## Why are these changes needed?
This change adds Python publisher and subscriber in `gcs_utils.py`, and GRPC handler on GCS for publishing iva GCS. Error info is migrated to use the GCS-based pubsub, if feature flag `RAY_gcs_grpc_based_pubsub=true`.

Also, add a `--gcs-address` flag to some Python processes. It is not set anywhere yet, but will be set aftering Redis-less bootstrapping work.

Unit tests are added for the Python publisher and subscriber. Migrated error info publishers and subscribers are tested with existing unit tests, e.g. tests calling `ray._private.test_utils.get_error_message()` to ensure error info is published.

GCS based pubsub has gaps in handling deadline, cancelled requests and GCS restarts. So 3 more unit tests are disabled in the `HA GCS` mode. They will be addressed in a separate change.

## Related issue number
2021-11-11 14:59:57 -08:00
..
ray [Core][Pubsub] Implement Python GCS publisher and subscriber (#20111) 2021-11-11 14:59:57 -08:00
requirements [CI] Pin shortuuid to fix CI (#20153) 2021-11-08 12:08:32 -08:00
asv.conf.json [docs] Move all /latest links to /master (#11897) 2020-11-10 10:53:28 -08:00
build-wheel-macos-arm64.sh [CI] [macOS] avoid installing latest setuptools (#20064) 2021-11-04 21:35:03 -07:00
build-wheel-macos.sh [CI] [macOS] avoid installing latest setuptools (#20064) 2021-11-04 21:35:03 -07:00
build-wheel-manylinux2014.sh [dashboard] Rename "new_dashboard" -> "dashboard" (#18214) 2021-09-15 11:17:15 -05:00
build-wheel-windows.sh [Dashboard] Include the dashboard in Windows wheels (#19575) 2021-10-22 17:57:36 -07:00
MANIFEST.in [Build] Another attempt at building Python 3.9 MacOS wheels (#16347) 2021-06-10 10:20:30 -07:00
README-building-wheels.md [build] Build wheels with manylinux2014 (#11621) 2020-11-03 19:36:32 -08:00
requirements.txt [serve] Add Google Cloud Storage as a backend (#20104) 2021-11-10 19:45:19 -08:00
requirements_linters.txt [Lint] Add flake8-bugbear (#19053) 2021-10-03 23:24:11 -07:00
requirements_ml_docker.txt [Deps] Bump tensorflow on Docker image and add Codeowners (#20041) 2021-11-05 00:58:34 -07:00
setup.py [autoscaler] AWS Autoscaler CloudWatch Integration (#18619) 2021-11-09 11:48:55 -08:00