mirror of
https://github.com/vale981/ray
synced 2025-03-06 18:41:40 -05:00
36ee3fd46b
## Why are these changes needed? In python, redis rpush is used to broadcast and store the keys. In this PR, we use gcs kv to store the keys. pubsub is still using redis which need to be remove later. The protocol before this PR: - worker subscribe to redis key spaces - worker write the key of function/actor to (export:sqn, key) - so the other worker will be notified and start to load the data by checking export:sqn This depends on redis for both kv and pubsub, and this PR fix the kv part. After this PR: - worker subscribe to redis key space - For exporting: - worker will find the first key not being hold. This is guaranteed by internal kv which right now is a single thread, atomic db. The worker will just check until it find one key not existing and write it (this is single operation). One optimization right now is to use the import counter as the start offset since this counter means all keys before the counter has already been used. - worker will then write a dummy key to redis key space for broadcasting - For importer - It's working as before, but instead of reading from redis, it will read from gcs kv. This is part in redis removal project. ## Related issue number https://github.com/ray-project/ray/issues/19443 |
||
|---|---|---|
| .. | ||
| ray | ||
| requirements | ||
| asv.conf.json | ||
| build-wheel-macos-arm64.sh | ||
| build-wheel-macos.sh | ||
| build-wheel-manylinux2014.sh | ||
| build-wheel-windows.sh | ||
| MANIFEST.in | ||
| README-building-wheels.md | ||
| requirements.txt | ||
| requirements_linters.txt | ||
| requirements_ml_docker.txt | ||
| setup.py | ||