hiro/ray
1
0
Fork 0
mirror of https://github.com/vale981/ray synced 2025-03-04 17:41:43 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions
ray/dashboard/modules/healthz
History
Exact
Yi Cheng dac7bf17d9
[serve] Make serve agent not blocking when GCS is down. (#27526) 
This PR fixed several issue which block serve agent when GCS is down. We need to make sure serve agent is always alive and can make sure the external requests can be sent to the agent and check the status.

- internal kv used in dashboard/agent blocks the agent. We use the async one instead
- serve controller use ray.nodes which is a blocking call and blocking forever. change to use gcs client with timeout
- agent use serve controller client which is a blocking call with max retries = -1. This blocks until controller is back.

To enable Serve HA, we also need to setup:

- RAY_gcs_server_request_timeout_seconds=5
- RAY_SERVE_KV_TIMEOUT_S=5

which we should set in KubeRay.
2022-08-08 16:29:42 -07:00
..
tests [dashboard][2/2] Add endpoints to dashboard and dashboard_agent for liveness check of raylet and gcs (#26408) 2022-07-09 13:09:48 -07:00
__init__.py [dashboard][2/2] Add endpoints to dashboard and dashboard_agent for liveness check of raylet and gcs (#26408) 2022-07-09 13:09:48 -07:00
healthz_agent.py [serve] Make serve agent not blocking when GCS is down. (#27526) 2022-08-08 16:29:42 -07:00
healthz_head.py [dashboard][2/2] Add endpoints to dashboard and dashboard_agent for liveness check of raylet and gcs (#26408) 2022-07-09 13:09:48 -07:00
utils.py [serve] Make serve agent not blocking when GCS is down. (#27526) 2022-08-08 16:29:42 -07:00