hiro/ray
1
0
Fork 0
mirror of https://github.com/vale981/ray synced 2025-03-06 10:31:39 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions

[log_monitor] Always reopen files (#26730)

Browse source 
This PR prevents the log monitor for keeping files open for long periods of time. In settings in which the autoscaler and head node are not tightly coupled, leaving files open implies that the inode for a file never changes, but depending on how fs synchronization between the autoscaler and head node containers works, the inode could change. Thus, we should keep try reopening files.

This is done via setting max open files to 1, so that it's easy to revert this behavior.

Co-authored-by: Alex <alex@anyscale.com>
This commit is contained in:
Alex Wu 2022-07-20 16:17:25 -07:00 • committed by GitHub
parent e043f49957
commit 9e7ddddff7
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 3 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
python/ray/_private/ray_constants.py
View file

@ -264,7 +264,9 @@ WORKER_PROCESS_TYPE_RESTORE_WORKER_DELETE = (
f"ray::DELETE_{WORKER_PROCESS_TYPE_RESTORE_WORKER_NAME}"
)
LOG_MONITOR_MAX_OPEN_FILES = 200
# Effectively reopen every file, every time. To fall back to the old behavior
# one should set this constant to 200.
LOG_MONITOR_MAX_OPEN_FILES = int(os.environ.get("RAY_LOG_MONITOR_MAX_OPEN_FILES", "1"))
# Autoscaler events are denoted by the ":event_summary:" magic token.
LOG_PREFIX_EVENT_SUMMARY = ":event_summary:"