mirror of
https://github.com/vale981/ray
synced 2025-03-04 17:41:43 -05:00
e8abffb017
This PR addresses recent failures in the tune cloud tests.
In particular, this PR changes the following:
The trial runner will now wait for potential previous syncs to finish before syncing once more if force=True is supplied. This is to make sure that the final experiment checkpoints exist in the most recent version on remote storage. This likely fixes some flakiness in the tests.
We switched to new cloud buckets that don't interfere with other tests (and are less likely to be garbage collected)
We're now using dated subdirectories in the cloud buckets so that we don't interfere if two tests are run in parallel. Objects are cleaned up afterwards. The buckets are configured to remove objects after 30 days.
Lastly, we fix an issue in the cloud tests where the RELEASE_TEST_OUTPUT file was unavailable when run in Ray client mode (as e.g. in kubernetes).
Local release test runs succeeded.
https://buildkite.com/ray-project/release-tests-branch/builds/189
https://buildkite.com/ray-project/release-tests-branch/builds/191
18 lines
429 B
Python
18 lines
429 B
Python
import logging
|
|
import sys
|
|
|
|
logger = logging.getLogger()
|
|
logger.setLevel(logging.INFO)
|
|
|
|
|
|
def add_handlers(logger: logging.Logger):
|
|
handler = logging.StreamHandler(stream=sys.stderr)
|
|
formatter = logging.Formatter(
|
|
fmt="[%(levelname)s %(asctime)s] %(filename)s: %(lineno)d %(message)s"
|
|
)
|
|
handler.setFormatter(formatter)
|
|
logger.addHandler(handler)
|
|
|
|
|
|
if not logger.hasHandlers():
|
|
add_handlers(logger)
|