hiro/ray
1
0
Fork 0
mirror of https://github.com/vale981/ray synced 2025-03-06 10:31:39 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions

[tune/ci] Fix GRPC resource exhausted test for tune trainables (#24467)

Browse source 
#24421 increased the default maximum GRPC limit to 250MB, which broke a Tune test that catches too large training functions.

This PR fixes this test by increasing the size of the experiment. However, please note that this leads to an inconsistency: For training functions of size 100 < fn < 250, an error will be raised only at runtime when trying to start the actor:

```
ValueError: The actor ImplicitFunc is too large (125 MiB > FUNCTION_SIZE_ERROR_THRESHOLD=95 MiB). Check that its definition is not implicitly capturing a large array or other object in scope. Tip: use ray.put() to put large objects in the Ray object store.
```

But it will successfully pass the registration stage `self._run_identifier = Experiment.register_if_needed(run)`.

cc @ericl should we set the default limit back to 100 MB (or maybe set the FUNCTION_SIZE_ERROR_THRESHOLD to 250 or whatever the GRPC limit is?)
This commit is contained in:
Kai Fricke 2022-05-04 18:32:13 +01:00 • committed by GitHub
parent 6bd65ceb1c
commit b05531177c
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
python/ray/tune/tests/test_tune_restore.py
View file

@ -556,7 +556,7 @@ class ResourceExhaustedTest(unittest.TestCase):
from sklearn.datasets import fetch_olivetti_faces from sklearn.datasets import fetch_olivetti_faces
a_large_array = [] a_large_array = []
for i in range(10): for i in range(25):
a_large_array.append(fetch_olivetti_faces()) a_large_array.append(fetch_olivetti_faces())
def training_func(config): def training_func(config):