mirror of
https://github.com/vale981/ray
synced 2025-03-07 02:51:39 -05:00
23a98a223f
* Doc: redis memory management / automatic flushing. * Address comments * Update redis-memory-management.rst * Change cross ref style
98 lines
3.4 KiB
ReStructuredText
98 lines
3.4 KiB
ReStructuredText
Redis Memory Management (EXPERIMENTAL)
|
|
======================================
|
|
|
|
Ray stores metadata associated with tasks and objects in one or more Redis
|
|
servers, as described in `An Overview of the Internals
|
|
<internals-overview.html>`_. Applications that are long-running or have high
|
|
task/object generation rate could risk high memory pressure, potentially leading
|
|
to out-of-memory (OOM) errors.
|
|
|
|
Here, we describe an experimental feature that transparently flushes metadata
|
|
entries out of Redis memory.
|
|
|
|
Requirements
|
|
------------
|
|
|
|
As of early July 2018, the automatic memory management feature requires building
|
|
Ray from source. We are planning on eliminating this step in the near future by
|
|
releasing official wheels.
|
|
|
|
Building Ray
|
|
~~~~~~~~~~~~
|
|
|
|
First, follow `instructions to build Ray from source
|
|
<installation.html#building-ray-from-source>`__ to install prerequisites. After
|
|
the prerequisites are installed, instead of doing the regular ``pip install`` as
|
|
referenced in that document, pass an additional special flag,
|
|
``RAY_USE_NEW_GCS=on``:
|
|
|
|
.. code-block:: bash
|
|
|
|
git clone https://github.com/ray-project/ray.git
|
|
cd ray/python
|
|
RAY_USE_NEW_GCS=on pip install -e . --verbose # Add --user if you see a permission denied error.
|
|
|
|
Running Ray applications
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
At run time the environment variables ``RAY_USE_NEW_GCS=on`` and
|
|
``RAY_USE_XRAY=1`` are required.
|
|
|
|
.. code-block:: bash
|
|
|
|
export RAY_USE_NEW_GCS=on
|
|
export RAY_USE_XRAY=1
|
|
python my_ray_script.py # Or launch python/ipython.
|
|
|
|
Activate memory flushing
|
|
------------------------
|
|
|
|
After building Ray using the method above, simply add these two lines after
|
|
``ray.init()`` to activate automatic memory flushing:
|
|
|
|
.. code-block:: python
|
|
|
|
ray.init(...)
|
|
|
|
policy = ray.experimental.SimpleGcsFlushPolicy()
|
|
ray.experimental.set_flushing_policy(policy)
|
|
|
|
# My awesome Ray application logic follows.
|
|
|
|
Paramaters of the flushing policy
|
|
---------------------------------
|
|
|
|
There are three `user-configurable parameters
|
|
<https://github.com/ray-project/ray/blob/8190ff1fd0c4b82f73e2c1c0f21de6bda494718c/python/ray/experimental/gcs_flush_policy.py#L31>`_
|
|
of the ``SimpleGcsFlushPolicy``:
|
|
|
|
* ``flush_when_at_least_bytes``: Wait until this many bytes of memory usage
|
|
accumulated in the redis server before flushing kicks in.
|
|
* ``flush_period_secs``: Issue a flush to the Redis server every this many
|
|
seconds.
|
|
* ``flush_num_entries_each_time``: A hint to the system on the number of entries
|
|
to flush on each request.
|
|
|
|
The default values should serve to be non-invasive for lightweight Ray
|
|
applications. ``flush_when_at_least_bytes`` is set to ``(1<<31)`` or 2GB,
|
|
``flush_period_secs`` to 10, and ``flush_num_entries_each_time`` to 10000:
|
|
|
|
.. code-block:: python
|
|
|
|
# Default parameters.
|
|
ray.experimental.SimpleGcsFlushPolicy(
|
|
flush_when_at_least_bytes=(1 << 31),
|
|
flush_period_secs=10,
|
|
flush_num_entries_each_time=10000)
|
|
|
|
In particular, these default values imply that
|
|
|
|
1. the Redis server would accumulate memory usage up to 2GB without any entries
|
|
being flushed, then the flushing would kick in; and
|
|
|
|
2. generally, "older" metadata entries would be flushed first, and the Redis
|
|
server would always keep the most recent window of metadata of 2GB in size.
|
|
|
|
**For advanced users.** Advanced users can tune the above parameters to their
|
|
applications' needs; note that the desired flush rate is equal to (flush
|
|
period) * (num entries each flush).
|