mirror of
https://github.com/vale981/ray
synced 2025-03-06 10:31:39 -05:00
83 lines
2.9 KiB
ReStructuredText
83 lines
2.9 KiB
ReStructuredText
.. _offline-reference-docs:
|
|
|
|
Offline RL
|
|
==========
|
|
|
|
|
|
InputReaders
|
|
------------
|
|
|
|
The InputReader API is used by an individual :py:class:`~ray.rllib.evaluation.rollout_worker.RolloutWorker`
|
|
to produce batches of experiences either from an simulator/environment or from an
|
|
offline source (e.g. a file).
|
|
|
|
Here, we introduce the generic API and its child classes used for reading offline data (for offline RL).
|
|
For details on RLlib's :py:class:`~ray.rllib.evaluation.sampler.Sampler` implementations
|
|
for collecting data from simulators/environments, see the :ref:`Sampler docs here<sampler-docs>`.
|
|
|
|
.. autoclass:: ray.rllib.offline.input_reader.InputReader
|
|
:members:
|
|
|
|
|
|
JsonReader (ray.rllib.offline.json_reader.JsonReader)
|
|
+++++++++++++++++++++++++++++++++++++++++++++++++++++
|
|
|
|
For reading data from offline files (for example when no simulator/environment is available),
|
|
you can use the built-in :py:class:`~ray.rllib.offline.json_reader.JsonReader` class.
|
|
|
|
You will have to change the ``input`` config value from "sampler" (default) to a
|
|
JSON file name (str), a list of JSON files, or a path name (str) that contains JSON files.
|
|
Alternatively, you can specify a callable that takes a :py:class:`~ray.rllib.offline.io_context.IOContext` object as only
|
|
arg and returns a new :py:class:`~ray.rllib.offline.input_reader.InputReader` instance, for example:
|
|
|
|
.. code-block:: python
|
|
|
|
config = {
|
|
"input": lambda io_ctx: MyReader([arg1], [arg2], [io_ctx]),
|
|
}
|
|
|
|
For details on the :py:class:`~ray.rllib.offline.io_context.IOContext` class, see below.
|
|
|
|
.. autoclass:: ray.rllib.offline.json_reader.JsonReader
|
|
:special-members: __init__
|
|
:members:
|
|
|
|
MixedInput (ray.rllib.offline.mixed_input.MixedInput)
|
|
++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
|
|
|
In order to mix different input readers with each other in different custom ratios, you can use
|
|
the :py:class:`~ray.rllib.offline.mixed_input.MixedInput` reader. This reader is chosen
|
|
automatically by RLlib when you provide a dict under
|
|
the ``input`` config key that maps input reader specifiers to probabilities, e.g.:
|
|
|
|
.. code-block:: python
|
|
|
|
"input": {
|
|
"sampler": 0.4, # 40% of samples will come from environment
|
|
"/tmp/experiences/*.json": 0.4, # the rest from different JSON files
|
|
"s3://bucket/expert.json": 0.2,
|
|
}
|
|
|
|
.. autoclass:: ray.rllib.offline.mixed_input.MixedInput
|
|
:special-members: __init__
|
|
:members:
|
|
|
|
|
|
D4RLReader (ray.rllib.offline.d4rl_reader.D4RLReader)
|
|
+++++++++++++++++++++++++++++++++++++++++++++++++++++
|
|
|
|
.. autoclass:: ray.rllib.offline.d4rl_reader.D4RLReader
|
|
:special-members: __init__
|
|
:members:
|
|
|
|
|
|
|
|
IOContext
|
|
---------
|
|
|
|
IOContext instances are used in every InputReader and OutputWriter class. They serve as
|
|
simple containers for the properties: ``log_dir``, ``config``, and ``worker_index``.
|
|
|
|
.. autoclass:: ray.rllib.offline.io_context.IOContext
|
|
:special-members: __init__
|
|
:members:
|