mirror of
https://github.com/vale981/ray
synced 2025-03-06 10:31:39 -05:00
99a0088233
* [RLlib] Unify the way we create and use LocalReplayBuffer for all the agents. This change 1. Get rid of the try...except clause when we call execution_plan(), and get rid of the Deprecation warning as a result. 2. Fix the execution_plan() call in Trainer._try_recover() too. 3. Most importantly, makes it much easier to create and use different types of local replay buffers for all our agents. E.g., allow us to easily create a reservoir sampling replay buffer for APPO agent for Riot in the near future. * Introduce explicit configuration for replay buffer types. * Fix is_training key error. * actually deprecate buffer_size field. |
||
|---|---|---|
| .. | ||
| tests | ||
| __init__.py | ||
| cql.py | ||
| cql_tf_policy.py | ||
| cql_torch_policy.py | ||
| README.md | ||
Conservative Q-Learning (CQL)
Overview
CQL is an offline RL algorithm that mitigates the overestimation of Q-values outside the dataset distribution via convservative critic estimates. CQL does this by adding a simple Q regularizer loss to the standard Belman update loss. This ensures that the critic does not output overly-optimistic Q-values and can be added on top of any off-policy Q-learning algorithm (in this case, we use SAC).
Documentation & Implementation:
Conservative Q-Learning (CQL).