ray/rllib/algorithms/ddppo

Decentralized Distributed Proximal Policy Optimization (DDPPO)

Overview

PPO is a model-free on-policy RL algorithm that works well for both discrete and continuous action space environments. PPO utilizes an actor-critic framework, where there are two networks, an actor (policy network) and critic network (value function).

Distributed PPO Algorithms

Distributed baseline PPO

See implementation here

Asychronous PPO (APPO)

See implementation here

Decentralized Distributed PPO (DDPPO) ..

.. removes the assumption that gradient-updates must be done on a central node. Instead, gradients are computed remotely on each data collection node and all-reduced at each mini-batch using torch distributed. This allows each worker’s GPU to be used both for sampling and for training.

See implementation here

Documentation & Implementation:

Decentralized Distributed Proximal Policy Optimization (DDPPO)

Detailed Documentation

Implementation