e6e10ce4cf
Why are these changes needed? Also: Add validation to make sure multi-gpu and micro-batch is not used together. Update A2C learning test to hit the microbatching branch. Minor comment updates. |
||
|---|---|---|
| .. | ||
| tests | ||
| __init__.py | ||
| a3c.py | ||
| a3c_tf_policy.py | ||
| a3c_torch_policy.py | ||
| README.md | ||
Asynchronous Advantage Actor-Critic (A3C)
Overview
Advantage Actor-Critic proposes two distributed model-free on-policy RL algorithms, A3C and A2C. These algorithms are distributed versions of the vanilla Policy Gradient (PG) algorithm with different distributed execution patterns. The paper suggests accelerating training via scaling data collection, i.e. introducing worker nodes, which carry copies of the central node's policy network and collect data from the environment in parallel. This data is used on each worker to compute gradients. The central node applies each of these gradients and then sends updated weights back to the workers.
In A3C, the worker nodes generate data asynchronously and compute gradients from the data. These computed gradients are then sent to the central node. Note that the workers in A3C may be slightly out-of-sync with the central node due to asynchrony, which may induce biases in learning (on-policy loss function).