Reinforcement Learning, Pong

The tuned DQN training run: reward, smoothed performance, win rate, loss, Q-value evolution, episode length and epsilon decay over 2,500 episodes.

Video

Recorded footage of the agent playing.

How it works

• Approach 1 is a basic DQN: a 1.68M-parameter CNN, one Q-network, MSE loss, a batch of 32, epsilon-greedy exploration. A clean reference point.
• Approach 2 adds a target network, prioritized replay, 8 parallel environments, reward shaping, Huber loss and a batch of 512.
• Comparing the two isolates what each change buys. Parallel environments and prioritized replay do most of the work; the target network mostly adds stability rather than score.
• Reward is sparse, one point at a time, and games run 800 to 1000 steps, so a random policy sits near minus 20.

Results

• The tuned DQN reached minus 11.23 average reward over 30 evaluation episodes, with a best episode of plus 6, its first win against the built-in opponent.
• It sustains 1,406-step rallies, six times the random baseline, and reaches minus 18 reward in a quarter of the episodes the baseline needs.
• Training loss dropped 27 times, and Q-value overestimation went from plus 3.5 to minus 0.5.

From the project deck. Baseline is a random policy.

In code

q = policy(states).gather(1, actions)
with torch.no_grad():
    next_a = policy(next_states).argmax(1, keepdim=True)   # double DQN
    tgt = rewards + gamma * target_net(next_states).gather(1, next_a) * (~done)
td = tgt - q
loss = (weights * huber(q, tgt)).mean()                    # prioritized replay
loss.backward(); opt.step()
replay.update_priorities(idx, td.abs().detach())▊

Approach 2's learning step: double-DQN target, Huber loss, priorities from the TD error.

Slides

The full project deck. You can read it right here.

Comments

Thoughts or questions on this project? Leave a note below.