RL-Games — Q-Learning, DQN and PPO from scratch

Stack

What happened?

1. 01

   The question

   Can you really understand reinforcement learning without opening the black box? My bet: no. You have to code every algorithm yourself.

2. 02

   The problem

   Stable-Baselines3 trains an agent in three lines. Handy, but you learn nothing about what happens inside: the Bellman equation, exploration, replay buffers, clipping…

3. 03

   The hypothesis

   By coding each algorithm from scratch (a NumPy Q-table, then DQN and PPO in PyTorch) and comparing it to reference implementations, I will understand why it works, and when it does not.

4. 04

   The build

   A progression from simple to complex: tabular Q-Learning (FrozenLake, Taxi), DQN with target network and experience replay (MountainCar), REINFORCE, then a full Actor-Critic PPO with GAE (LunarLander). Finally, a Flappy Bird rebuilt from scratch as a Gymnasium environment, used to train a PPO agent.

5. 05

   The failure

   On Flappy Bird, the curves plateaued at a reward of 300 with a score of 0. Yet, watching the agent play, it was clearly passing the pipes. The bug was in my environment: the score was only checked on the next pipe, which by design ignores pipes already passed. The agent played well; my counter just could not see it.

6. 06

   The discovery

   In RL, the environment is part of the experiment: a badly coded reward or counter skews every conclusion. You have to watch the agent play, not just the curves.

7. 07

   The result

   Flappy Bird: 80 pipes per game in 20 out of 20 games, without a single crash, which is the maximum possible before the 3,000-step limit. On LunarLander, my hand-coded PPO reaches 269.6 on its best game, with a mean of 77.8: not yet the 200 threshold that marks the environment as solved.

8. 08

   The next frontier

   Stabilise my own PPO on LunarLander, make Flappy Bird harder (variable gaps, moving pipes), and move to learning directly from pixels.