Snake, learned by playing itself.

How it works

The game. Each turn every snake sees the board $s_t$ and picks a move $a_t \in \{\uparrow, \downarrow, \leftarrow, \rightarrow\}$. The only reward is the outcome: $r = +1$ for winning, $-1$ for dying.

The network. One small CNN reads 8 planes centred on its own snake (bodies, heads, food, health) and outputs a policy and a value: $$\pi_\theta(a \mid s), \qquad V_\theta(s) \approx \mathbb{E}\Big[\textstyle\sum_k \gamma^k r_{t+k}\Big].$$

Learning. PPO nudges the policy toward moves that turned out better than expected, but never too far in one update: $$L(\theta) = \mathbb{E}_t\Big[\min\big(\rho_t \hat A_t,\ \operatorname{clip}(\rho_t, 1-\epsilon, 1+\epsilon)\, \hat A_t\big)\Big],\qquad \rho_t = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_\text{old}}(a_t \mid s_t)},$$ where the advantage $\hat A_t = \sum_l (\gamma\lambda)^l \delta_{t+l}$, with $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$, says how much better a move was than the network expected.

Self-play. Every snake is the same network, so it learns by playing copies of itself (and older versions of itself). After 2.5 hours on a laptop it beats the best hand-written bot in 91% of games.

source