Reinforcement Learning

Q-learning on a Gridworld

An agent that knows nothing learns a path from reward alone. Watch value fill backward from the goal and the greedy arrows snap into a route, then see how cutting off exploration too soon traps it on a worse one.

reinforcement-learningq-learningtemporal-differenceexploration

LiveInteractive · drag, toggle, run it

Reinforcement Learning / Tabular control

Q-learning on a Gridworld

The agent begins with every value at zero and no map of the world. It only feels reward: a small cost each step, a big payoff at the goal, a penalty in a trap. Run episodes and watch value seep backward from the goal until the greedy arrows form a path.

Episodes

goal or trap reached

Greedy policy

—

does not reach goal

Last return

—

reward summed over episode

Epsilon

1.00

decaying with episodes

Click any cell to read its four Q-values, one per action. Q(s, a) is the expected discounted future reward of taking action a in state s, then acting greedily ever after. The greedy arrow points at the action with the largest Q.

Learning rate α0.50

Discount γ0.95

Exploration ε1.00

Slip probability0%

Speed2x

Decay ε over episodesShow policy arrows

Edit the world

Inspect mode: clicking a cell reads its Q-values. Switch to a tool to reshape the world; any edit zeroes the Q-table so learning restarts.

What to watch for

Value flows backward from reward

Reset and run episodes. Only cells next to the goal warm up first, because only they ever see the +1. Each update carries a slice of that value one cell further back: Q(s, a) moves toward r + γ max Q(s', a'), the reward you just got plus the best value of where you landed. After enough episodes the warm front reaches the start.

Explore too little and you get stuck

Drop ε to 0 and reset: the agent always takes its current best guess, which early on is an arbitrary tie, so it can lock onto a dead end and never discover the goal. Keep ε high, or leave decay on so it explores first and exploits later, and the greedy path turns optimal.

Gamma sets how far the agent looks

With γ near its floor the agent leans on immediate reward, so distant cells stay faint and a long route is hard to value. Push γ toward 1 and value carries across the whole grid, letting the agent trade many small step costs for the payoff far away.

Tabular Q-learning, four actions, step reward -0.04, goal +1, trap -1. Cell shade is V(s) = max over actions of Q(s, a); the terracotta outline traces the current greedy path. Update rule: Q(s, a) ← Q(s, a) + α[r + γ max Q(s', a') − Q(s, a)].