Why Go Was Called Impossible
By 1997 chess had fallen to brute force: Deep Blue searched millions of positions per second, scored each with a handcrafted evaluation function, and outcalculated Kasparov. Everyone assumed Go was next. It took nineteen more years, and the reason is two separate walls, each fatal on its own.
Wall 1: the tree is too big to search. Chess offers about 35 legal moves per turn over roughly 80 turns, a game tree near 10123. Go offers about 250 moves per turn over roughly 150 turns: near 10360. The number of legal board positions alone is about 2 × 10170, dwarfing the 1080 atoms in the observable universe. No conceivable brute force touches this. You met this wall before: it is the curse that killed the Q-table in the Q-learning blog, at planetary scale.
Wall 2: no one can write the evaluation function. Deep Blue's search worked because a human could hand it a scoring rule: count material, add positional bonuses. Go has no material. A stone's worth depends on subtle whole-board relationships that even top professionals describe as intuition, not calculation. Cutting the search short is useless if you cannot score the position where you stopped.
AlphaGo's thesis, in one sentence: replace both impossible ingredients with learned neural networks. Learn where to look (a policy) so the tree gets narrow, and learn who is winning (a value) so the tree can be cut short. Search still happens, but guided and truncated by intuition that was trained, not written.
The Two Instincts: Policy and Value, Reunited
Watch a Go professional study a position and two instincts fire at once. First: of the hundreds of legal moves, only a handful even deserve a glance. Second: a felt sense of who is ahead, before reading a single sequence. Name these in the vocabulary of this series and they are old friends:
The policy tames breadth: instead of 250 candidate moves, search only the dozen the policy rates plausible. The value tames depth: instead of playing every line to the end, stop anywhere and ask the value network who stands better. Wall 1 falls to the policy, wall 2 falls to the value, and what remains between them is a search small enough to run in match time.
The 2016 Pipeline: Four Networks, Built in Order
The original AlphaGo built its instincts in an assembly line of four networks, and every stage is a technique you already own.
Stage 1: the SL policy, learned from humans
A 13-layer convolutional network pσ trained by supervised learning on about 30 million positions from strong human games on the KGS server, predicting the expert's next move. It reached about 57 percent accuracy, and that modest-sounding number already encodes a lifetime of Go pattern knowledge. In modern language this is exactly pretraining on human data, the SFT stage of the LLM recipe:
Stage 2: a fast rollout policy, for speed
A tiny linear policy pπ, only about 24 percent accurate but roughly a thousand times faster (microseconds instead of milliseconds per move). Its job comes later: playing entire games to the end inside the search, where volume matters more than precision.
Stage 3: the RL policy, sharpened by self-play REINFORCE
Predicting human moves is not the same as winning. So AlphaGo initialized a new policy pρ from the SL weights and improved it with exactly the REINFORCE you know from the PPO guide: play full games against randomly sampled previous versions of itself, and push up the probability of every move in won games, down in lost ones:
Stage 4: the value network, a critic for Go
Finally, a value network vθ was trained by regression to predict the winner from a single position, using games the RL policy played against itself. One crucial detail worth remembering: training on many positions from the same game overfits badly, because consecutive positions are nearly identical with the same label, so DeepMind generated 30 million separate self-play games and sampled one position from each. Correlated data breaking a value learner is the same disease the replay buffer treats in DQN.
MCTS: Where the Networks Meet the Tree
The networks alone play strong amateur Go. The leap to superhuman comes from wiring them into Monte Carlo Tree Search, the same search you met guiding reasoning models in the PRM blog. Each move in a real game triggers thousands of simulated look-aheads, and every simulation walks four phases.
Select. From the root, repeatedly pick the child that maximizes value plus an exploration bonus shaped by the policy prior:
Run the selection rule on real numbers. A node has been visited 100 times (√100 = 10), c = 1.5, three candidates:
Expand. Reaching a leaf, add its children with priors from the policy network. Evaluate. Score the leaf twice: ask the value network, and also play one fast game to the end with the rollout policy, then blend:
Backup. The leaf's score flows up the path it came from, updating every ancestor's N and Q. After thousands of simulations, AlphaGo plays the root move with the most visits, the most robust statistic the search produced.
BUT WAIT The raw policy network already beats strong classical programs without any search. Why bother with MCTS at all instead of just playing the network's favorite move? ▶
Because a single forward pass is a reflex, and reflexes have blind spots that verification catches.
First, search is an error corrector. The policy's probabilities encode pattern intuition, not reading. A move can look shapely and lose to a forced ten-move sequence; only rolling the position forward exposes it. The search evaluates thousands of concrete futures, and the averaged Q values quietly overrule the prior wherever the prior is wrong.
Second, averaging beats a point estimate. The value network is imperfect, but the backup phase averages its evaluations over thousands of distinct leaves. Uncorrelated errors partially cancel, so the search's aggregate judgment is sharply better calibrated than any single network call.
Third, and deepest: search converts compute into strength at play time. The same trained network plays markedly stronger with more simulations per move, no retraining involved. You know this phenomenon by its modern name: test-time compute scaling, the exact effect behind PRM-guided search and long-thinking reasoning models. AlphaGo is where the field first demonstrated it at scale, and section 6 turns it into a training principle.
March 2016, and Move 37
First blood came quietly: in October 2015 AlphaGo beat the European champion Fan Hui 5 to 0, the first time a program defeated a professional on the full board without handicap. The world match came five months later: Lee Sedol, holder of 18 international titles, one million dollars, tens of millions watching live. AlphaGo won 4 to 1.
The moment everyone remembers is move 37 of game 2: a shoulder hit on the fifth line, a move that violates centuries of orthodox joseki instinct. Commentators assumed a bug. Lee Sedol left the room. The telling number came from AlphaGo itself: its own policy network, trained on human play, assigned the move a prior of roughly 1 in 10,000. The pattern instinct said humans almost never play this; the search played thousands of futures through it anyway, and the value network kept insisting those futures were winning. The move was not in the training data. It was discovered, by exploration overruling the prior, the PUCT bonus doing exactly what the worked example in section 4 showed at small scale.
Honesty requires game 4 too: Lee's move 78, a wedge his own peers called divine, pushed AlphaGo into positions so far outside its self-play distribution that its evaluations broke down and it played a string of amateurish moves before resigning. Out-of-distribution collapse in a value function: a failure mode this series has met before, and one that search alone does not fix.
AlphaGo Zero: Remove the Humans
Eighteen months later DeepMind published the version that matters most for everything that came after. AlphaGo Zero threw away the assembly line: no human games, no separate networks, no rollout policy, no handcrafted features. One residual network with two heads, policy p and value v, starting from random weights, learning entirely from games against itself. Within 72 hours it beat the version that defeated Lee Sedol by 100 games to 0.
The engine of it is one beautiful realization: MCTS is a policy improvement operator. Feed the network's raw policy into a search, and the visit counts that come out are a strictly better policy than what went in, because the search verified and corrected the prior against thousands of concrete futures. So make the search's output the training target:
Now close the loop. A better policy makes a stronger search (better priors focus the tree). A stronger search makes better training targets. Better targets make a better policy. Around and around, from random weights to superhuman, powered by nothing but the rules of the game: 4.9 million self-play games with just 1,600 simulations per move, and along the way Zero rediscovered centuries of human joseki, kept the ones it liked, and discarded the rest for lines of its own.
Two simplifications inside the loop deserve a highlight. The rollouts are gone: the value head alone evaluates leaves, the network having grown trustworthy enough to drop the Monte Carlo half of the λ blend, sliding fully to the bootstrapped end of the MC-TD dial. And the dual heads share one body: position understanding is one representation with two read-outs, policy and value as two views of the same knowledge.
BUT WAIT Zero starts from random weights and only ever plays itself. Where does new information enter? This sounds like garbage teaching garbage again. ▶
Same question you asked about Q-learning and TD, and the answer has the same skeleton: there is a source of ground truth, and there is an amplifier that spreads it.
The ground truth is the game itself. Every self-play game ends, and the rules declare a winner. That z = ±1 is not a network's opinion, it is reality, exactly like the terminal reward that anchored the Q-table. The value head is regressed onto real outcomes from move one, so truth enters the system thousands of times per training batch, no human required.
The amplifier is the search. At any skill level, search-with-the-network beats the raw network, provably improving on its own prior by verification. So the training target is always slightly stronger than the current student. The gap between student and target is the new information, manufactured fresh at every iteration by spending compute.
And the curriculum is automatic. The opponent is always exactly at the learner's level, because it is the learner. No wasted games against too-strong or too-weak opposition; the difficulty ratchets in lockstep with ability, from random flailing to superhuman, with every intermediate lesson arriving exactly when the student can absorb it.
Garbage in, yes, on day one. But reality leaks in at every game's end, the search amplifies it, and the loop compounds. Seventy-two hours later the garbage was the strongest Go player that had ever existed.
The Lineage: a Story of Removal
You have seen this narrative shape before: the Beyond-PPO story was a sequence of removals, each version deleting a component and getting stronger. AlphaGo's family tree is the same story, and each deletion taught the field something permanent.
AlphaGo (2016) proved learned intuition plus search beats humans, using human data, four networks, and rollouts. AlphaGo Zero (2017) removed the human data, the rollouts, and the separate networks: self-play with search-as-teacher was enough, and stronger. AlphaZero (late 2017) removed the Go-specific design: the identical algorithm, with zero game-specific tuning beyond the rules, mastered chess and shogi as well, crushing the strongest conventional engines. MuZero (2019) removed the rules themselves: it learns an internal model of the environment's dynamics and runs its search entirely inside that learned latent model, matching AlphaZero at board games while also conquering Atari from pixels, planning in a world it imagined.
The Same Ideas in Your World
Line AlphaGo's components up against this series and the mapping is nearly one to one:
| AlphaGo component | What you know it as | The modern LLM echo |
|---|---|---|
| SL policy on human games | imitation learning | pretraining + SFT on human text |
| RL policy via self-play | REINFORCE from the PPO guide | the RL stage: PPO, GRPO on model outputs |
| Value network | the critic, V(s) | reward and value models, PRMs scoring partial work |
| λ blend of rollout and v_θ | the MC-versus-TD dial, GAE's spirit | blending sampled outcomes with learned scores |
| PUCT exploration bonus | ε-greedy, clip-higher: keep rare moves alive | sampling temperature and entropy in RL rollouts |
| MCTS with priors | PRM-guided search from the PRM blog | best-of-N, step beam search, test-time compute |
| Zero's search-to-target distillation | expert iteration | reasoning training: generate with heavy compute, verify, retrain on the survivors |
| The game's win signal z | truth entering at terminals | RLVR: rule-based verifiable rewards, the R1 recipe |
The last two rows are the ones to sit with. AlphaGo Zero's loop, spend compute searching, keep what verification approves, distill it back into the model, repeat, is structurally the recipe training today's reasoning models: sample many chains of thought, keep the ones a verifier confirms, train on those. And Zero's reliance on the game's built-in win signal rather than a learned judge is the same design choice DeepSeek-R1 made when it picked rule-based rewards over neural reward models. The board changed from 19 by 19 to a token sequence; the loop survived intact.
The Cheat Sheet
| Question | The crisp answer |
|---|---|
| Why was Go hard | 10³⁶⁰ game tree kills search, and no human-writable evaluation function exists to truncate it |
| The core idea | learn a policy to narrow the tree and a value to cut it short, then search the little that remains |
| The 2016 networks | SL policy (imitate humans), fast rollout policy (speed), RL policy (REINFORCE self-play), value net (predict winner, de-correlated data) |
| The search | MCTS with PUCT: argmax of Q plus a prior-scaled bonus that decays with visits; leaf scored by value net blended λ = 0.5 with a fast rollout; play the most-visited root move |
| Move 37 | prior said 1 in 10,000, search and value said winning: discovery by exploration overruling imitation |
| Zero's twist | MCTS is a policy improvement operator, so train the network to imitate its own search: (z − v)² − π log p, no human data, rollouts gone |
| The lineage | AlphaGo → Zero (drop human data) → AlphaZero (drop game-specific design) → MuZero (drop the rules, learn the model) |
| Why it still matters | propose with a model, verify with search, distill the verified result back: the template of test-time compute and reasoning-model training |
Go defeated brute force with a 10³⁶⁰ tree and an evaluation no programmer could write, so AlphaGo learned both missing pieces: a policy network for where to look and a value network for who is winning, then wired them into Monte Carlo Tree Search, priors steering exploration, values truncating depth, rollouts and bootstraps blended on the MC-TD dial. Trained by imitation, sharpened by REINFORCE self-play, it beat Lee Sedol and produced move 37, a move its own human-trained prior rated one in ten thousand. AlphaGo Zero then closed the deeper loop: search improves on the policy that guides it, so let the network imitate its own search, and from random weights and pure self-play it surpassed everything in three days. Strip away the humans, the game-specific design, and finally the rules, and what remains is the recipe that outlived Go: propose with a network, verify by spending compute, distill what survives, repeat.