The single method behind how ChatGPT learns.
Fog so thick you can't see a step ahead. All you can feel is the slope under your feet. How do you get down?
Descending the mountain of error with the slope underfoot as your compass — gradient descent.
Experiment
Hands-on experiment
Predict first — keep walking against the slope underfoot. Do you always reach the mountain's lowest point?
The mountain of error — descend one step at a time (η=0.3)
Position x
5.50
Error f(x)
7.25
Slope underfoot
5.00
Read more — why it exists · insights · common mistakes · formulasExpand ▾
Why
Why does this exist?
What is AI 'learning', really? Turning the model's knobs (parameters) to reduce error. But with billions of knobs, which way do you turn them?
Trying every direction is impossible. But the derivative instantly reports 'the direction in which error falls steepest from right here' — just as you can feel the slope underfoot even in fog.
Move a little that way, differentiate again, move again — this simple loop is gradient descent, and all of modern AI, from ChatGPT to recommenders, learns on top of it.
Insight
Insights from the video
“AI doesn't 'know' answers — it 'reduces' error.”
The goal of learning isn't truth but minimizing an error function. This shift of view is the key to all of machine learning — define the mountain (the error function) and learning becomes a descent problem.
“The tangent slope from the Derivative lab becomes the compass here.”
'Minima live where the slope is zero' (derivatives) and 'walk against the slope to go down' (gradient descent) are two faces of the same fact.
Misconception
Common misconceptions
Follow the slope down and you always reach the lowest point.
With multiple valleys you can get trapped in a local minimum near your start. No guarantee of the global minimum — which is why AI training restarts from many points and uses tricks like momentum.
Bigger steps (learning rate) mean faster arrival, so bigger is better.
Too big and you leap across the valley to the other side; bigger still and you diverge right off the mountain. Too small and it takes forever. Tuning the learning rate is half of AI training.
Formula
Writing it as math
One step of the foggy descent, written as a formula, is the whole algorithm.
The update rule
New position = current position − learning rate × slope. Positive slope moves you left, negative right — always downhill.
The learning rate's three fates
Exactly what you watched. The right η depends on the terrain's curvature, so practice uses schedules and adaptive methods (Adam, etc.).
Where it stops
Updates halt at zero slope — the valley floor, or possibly a local minimum. The Derivative lab's 'slope hunt' completes here.
In Real Life
Where you meet it in real life
Training deep networks
Even GPT's hundreds of billions of parameters are adjusted by this update rule. 'Training' means gradient descent is running.
Recommendation engines
Your recommendation list is the result of descending the mountain of 'predicted rating minus actual rating'.
Auto-adjust in photo editing
Tuning filter parameters toward 'minimal difference from the original' is the same structure.
Optimization in economics
Cost minimization, utility maximization — following slopes to optima was economics' tool long before AI.
Try Yourself
Test yourself
Q1f(x)=(x−3)², current x=5, learning rate 0.1 — the next position?Show answer ▾
Slope f'(5)=2(5−3)=4. Next = 5 − 0.1×4 = 4.6. One step closer to the valley (x=3).
Q2You land exactly on the valley floor (slope 0). Where does the next step go?Show answer ▾
Nowhere — the move is η×0 = 0. That's why zero-slope points are gradient descent's terminals. (Though it could be a local minimum.)
Q3To climb UP (maximize), how does the update rule change?Show answer ▾
One sign: x ← x + η·f'(x) — gradient ascent. Profit-maximization problems solve with the same algorithm, sign flipped.
Try answering yourself before revealing it — getting it wrong is where learning starts.
Connection
Concepts connect
Previous concept
Correlation
After learning to READ relationships in data comes how a machine FINDS them.
← Correlation labLeads to next
Neural Networks
Descent mastered — now build the mountain itself: the pattern machine stacked from simple functions. The final station.
Go to the Neural Networks lab →Related
Labs worth exploring together
Related lab
Derivatives
The identity of the slope underfoot — build the tangent first, and the compass makes sense.
Go to the Derivatives lab →Related lab
Quadratic Functions
The most common terrain for the mountain of error is a parabola — the vertex is the destination.
Go to the Quadratic Functions lab →