How Networks Learn: Loss & Gradient Descent
Modules 5 and 7 both ended in the same place: a model has adjustable parameters, and a loss function measures how wrong it currently is. This module covers the algorithm that actually does the adjusting — gradient descent — which trains everything from simple linear regression to the largest neural networks in existence, including large language models.
Picture the loss as a landscape
Imagine plotting the loss for every possible value of a parameter — you get a curve (or, with more parameters, a higher-dimensional surface). Training a model means finding the lowest point on that surface: the parameter values that make the loss as small as possible. Gradient descent finds that low point the way a ball would find the bottom of a valley if you just let it roll downhill.
The gradient: which way is "downhill"?
The gradient of the loss function at a given point tells you the direction of steepest increase. To go downhill, you move in the opposite direction of the gradient. The update rule for a parameter is:
The learning rate controls how big a step you take each time. Too small, and training crawls along agonizingly slowly. Too large, and you can overshoot the valley entirely and bounce back and forth — or even diverge, with the loss getting worse each step. Try it below.
Scaling up: many parameters, and backpropagation
The demo above adjusts one number. A real neural network might have millions or billions of parameters (every weight and bias in every layer). Gradient descent still works the same way — compute the gradient for every single parameter, then nudge each one downhill — but computing all of those gradients efficiently requires an algorithm called backpropagation, which uses calculus's chain rule to work backward from the output layer's error to figure out how much each earlier weight contributed to the mistake.
You don't need to compute backpropagation by hand to use or understand modern AI — every deep learning framework does it automatically — but knowing that it exists explains why: training a large model takes so much computation (every parameter needs a gradient, every step), and why more layers means more to compute per step.
An epoch, a batch, and "training a model"
In practice, gradient descent runs on batches of examples at a time rather than one point at a time (that variant is called stochastic gradient descent, or SGD), and a full pass through the entire training dataset is called an epoch. "Training a model" typically means running many epochs — thousands of small downhill steps — until the loss stops meaningfully improving.