Gradient Descent

Lesson notes

Overview

Step downhill along the negative gradient to minimise a loss; too big a step overshoots. This topic is rated advanced and is number 48 of 117 in Computer Science. Deeper models, formal results and harder reasoning.

What you are seeing

The panel above shows an interactive animation of the idea. Every slider changes one quantity so you can see its effect on the outcome. The animation is drawn live from the model, so everything you see responds to the sliders. Press Space to pause, use Restart to replay from the start, and change the speed to slow down quick events.

In depth

Gradient descent minimises a function by repeatedly stepping in the direction of steepest descent: x ← x − η∇f(x), where η is the learning rate. Too small a step makes learning slow, too large a step overshoots and may diverge. It can get stuck in local minima or plateaus, though in high-dimensional problems this is less severe than feared. It is the workhorse for training machine-learning models, usually in stochastic or mini-batch versions with gradients computed by backpropagation.

Reading the relationship

The expression "x ← x − η∇f(x)" is a compact summary of the idea. It uses the symbols x, η, f; name what each one stands for before you start. Then use the sliders to see how changing one quantity changes the others.

A guided exploration

1. Run the animation once with the default settings and describe in words what happens. 2. Change one slider at a time and note which quantity responds most. 3. Push each slider to its extreme and decide whether the result still makes sense.

Try this

Try to explain why the pattern holds, not just what it is. Look for limiting cases and what happens at the edges.

Common pitfalls

Test small inputs by hand and then the largest ones. Off-by-one errors and forgetting how cost grows with input size are the most common traps.

How it connects

It builds on “A Single Artificial Neuron”, the topic before it in Computer Science. Once you are comfortable, continue with “k-Means Clustering”.

Why it matters

Computing ideas decide how fast, how safely and how well software can solve problems.