Training a Linear Model

Lesson notes

Overview

Gradient descent adjusts slope and intercept until the line fits the data. This topic is rated intermediate and is number 50 of 117 in Computer Science. Builds on the basics with relationships, methods and trade-offs.

What you are seeing

The panel above shows an interactive animation of the idea. Every slider changes one quantity so you can see its effect on the outcome. The animation is drawn live from the model, so everything you see responds to the sliders. Press Space to pause, use Restart to replay from the start, and change the speed to slow down quick events.

In depth

To fit a line ŷ = mx + b to data, define a loss such as the mean squared error, MSE = (1/n)Σ(ŷ − y)², and use gradient descent to adjust the slope and intercept in the direction that reduces it. As training proceeds the line moves towards the best fit. The process is the same as for large neural networks, only with two parameters. Plotting the loss against iterations shows whether the learning rate is suitable.

Reading the relationship

The expression "MSE = ⅟n Σ(ŷ − y)²" is a compact summary of the idea. It uses the symbols M, S, E, n, Σ, y; name what each one stands for before you start. Notice that it contains a squared term, so doubling that quantity changes the result by a factor of four, and a sum, so many small contributions are added together. Then use the sliders to see how changing one quantity changes the others.

A guided exploration

1. Run the animation once with the default settings and describe in words what happens. 2. Change one slider at a time and note which quantity responds most. 3. Push each slider to its extreme and decide whether the result still makes sense.

Try this

Predict the outcome first, then test extreme slider values to find where the pattern changes or breaks.

Common pitfalls

Test small inputs by hand and then the largest ones. Off-by-one errors and forgetting how cost grows with input size are the most common traps.

How it connects

It builds on “k-Means Clustering”, the topic before it in Computer Science. Once you are comfortable, continue with “Finite State Machines”.

Why it matters

Computing ideas decide how fast, how safely and how well software can solve problems.