Which city is this home in?
Here are home listings. Each dot is one home: ● San Francisco or ● New York. Hide the city names, and all that's left is numbers.
The goal of machine learning is to find a rule — from the data alone — that guesses the city using only those numbers.
Start with elevation
Take just one recorded value per home — elevation — and line the dots up vertically.
See a pattern? San Francisco, city of hills, spreads its homes far up the slope, while New York's homes crowd the low ground. If a home sits very high, it's very likely San Francisco.
Add price, and the picture sharpens
Now spread the dots horizontally by price per m² to make a scatter plot.
Even at low elevations the two cities start to separate: on the same low ground, New York homes tend to be clearly more expensive. Every added dimension is more material for drawing boundaries.
Draw one line
Let's find the line that splits the data most cleanly in two. After sweeping every candidate, the algorithm's best cut is at m of elevation.
Above the line: almost entirely San Francisco. This one line is our first splitting rule.
Turn the line into a question, and you get a tree
That line reads as a question: “Is elevation ≥ m?” — a fork (node) that splits into yes / no.
But one line leaves plenty of misclassifications. The dots ringed in red are the homes still sorted into the wrong city.
Split again, inside each region
The trick is simple: repeat the same process inside each region. The lowlands split next on price; inside that, on elevation again… questions beget questions, and the tree grows.
Watch the red-ringed errors thin out as the boundaries multiply.
Training accuracy:
Keep adding splits, and the model can get the training data almost perfectly right — the regions have been carved fine enough to wrap around nearly every single dot.
But — can we actually trust this report card?
Meeting data it has never seen
Drop test homes — which the model has never seen — onto the same boundaries. Accuracy falls to .
The model memorized the training data's little accidents as if they were rules. This is overfitting — the most fundamental dilemma in machine learning.