Bias and Variance

This is the first topic in the Advanced Machine Learning series. Before learning Support Vector Machines, it helps to understand the two basic ways a model can go wrong: it can be too simple (bias) or too sensitive (variance).

1. Why models make mistakes

Every model you train will make some errors on new, unseen data. There are exactly two reasons this happens:

1. The model is too simple to capture the real pattern. This is called bias.
2. The model is too sensitive to the exact training data it saw. This is called variance.

Understanding these two error sources is the single most useful mental model in machine learning — it explains overfitting, underfitting, and why some models need more data than others.

2. Step 1: What is bias?

Bias is the error that comes from a model making overly simple assumptions about the data.

Imagine you are trying to predict house prices, and the true relationship between size and price is curved (bigger houses get disproportionately more expensive). If you fit a straight line to this curved pattern, the line can never bend to match the curve — no matter how much data you give it.

Price
  │           actual data (curved pattern)
  │        ●        ●
  │     ●        ●
  │  ●    straight line (the model)
  │________________________ Size

The straight line has high bias — it is systematically wrong in the same way, everywhere, because it is not flexible enough to represent the true pattern.

Memory trick: Bias is like wearing the wrong prescription glasses. No matter how carefully you look, everything is blurry in the same way.

3. Step 2: What is variance?

Variance is the error that comes from a model being too sensitive to the specific training data it happened to see.

Imagine the opposite extreme: instead of a straight line, you fit a wiggly curve that passes through every single training point exactly.

Price
  │        ●
  │      ╱   ╲    ●
  │    ●       ╲ ╱  ╲
  │  ╱           ●    ╲
  │________________________ Size
        (wiggly curve threads through every point)

This curve fits the training data perfectly. But if you trained it again on a slightly different sample of houses, you would get a completely different wiggly curve. The model’s predictions swing wildly depending on which exact data points it happened to see.

Memory trick: Variance is like a shaky camera. Every photo (every new training sample) comes out looking different, even of the same subject.

4. Step 3: Underfitting vs overfitting

These two error types map directly onto two familiar terms:

Underfitting          Just right           Overfitting
(high bias)         (low bias, low var)    (high variance)

  ────────             ╱‾╲___╱‾╲              ╱╲╱╲╱╲╱╲
straight line       gentle curve            wiggly mess
misses the         follows the real       memorizes noise
real pattern            pattern           in the training data

The tell-tale sign of each:

• Underfitting: training accuracy is low, test accuracy is also low. The model never learned the pattern.
• Overfitting: training accuracy is very high, test accuracy is much lower. The model memorized the training data instead of learning the general pattern.

5. Step 4: A concrete Python example

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

# Generate a curved true relationship with noise
rng = np.random.RandomState(42)
X = np.sort(rng.uniform(0, 1, 60)).reshape(-1, 1)
y = np.sin(2 * np.pi * X).ravel() + rng.normal(0, 0.2, X.shape[0])

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)

# Try three levels of model complexity
for degree in [1, 4, 15]:
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
    model.fit(X_train, y_train)

    train_error = mean_squared_error(y_train, model.predict(X_train))
    test_error = mean_squared_error(y_test, model.predict(X_test))

    print(f"Degree {degree:2d}: train MSE={train_error:.3f}  test MSE={test_error:.3f}")

Typical output pattern:

Degree  1: train MSE=0.229  test MSE=0.251   ← underfitting: both errors high
Degree  4: train MSE=0.035  test MSE=0.031   ← just right: both errors low
Degree 15: train MSE=0.024  test MSE=0.168   ← overfitting: train low, test much higher

Notice the degree-15 model has almost zero training error, but a much higher test error — a classic overfitting signature.

6. Step 5: The bias-variance tradeoff

Here is the important part: you cannot reduce bias and variance at the same time by adjusting model complexity alone.

Error
  │  ╲                              ╱
  │   ╲  bias (decreases with      ╱  variance (increases with
  │    ╲  more complexity)        ╱   more complexity)
  │     ╲                        ╱
  │      ╲____total error_______╱
  │            ╲       ╱
  │             ╲_____╱  ← sweet spot (lowest total error)
  │________________________________ Model complexity →
   simple                          complex

• As a model becomes more complex (more features, deeper trees, higher polynomial degree), bias goes down but variance goes up.
• As a model becomes simpler, bias goes up but variance goes down.

The goal of model selection is not to eliminate bias or variance — it is to find the complexity level where their combined effect on test error is smallest.

7. Step 6: How to fix each problem

# Example: controlling variance with regularization strength
from sklearn.linear_model import Ridge

model_low_variance = Ridge(alpha=10.0)   # strong regularization → less overfitting
model_high_variance = Ridge(alpha=0.001)  # weak regularization → more overfitting risk

8. Why this matters before learning SVM

Support Vector Machines have their own version of this exact tradeoff, controlled by a parameter called C (and, for non-linear kernels, a parameter called gamma):

• A very strict SVM (high C) tries hard to classify every training point correctly → low bias, high variance → risk of overfitting.
• A very relaxed SVM (low C) allows more misclassifications on training data → high bias, low variance → risk of underfitting.

Keep this mental model in mind. Every time you see the letter C in the next few topics, think back to this section: it is the exact same bias-variance tradeoff, just wearing an SVM costume.

9. Practice questions

1. A model has 2% training error and 25% test error. Is this underfitting or overfitting? What would you do about it?
2. A model has 30% training error and 31% test error. Is this underfitting or overfitting? What would you do about it?
3. If you increase the degree of a polynomial regression model, does bias go up or down? What about variance?
4. Why can’t you simply make a model as complex as possible to minimize all error?

Answers:

1. Overfitting (huge gap between train and test error). Fix: simplify the model, add regularization, or get more data.
2. Underfitting (both errors are high and close together). Fix: use a more flexible/complex model.
3. Bias goes down (the model can fit more complex patterns); variance goes up (the model becomes more sensitive to the specific training data).
4. Because past a certain point, added complexity increases variance faster than it decreases bias, so total error on new data starts rising even though training error keeps falling.

10. Summary

• Bias = error from a model being too simple to capture the real pattern → underfitting.
• Variance = error from a model being too sensitive to the exact training data → overfitting.
• You can spot each by comparing training error to test error.
• Model complexity trades one for the other — the goal is the sweet spot with the lowest combined error.
• SVM’s C parameter (covered later in this series) is a direct expression of this same tradeoff.

Next topic: Hyperplanes and Linear Classification — the first building block of Support Vector Machines.