SVM Part 2: Maximal Margin Classifier

This is Part 2 of the SVM mini-series. It assumes you’ve read Part 1: Hyperplanes and Linear Classification.

1. The problem left over from Part 1

In Part 1 we saw that for separable data, there isn’t just one valid hyperplane — there can be many:

      ●  ●  ●
   ╲    ╲     ╲          ← three different valid separating lines
    ╲    ╲     ╲
      ■  ■  ■

All three lines classify the training data perfectly. So which is best?

2. Step 1: Introducing the margin

Look closely at the three lines above. Some lines pass very close to the data points; others leave more breathing room on both sides.

The margin is the width of empty space around the hyperplane before it touches the nearest data point of either class:

        ●   ●    ●
      ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄     ← margin boundary (upper edge)
      ─────────────────   ← the hyperplane itself
      ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄     ← margin boundary (lower edge)
        ■   ■    ■

      ←──── margin width ────→

The Maximal Margin Classifier is the rule: choose the hyperplane that makes this margin as wide as possible.

Memory trick: Think of building a road between two neighboring towns. You don’t build the road hugging one town’s fence — you build it exactly in the middle, giving maximum safety distance to both sides.

3. Step 2: Refresher — hyperplane as a dot product

Before measuring distance to the hyperplane, it helps to rewrite the hyperplane equation from Part 1 using vector notation, since this is what the distance formula will need.

Recall the hyperplane equation:

W0 + W1·x1 + W2·x2 + ... + Wd·xd = 0

This is exactly the dot product of a weight vector W = [W1, W2, ..., Wd] and a feature vector X = [x1, x2, ..., xd], plus the bias W0:

W · X + W0 = 0

import numpy as np

W = np.array([-1, 3])     # weight vector
X = np.array([1, 1])      # feature vector
W0 = 2

value = np.dot(W, X) + W0
print(value)   # same computation as Part 1: 2 - 1 + 3 = 4

This dot-product form is what lets the same distance formula work in 2D, 3D, or 100D without changing shape.

4. Step 3: Refresher — distance from a point to a hyperplane

The (perpendicular) distance from any point X to the hyperplane W·X + W0 = 0 is:

distance = | W·X + W0 |  /  ||W||

where ||W|| is the length (magnitude) of the weight vector:

||W|| = √(W1² + W2² + ... + Wd²)

import numpy as np

def distance_to_hyperplane(X, W, W0):
    numerator = abs(np.dot(W, X) + W0)
    denominator = np.linalg.norm(W)   # this computes ||W||
    return numerator / denominator

W = np.array([-1, 3])
W0 = 2
X = np.array([1, 1])

print(distance_to_hyperplane(X, W, W0))

This is the exact formula the Maximal Margin Classifier tries to maximize — but specifically for the closest points of each class.

5. Step 4: A more concise notation via rescaling

The distance formula above works, but the ||W|| in the denominator makes every formula involving it a little clumsy. There’s a neat simplification worth knowing, because you’ll see it again when the margin is formalized in Step 6.

The trick: a hyperplane equation can be multiplied (or divided) by any non-zero constant and still describe the exact same line. For example, 2x1 + 3x2 + 4 = 0 and x1 + 1.5x2 + 2 = 0 are the same line — one is just the other multiplied by 2.

So, take the original equation and divide every coefficient — including W0 — by ||W|| = √(W1² + ... + Wd²):

W0' = W0 / ||W||     W1' = W1 / ||W||     ...     Wd' = Wd / ||W||

After this rescaling, the new coefficients satisfy W1'² + ... + Wd'² = 1 by construction — the denominator has been “absorbed” into the coefficients themselves. Plugging the rescaled coefficients back into the distance formula, the ||W'|| in the denominator is now just 1, so it disappears:

distance = | W0' + W1'·x1 + ... + Wd'·xd |

Now go one step further: define an augmented data vector Y that has a 1 stitched onto the front of the feature vector, and an augmented weight vector W that includes W0' as its first entry:

Y = [1, x1, x2, ..., xd]
W = [W0', W1', ..., Wd']

With this notation, the distance from any point to the hyperplane is simply:

distance = W · Y

No division, no separate bias term to track — just a single dot product.

import numpy as np

# Original hyperplane: 4 + 2*x1 + 3*x2 = 0
W0, W1, W2 = 4, 2, 3
norm = np.sqrt(W1**2 + W2**2)

# Rescale so that W1'^2 + W2'^2 = 1
W0r, W1r, W2r = W0 / norm, W1 / norm, W2 / norm
print("Check W1'^2 + W2'^2 =", W1r**2 + W2r**2)   # should be 1.0

# Augmented vectors
W = np.array([W0r, W1r, W2r])   # [W0', W1', W2']

def distance_concise(x1, x2):
    Y = np.array([1, x1, x2])   # augmented data point
    return abs(np.dot(W, Y))

# Compare against the original (unrescaled) formula
def distance_standard(x1, x2):
    return abs(W0 + W1 * x1 + W2 * x2) / norm

print("Standard formula:", distance_standard(5, -1))
print("Concise formula: ", distance_concise(5, -1))

Both give the same answer — the concise version is just a cleaner way of writing the same computation. This W · Y notation generalizes to any number of dimensions without changing shape, which is exactly why it becomes the standard way the margin is written once we formalize the optimization in the next step.

6. Step 5: The maximal margin hyperplane

Now combine the pieces. Maximizing the margin means: find the W and W0 that make the distance from the hyperplane to the nearest point of either class as large as possible.

Mathematically, this optimization has two important constraints:

Constraint 1 — standardize the weights. The sum of squares of all coefficients must equal 1:

        d
        ∑ (Wi²) = 1
       i=1

This constraint exists because a hyperplane equation can be scaled by any constant and still represent the same line (e.g. 2x + 2y = 0 is the same line as x + y = 0). Standardizing removes this ambiguity so “distance” is measured consistently.

Constraint 2 — every point must sit at least a margin’s distance away, on its correct side. Using the concise W · Y notation from Step 4, and labeling each class as li = +1 or li = -1, this constraint is written as:

li · (W · Yi)  ≥  M      for every training point i

where:

li = the label of point i (+1 or -1)
W = the rescaled, augmented weight vector [W0', W1', ..., Wd']
Yi = the augmented data point [1, x1, x2, ..., xd]
M = the margin width — the quantity we are trying to maximize

Why multiplying by the label works

Think of the hyperplane as splitting space into a +L side and a -L side:

Figure A: red dots below the "-L" hyperplane        Figure B: blue dots above the "+L" hyperplane

        ┄┄┄┄┄┄┄┄┄┄┄┄  +L                                ●  ●  ●   (li = +1)
             ●●●        (blue, li = +1)               ┄┄┄┄┄┄┄┄┄┄┄┄  +L
      ─────────────── hyperplane                    ─────────────── hyperplane
             ■■■        (red,  li = -1)               ┄┄┄┄┄┄┄┄┄┄┄┄  -L
        ┄┄┄┄┄┄┄┄┄┄┄┄  -L                                ■  ■  ■   (li = -1)

For a red point (li = -1), the raw dot product W · Yi comes out negative (it’s on the -L side). Multiplying by li = -1 flips the sign, giving a positive result.
For a blue point (li = +1), the raw dot product W · Yi is already positive. Multiplying by li = +1 leaves it positive.

Either way, li · (W · Yi) is always positive when the point is correctly classified, and its size tells you how far past the margin the point sits. Requiring this value to be ≥ M for every point is exactly the constraint “every point must be correctly classified and at least a margin’s distance from the hyperplane.”

Worked 2D example

from sklearn.svm import SVC
import numpy as np

# A simple, cleanly separable 2D dataset, using +1/-1 labels to match the li notation
X = np.array([
    [3, 3], [4, 3], [4, 4],     # class li=+1 (upper right)
    [1, 1], [1, 2], [2, 1],     # class li=-1 (lower left)
])
y = np.array([1, 1, 1, -1, -1, -1])

# A hard-margin SVM: extremely high C forces near-zero tolerance
# for misclassification, which approximates a Maximal Margin Classifier
model = SVC(kernel='linear', C=1e6)
model.fit(X, y)

W1, W2 = model.coef_[0]
W0 = model.intercept_[0]
norm = np.sqrt(W1**2 + W2**2)

# Rescale so W1'^2 + W2'^2 = 1 (Step 4's trick)
W = np.array([W0 / norm, W1 / norm, W2 / norm])
M = 1 / norm   # the resulting margin width

print("Rescaled W:", W)
print("Margin M:", M)
print()

for xi, li in zip(X, y):
    Yi = np.array([1, xi[0], xi[1]])
    value = li * np.dot(W, Yi)
    print(f"point={xi}  li={li:+d}  li*(W.Yi)={value:.4f}  {'← equals M (support vector)' if abs(value - M) < 1e-6 else ''}")

print("\nsklearn's own support vectors:", model.support_vectors_)

Running this shows every point satisfies li·(W·Yi) ≥ M, and the three points that hit exactly M are the support vectors from Step 6 — confirming that support vectors are precisely the points where this constraint is “tight” (an equality, not a strict inequality).

7. Step 6: Why they’re called “support vectors”

Intuition first: distance from the boundary = difficulty of the decision

Before defining support vectors formally, it helps to notice something about how confident you’d be classifying a point, just based on how far it sits from the separator.

Imagine classifying animals as “small” or “large” based on weight, and your SVM finds the discriminating boundary at 50 kg. If you’re handed an animal that weighs 200 kg, you don’t need the model at all — it’s obviously “large,” no matter where exactly the boundary sits. The same is true for a 2 kg animal — obviously “small.” These far-away points are “no-brainers.”

Now you’re handed an animal that weighs 49 kg. This is where it gets tricky — it’s right on the edge, and small shifts in the boundary could flip its classification. Points like this, sitting close to the separator, are the ones that actually determine where the boundary should go.

Far from boundary          Close to boundary          Far from boundary
(easy: obviously  ●)      (hard: could go       )      (easy: obviously  ■)
                            either way, ● or ■)

  ●───●───●   ...   ●   ┄┄┄┄┄┄┄ margin ┄┄┄┄┄┄┄   ■   ...   ■───■───■

The farther a point is from the separator, the easier and more obvious its classification is. The closer it is, the harder — and it’s precisely these hard, boundary-hugging points that the model needs to pay attention to.

The formal definition

Only a handful of points — model.support_vectors_ — actually determine the margin. These are the points closest to the hyperplane on each side.

Class 1 (upper right):  ●  ●  [●]  ← closest point "supports" the margin
                                       ↕ margin
Class 0 (lower left):  [■]  ■  ■   ← closest point "supports" the margin

Proof: retraining on only the support vectors gives the same hyperplane

This isn’t just a claim — you can verify it directly. Train on the full dataset, extract only the support vectors, then retrain using just those points and compare:

import numpy as np
from sklearn.svm import SVC

X = np.array([
    [3, 3], [4, 3], [4, 4], [5, 5], [6, 3],    # class +1: some far, some close to the boundary
    [1, 1], [1, 2], [2, 1], [0, 0], [-1, 2],   # class -1: some far, some close to the boundary
])
y = np.array([1, 1, 1, 1, 1, -1, -1, -1, -1, -1])

# Train on the full dataset
full_model = SVC(kernel='linear', C=1e6)
full_model.fit(X, y)
print("Full dataset  -> W:", full_model.coef_, " W0:", full_model.intercept_)

# Keep only the support vectors, throw away every other point
sv_idx = full_model.support_
X_sv, y_sv = X[sv_idx], y[sv_idx]
print("Support vectors kept:", X_sv.tolist())

# Retrain using only those points
sv_model = SVC(kernel='linear', C=1e6)
sv_model.fit(X_sv, y_sv)
print("SV-only dataset -> W:", sv_model.coef_, " W0:", sv_model.intercept_)

Both models produce essentially identical weights — the seven points that weren’t support vectors were completely redundant for finding the separator. Only the points closest to the boundary — the tricky, ambiguous ones from the intuition above — actually determine where the hyperplane goes.

This is why the algorithm is called a Support Vector Machine — the boundary is “supported” by a small subset of the data, not by all of it.

8. Step 7: Why the maximal margin classifier is fragile

The Maximal Margin Classifier has a serious weakness: it requires the two classes to be perfectly separable, and it is extremely sensitive to the exact position of the support vectors.

Before adding one noisy point:          After adding one noisy point:

      ●  ●  ●                                 ●  ●  ●
   ╲                                        ╲
    ╲   wide margin                          ╲    (margin collapses,
     ╲                                         ╲    or classifier fails
      ■  ■  ■                             ■  ■ ●■   entirely — no line
                                                       can separate them)

A single mislabeled or unusual training point — a single “outlier” — can:

Force the margin to shrink dramatically, or
Make it impossible to find any separating hyperplane at all.

This is exactly the high-variance problem described in Bias and Variance: the Maximal Margin Classifier fits its training data extremely tightly and can perform poorly when new, unseen data doesn’t look exactly like the training set.

Real-world data is almost never perfectly separable — there are always some ambiguous or noisy points. The Maximal Margin Classifier, as described in this topic, cannot handle that. The next topic fixes this.

9. Practice questions

What does “margin” mean in the context of SVM?
Why is the weight vector standardized (∑Wi² = 1) before maximizing the margin?
If you remove a training point that is not a support vector and retrain the model, does the hyperplane change? Why or why not?
What is the main weakness of the Maximal Margin Classifier on real-world data?
After rescaling W so that W1'² + ... + Wd'² = 1, why does the distance formula simplify to W · Y instead of |W·X + W0| / ||W||?
Why does multiplying by the label li (+1 or -1) turn W · Yi into a value that is always positive for a correctly classified point?
Why do points that are far from the separating hyperplane matter less than points close to it, when it comes to determining where the hyperplane goes?

Answers:

The margin is the width of the empty space between the hyperplane and the closest training points of each class. SVM chooses the hyperplane that makes this width as large as possible.
Because a hyperplane equation can be scaled by any constant and still describe the same line. Without standardizing, “distance” would be ambiguous. Standardizing fixes this so the optimization is well-defined.
No — non-support-vector points do not affect the hyperplane at all. Only the closest points (support vectors) determine its position.
It requires the data to be perfectly linearly separable and is extremely sensitive to outliers — a single noisy point can shrink the margin drastically or make separation impossible.
Because after rescaling, ||W'|| = 1 by construction, so the denominator in the distance formula disappears. Folding W0' into W as its first entry and prepending a 1 to X (giving Y) lets the whole numerator be written as a single dot product W · Y.
For a point on the -L side (li = -1), the raw dot product W · Yi is negative; multiplying by -1 flips it positive. For a point on the +L side (li = +1), the dot product is already positive, and multiplying by +1 leaves it unchanged. Either way, correct classification always produces a positive li · (W · Yi), which is why the constraint li · (W · Yi) ≥ M can require this value to be at least the margin M for every point.
A point far from the hyperplane is classified with obvious confidence — no matter small shifts to the boundary, it stays on the same side. A point close to the hyperplane is ambiguous — a small shift in the boundary’s position could flip its predicted class. Since the optimization is choosing the boundary’s exact position, only the close, ambiguous points (the support vectors) actually constrain that choice; far-away points impose no real constraint and could be removed without changing the result.

10. Summary

The margin is the empty gap between the hyperplane and the nearest points of each class.
The Maximal Margin Classifier picks the hyperplane that makes this margin as wide as possible.
The hyperplane is measured using the dot product W·X + W0, and distance to it uses |W·X + W0| / ||W||.
Rescaling W so ∑Wi² = 1 and augmenting the point with a leading 1 simplifies this to a single dot product, W · Y.
Only the closest points — the support vectors — determine where the boundary goes.
This approach is fragile: it requires perfectly separable data and is very sensitive to outliers.

Next topic: Soft Margin Classifier and the Cost Parameter — how SVM handles real, noisy, imperfectly separable data.