SVM Part 4: Kernels and the Kernel Trick

This is Part 4, the final topic in the SVM mini-series. It assumes you’ve read Part 3: Soft Margin Classifier and Cost.

0. Kernels are bigger than just SVM

Before diving in, it’s worth knowing that kernel methods are not an SVM-only trick. They’re a general set of principles that show up across many algorithms — kernel ridge regression, kernel PCA, Gaussian processes, and more all reuse the same idea. SVM is simply the most common place you’ll encounter kernels first.

The key conceptual insight — one that makes kernels feel almost like magic — is this: the underlying algorithm never changes. Whether you’re doing SVM or regression, the core method stays a purely linear technique the entire time. A kernel is not a modification to the algorithm itself; it’s a substitution slotted in on top of it — a “topping,” so to speak — that lets a linear method behave as if it were fitting a highly non-linear boundary.

This is exactly why kernels are so powerful: you keep all the simplicity, speed, and mathematical elegance of linear methods, while getting the flexibility of arbitrarily non-linear models. The rest of this topic walks through how that substitution works for SVM specifically.

1. The problem left over from Part 3

Every SVM variant so far — Maximal Margin, Soft Margin — draws a straight line (or flat hyperplane). But some data simply cannot be separated by any straight line:

        ●  ●  ●
      ●           ●
    ●    ■  ■  ■    ●
      ●           ●
        ●  ●  ●

The ■ points are surrounded on all sides by ● points. No matter how you rotate or shift a straight line, you cannot separate the inner cluster from the outer ring. We need a genuinely new idea, not just a tolerance adjustment.

2. Step 1: Map the data to a new space

The key insight: data that is not linearly separable in its original form can become linearly separable if you transform it into a different space.

Call the original space the attribute space (X, Y), and the transformed space the feature space (X', Y').

Why squaring turns a circle/ellipse into a straight line

If you suspect the boundary between two classes looks like a circle or an ellipse, recall the equation of an ellipse from analytic geometry:

X² / a  +  Y² / b  =  C

(when a = b, this is a circle). Now define new coordinates X' = X² and Y' = Y², and substitute directly into the equation:

X' / a  +  Y' / b  =  C

This is exactly the equation of a straight line in (X', Y') coordinates! A quadratic (curved) boundary in the original attribute space becomes a perfectly linear boundary once you plot X² and Y² instead of X and Y.

import numpy as np

a, b, c = 4, 9, 1  # example ellipse: X^2/4 + Y^2/9 = 1

# Sample a few points that sit exactly on this ellipse
thetas = np.linspace(0, 2*np.pi, 8, endpoint=False)
X = np.sqrt(a*c) * np.cos(thetas)
Y = np.sqrt(b*c) * np.sin(thetas)

for x, y in zip(X, Y):
    on_ellipse = abs(x**2/a + y**2/b - c) < 1e-9
    x_prime, y_prime = x**2, y**2
    on_line = abs(x_prime/a + y_prime/b - c) < 1e-9
    print(f"(x,y)=({x:.2f},{y:.2f})  on ellipse: {on_ellipse}   ->   (x',y')=({x_prime:.2f},{y_prime:.2f})  on line: {on_line}")

Every point that satisfies the ellipse equation in (X, Y) also satisfies the linear equation in (X', Y') — confirmed above.

Applying this to a real example

Example: classify emails into spam/ham using two attributes — word_freq_office (X) and word_freq_lottery (Y). Suppose the data is arranged in a circle: spam emails cluster in the center, ham emails surround them.

Following the same idea, here’s a slightly more general transformation (squaring the distance from some center point (a, b) rather than the origin):

X' = (X - a)²
Y' = (Y - b)²

import numpy as np

# Simulate a circular pattern: inner cluster (spam), outer ring (ham)
rng = np.random.RandomState(0)

# Inner cluster near the center
inner_r = rng.uniform(0, 1, 40)
inner_theta = rng.uniform(0, 2*np.pi, 40)
X_inner = inner_r * np.cos(inner_theta)
Y_inner = inner_r * np.sin(inner_theta)

# Outer ring farther from the center
outer_r = rng.uniform(3, 4, 40)
outer_theta = rng.uniform(0, 2*np.pi, 40)
X_outer = outer_r * np.cos(outer_theta)
Y_outer = outer_r * np.sin(outer_theta)

# Transform: square the distance from the center
inner_transformed = X_inner**2 + Y_inner**2
outer_transformed = X_outer**2 + Y_outer**2

print("Inner cluster transformed range:", inner_transformed.min(), "to", inner_transformed.max())
print("Outer ring transformed range:  ", outer_transformed.min(), "to", outer_transformed.max())

After the transformation, the inner cluster’s values and the outer ring’s values fall into two non-overlapping ranges — meaning a single threshold (a linear boundary in the new 1D feature) now separates them perfectly.

Mapping the separator back to the original space

Once you run a linear method (SVM, logistic regression, or plain linear regression — any of them work, since the transformation did all the hard work) in the transformed (X', Y') space, it will hand you back a straight-line equation like:

X'/a' + Y'/b' = C'

To interpret this boundary in terms of the original attributes, simply substitute X' = X² and Y' = Y² back in:

X²/a' + Y²/b' = C'

This is once again the equation of an ellipse (or circle) — but now it’s expressed back in the original (X, Y) attribute space, giving you the actual curved decision boundary you wanted all along. You never had to search for a circular/elliptical separator directly; you found an easy linear one in the transformed space, then substituted back to recover the non-linear one.

3. Step 2: Feature transformation gets expensive fast

You usually don’t know the exact functional form

The circle/ellipse example in Step 1 was convenient because we guessed the exact shape of the boundary in advance, and that guess happened to be correct. In practice, you rarely know the precise functional form of the separator — you can often tell from the data that it’s not linear, and that it’s probably not wildly complicated (not some jagged, arbitrary shape), but you usually can’t be sure whether it’s exactly a circle, an ellipse, or some other quadratic curve.

To handle this uncertainty, write out the most general quadratic equation possible, rather than assuming the specific X²/a + Y²/b = C form:

A·X² + B·Y² + C·XY + D·X + E·Y + F = 0

This general form includes every possible quadratic curve — circles, ellipses, parabolas, hyperbolas, and skewed/rotated versions of all of them — because it keeps the cross-term XY and the linear terms X, Y, which the earlier circle-specific example conveniently assumed were zero.

Mapping 2 attributes to 6 features

To make this general quadratic become a linear equation, you now need six transformed features instead of two — one for every term in the equation above:

X², Y², XY, X, Y, 1     (6 features, extracted from the original 2)

Every point (X, Y) gets mapped to a point in this 6-dimensional feature space. The general quadratic equation above is now exactly a linear equation in these 6 coordinates — confirmed below with a skewed quadratic curve that a simple X²/a + Y²/b = C guess would have missed entirely:

import numpy as np

# A general (skewed) quadratic: A*x^2 + B*y^2 + C*x*y + D*x + E*y + F = 0
A, B, C, D, E, F = 1, 2, 1.5, -3, 2, -5

def on_curve(x, y):
    return abs(A*x**2 + B*y**2 + C*x*y + D*x + E*y + F) < 1e-9

def on_hyperplane_6d(x_sq, y_sq, xy, x, y):
    return abs(A*x_sq + B*y_sq + C*xy + D*x + E*y + F) < 1e-9

# Find a few points that lie exactly on this curve, then check the 6D features
for x in np.linspace(-1.1, -0.5, 3):
    b_, c_ = B, C*x + E
    disc = c_**2 - 4*b_*(A*x**2 + D*x + F)
    y = (-c_ + np.sqrt(disc)) / (2*b_)
    x_sq, y_sq, xy = x**2, y**2, x*y
    print(f"(x,y)=({x:.2f},{y:.2f})  on_curve={on_curve(x,y)}  ->  "
          f"6D features satisfy linear eq: {on_hyperplane_6d(x_sq, y_sq, xy, x, y)}")

Once you’re in this 6D feature space, run any linear method (SVM, logistic regression) on the transformed points, get back a linear separator, then substitute X², Y², XY back in to recover the actual quadratic curve in the original 2D attribute space — exactly the same “transform, solve linearly, map back” workflow from Step 1, just with more features.

The blowup problem

This already reveals the core issue: transforming just 2 attributes into a general quadratic feature space required 6 dimensions — a 3x blowup. And it gets much worse as the number of original attributes grows.

For example, with 4 original attributes and a degree-2 polynomial transformation, you end up with 15 new features (all pairwise products, squares, and originals):

Original attributes:  x1, x2, x3, x4                      (4 features)

Degree-2 transformed:
  x1, x2, x3, x4,                                          (4)
  x1², x2², x3², x4²,                                       (4)
  x1x2, x1x3, x1x4, x2x3, x2x4, x3x4                        (6)
  ────────────────────────────────────────
  Total: 15 features

from sklearn.preprocessing import PolynomialFeatures
import numpy as np

X = np.array([[1.0, 2.0, 3.0, 4.0]])
poly = PolynomialFeatures(degree=2)  # include_bias=True by default, adds the constant term
X_transformed = poly.fit_transform(X)

print("Original number of features:", X.shape[1])
print("Transformed number of features:", X_transformed.shape[1])

With many original attributes and a higher degree, this transformation becomes computationally very expensive — sometimes practically impossible (imagine 1,000 attributes transformed to millions of features). This is the problem the kernel trick solves.

4. Step 3: The kernel trick — never actually transform the data

Pre-reading: the inner product (dot product)

Before we get to the trick itself, recall what an inner product (also called a dot product) of two vectors is: multiply the vectors element-wise, then sum the results. It always returns a single scalar number.

For example, if A = [2, 3, 5] and B = [1, 0, 4]:

Aᵗ B = A · B = (2)(1) + (3)(0) + (5)(4) = 2 + 0 + 20 = 22

import numpy as np

A = np.array([2, 3, 5])
B = np.array([1, 0, 4])
print(A @ B)  # 22

In any dataset with n attributes, each row (observation) is a vector of length n. So the dot product of any two rows is a single scalar — this is exactly what the SVM’s learning algorithm uses over and over again.

Here is the key mathematical fact that makes the kernel trick possible:

To fit an SVM, the learning algorithm never needs the individual data points on their own. It only ever needs the inner (dot) products between pairs of points: Xi · Xj.

A kernel function computes what the inner product would be in the transformed feature space — without ever actually computing the transformation.

                    ┌─────────────┐
   Original         │             │      Result: as if the data
   attributes  ───▶ │ Kernel      │ ───▶ had been transformed,
   (Xi, Xj)         │ "black box" │      compared, and separated
                    │             │      by a linear boundary
                    └─────────────┘

This means you get all the benefit of transforming into a huge, complex feature space, while paying only the computational cost of the original, much smaller feature space. This is why it’s called a “trick” — it bypasses the expensive step entirely.

Naming the transformation: the feature vector Φ

We give the transformation a name: Φ (phi). If your original attributes are, say, age and salary, you might construct a new feature vector:

Φ(X) = [ age², log(salary), age × log(salary), ... ]

As far as the SVM algorithm is concerned, Φ(X) is the data — it has no idea (and doesn’t care) that these features were derived from something else. All the SVM ever asks for is the dot product between two such feature vectors: Φ(Xi) · Φ(Xj).

What makes a function a valid kernel

Suppose we define some scalar-valued function of two vectors, e.g. the popular RBF (Gaussian) kernel:

K(Xi, Xj) = exp( -γ · ‖Xi − Xj‖² )

This just takes the Euclidean distance between two points, squares it, scales it by -γ, and exponentiates — clearly cheap to compute directly from the original attributes.

K is a valid kernel function if we can show that there exists some feature transformation Φ such that, for every pair of vectors:

K(Xi, Xj) = Φ(Xi) · Φ(Xj)

Critically, we don’t need to know the explicit form of Φ — only that it exists. For the RBF kernel above, the corresponding Φ turns out to map into an infinite-dimensional feature space, yet K itself takes only microseconds to evaluate.

Why this makes the two scenarios identical

Compare two ways of running SVM:

Explicit transformation: compute Φ(X) for every data point, then run SVM in that (possibly huge or infinite-dimensional) feature space.
Kernel trick: whenever the SVM algorithm asks for Φ(Xi) · Φ(Xj), secretly hand back K(Xi, Xj) instead — never computing Φ at all.

Since K(Xi, Xj) = Φ(Xi) · Φ(Xj) by definition of a valid kernel, the SVM receives exactly the same numbers in both scenarios. The resulting decision boundary is therefore identical — the only difference is that scenario 2 is computationally feasible even when Φ would blow up to millions (or infinite) dimensions.

Practically, this means the kernel argument you pass to SVC(...) doesn’t change the algorithm at all — it only changes which function is used to compute the dot product whenever the algorithm needs one.

from sklearn.svm import SVC
from sklearn.datasets import make_circles
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Generate the exact "ring" pattern that a straight line cannot separate
X, y = make_circles(n_samples=200, factor=0.4, noise=0.08, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

linear_model = SVC(kernel='linear')
linear_model.fit(X_train, y_train)
linear_acc = accuracy_score(y_test, linear_model.predict(X_test))

rbf_model = SVC(kernel='rbf', gamma='scale')
rbf_model.fit(X_train, y_train)
rbf_acc = accuracy_score(y_test, rbf_model.predict(X_test))

print(f"Linear kernel accuracy: {linear_acc:.2f}")
print(f"RBF kernel accuracy:    {rbf_acc:.2f}")

You should see the linear kernel perform close to random guessing (around 50%) on this ring-shaped data, while the RBF kernel separates it almost perfectly.

5. Step 4: The three common kernel types

Kernel	What it does	When to use
Linear	No transformation — the plain hyperplane from Parts 1–3	Data is already roughly linearly separable, or you have very high-dimensional data (e.g. text)
Polynomial	Creates curved, polynomial-shaped boundaries	Moderate non-linearity
RBF (Radial Basis Function)	Creates highly flexible, even circular/elliptical boundaries	Most common default for non-linear data

from sklearn.svm import SVC

model_linear = SVC(kernel='linear')
model_poly   = SVC(kernel='poly', degree=3)
model_rbf    = SVC(kernel='rbf', gamma='scale')

Linear kernel:              Polynomial kernel:          RBF kernel:
   ●●●   |   ■■■              ●●●  ╱‾╲  ■■■               ●●● ( ■■■ ) ●●●
  straight boundary        curved polynomial          flexible, even circular
                                boundary                    boundary

6. Step 5: The kernel parameter (gamma / sigma)

For non-linear kernels like RBF, a parameter called gamma (sometimes written sigma) controls how much non-linearity (how “wiggly”) the boundary can become.

Low gamma                Medium gamma              High gamma
(smooth, simple           (moderate                (highly wiggly,
 boundary)                 flexibility)              memorizes noise)

   ●●●●●                    ●●●○●                    ●○●●○
   ┄┄┄┄┄                    ┄┄╱‾╲┄                   ┄╱╲╱╲╱╲
   ■■■■■                    ■■●■■                    ■●■■●

High gamma → each training point’s influence reaches only a very short distance, creating a highly localized, wiggly boundary. This can perfectly separate training data but is prone to overfitting.
Low gamma → each point’s influence reaches farther, creating a smoother, simpler boundary. This can underfit if the true pattern needs more flexibility.

This is, once again, the exact same bias-variance tradeoff introduced in Bias and Variance — now controlled by gamma instead of (or alongside) C.

from sklearn.svm import SVC
from sklearn.datasets import make_circles
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = make_circles(n_samples=200, factor=0.4, noise=0.15, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

for gamma in [0.01, 1, 100]:
    model = SVC(kernel='rbf', C=1.0, gamma=gamma)
    model.fit(X_train, y_train)
    train_acc = accuracy_score(y_train, model.predict(X_train))
    test_acc = accuracy_score(y_test, model.predict(X_test))
    print(f"gamma={gamma:6.2f}  train_acc={train_acc:.3f}  test_acc={test_acc:.3f}")

7. Step 6: Choosing a kernel in practice

Thumb rule: complexity vs. overfitting

There is no formula that tells you the “correct” kernel just by looking at raw data with many features — kernel selection is largely trial and error. But one general rule always holds:

As the kernel becomes more complex, the model becomes more complex — and so does the danger of overfitting.

A kernel that is too simple (e.g. linear, on data with real curvature) will underfit — it isn’t flexible enough to capture the true pattern.
A kernel that is too complex (e.g. high-degree polynomial, or RBF with very high gamma) can fit the training data almost perfectly, but overfits and generalizes poorly.

This is the exact same bias-variance tradeoff from Bias and Variance — the kernel is just one more tool (alongside C and gamma) that lets you exploit non-linearity in your data while still technically running a linear method underneath.

Thumb rule: judge the boundary visually

When you have just a couple of attributes, plot the two classes on a scatter plot and visually judge how clean the separation looks:

Reds and Blues clearly bunched apart, maybe with a few outliers (which is fine — chasing every outlier just leads to overfitting) → a linear kernel will do. This is exactly what happens with email spam detection.
Reds and Blues intermingled, but you could still draw a curved or wavy line (or a circle) that separates them reasonably well → try a polynomial kernel first.
Completely intermingled with no hope of drawing any clean boundary curve → fall back to the RBF kernel, the most flexible and common default when nothing simpler works.

So the order of preference is always: linear → polynomial → RBF, moving to the next only when the previous, simpler option visibly can’t separate the classes.

The practical approach:

If you can visually inspect the data (or it’s very high-dimensional and sparse, e.g. text with TF-IDF features), follow linear → polynomial → RBF as above.
Otherwise, kernel='rbf' is a reasonable general-purpose default to start experimenting with.
Use cross-validation to compare kernels and tune C and gamma together, since they interact.

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC

param_grid = {
    'kernel': ['linear', 'rbf'],
    'C': [0.1, 1, 10],
    'gamma': ['scale', 0.01, 0.1, 1],
}

grid = GridSearchCV(SVC(), param_grid, cv=5, scoring='accuracy', n_jobs=1)
grid.fit(X_train, y_train)

print("Best combination:", grid.best_params_)
print("Best CV accuracy:", grid.best_score_)

8. Applied lab — tuning an RBF spam classifier and comparing it to linear

Let’s continue the email spam classifier and apply everything from this part: build a nonlinear RBF model, tune its two hyperparameters (C and gamma) with cross-validation, and compare the result against a tuned linear SVM.

import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, GridSearchCV, KFold
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.metrics import confusion_matrix, accuracy_score, precision_score, recall_score

# Simulated spam dataset: 57 word-frequency-style attributes (like the classic spam email dataset)
X, y = make_classification(
    n_samples=800, n_features=57, n_informative=57, n_redundant=0,
    n_clusters_per_class=1, flip_y=0.18, class_sep=1.8, random_state=7
)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

# Standardization is mandatory for the RBF kernel — it relies on Euclidean
# distance ‖Xi - Xj‖, so every feature must contribute on the same scale.
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = scaler.transform(X_test)

Step 1: A first, untuned RBF model

rbf = SVC(kernel='rbf', C=1)
rbf.fit(X_train_s, y_train)
pred_rbf = rbf.predict(X_test_s)

print(confusion_matrix(y_test, pred_rbf))
print("accuracy:", accuracy_score(y_test, pred_rbf))
print("precision:", precision_score(y_test, pred_rbf))
print("recall:", recall_score(y_test, pred_rbf))

Step 2: Tune C and gamma together with GridSearchCV

The RBF kernel has three hyperparameters to choose from in practice: the kernel itself, C, and gamma. Since C and gamma interact, they must be searched together, not one at a time.

kf = KFold(n_splits=5, shuffle=True, random_state=42)
param_grid = {'C': [1, 10, 100, 1000], 'gamma': [0.01, 0.001, 0.0001]}

grid = GridSearchCV(SVC(kernel='rbf'), param_grid, cv=kf, scoring='accuracy', n_jobs=1)
grid.fit(X_train_s, y_train)

print("Best params:", grid.best_params_)
print("Best CV accuracy:", grid.best_score_)

results_df = pd.DataFrame(grid.cv_results_)
for g in [0.01, 0.001, 0.0001]:
    sub = results_df[results_df['param_gamma'] == g][['param_C', 'mean_test_score']]
    print(f"gamma={g}")
    print(sub.to_string(index=False))

Reading the effect of C at each fixed gamma value tells the same overfitting story as the last section, just concretely:

gamma=0.01 (high gamma): accuracy peaks earliest, at C=1 (0.920), then drops sharply as C grows (0.880 at C=10 and beyond) — the model overfits quickly once it’s allowed to fit tightly and the kernel is already highly localized.
gamma=0.001 (middle gamma): accuracy peaks later, at C=10 (0.920), and stays high over a broader range of C before declining — a more robust setting.
gamma=0.0001 (low gamma): the model underfits badly at C=1 (0.466, near random guessing) because the kernel is too smooth to do anything useful, but accuracy climbs steadily as C increases, eventually matching the other settings (0.920 at C=100).

This is the practical version of the bias-variance curve from earlier: too much flexibility (high gamma + high C) overfits, too little (low gamma + low C) underfits, and the best operating point sits in between.

Step 3: Evaluate the best RBF model, and compare to a tuned linear SVM

best_rbf = grid.best_estimator_
pred_best = best_rbf.predict(X_test_s)
print("Tuned RBF — accuracy:", accuracy_score(y_test, pred_best))
print("Tuned RBF — precision:", precision_score(y_test, pred_best))
print("Tuned RBF — recall:", recall_score(y_test, pred_best))

param_grid_lin = {'C': [0.01, 0.1, 1, 10, 100]}
grid_lin = GridSearchCV(SVC(kernel='linear'), param_grid_lin, cv=kf, scoring='accuracy', n_jobs=1)
grid_lin.fit(X_train_s, y_train)

best_lin = grid_lin.best_estimator_
pred_best_lin = best_lin.predict(X_test_s)
print("Tuned linear — accuracy:", accuracy_score(y_test, pred_best_lin))
print("Tuned linear — precision:", precision_score(y_test, pred_best_lin))
print("Tuned linear — recall:", recall_score(y_test, pred_best_lin))

Running this: the tuned RBF model (C=1, gamma=0.01) achieves 91.7% test accuracy, while the tuned linear model (C=0.01) reaches the exact same 91.7% test accuracy — with identical precision and recall too. The linear model got there with a single hyperparameter to tune, instead of two that interact with each other.

The takeaway: in high-dimensional feature spaces (57 attributes here), a simple linear boundary can be just as good as — or even better than — a carefully tuned nonlinear RBF boundary, while being far cheaper and simpler to tune. Always compare a tuned linear SVM against a tuned nonlinear one; don’t assume “nonlinear = better.”

9. Practice questions

What is the key mathematical fact that makes the kernel trick computationally efficient?
Why does feature transformation become expensive as the number of original attributes grows?
Which kernel would you try first for a text classification problem with thousands of TF-IDF features? Why?
What happens to the decision boundary when gamma is set very high? What risk does this create?

Answers:

The SVM optimization only ever needs inner (dot) products between pairs of points, never the individual transformed points themselves. Kernels compute this dot product directly, skipping the expensive transformation step.
Because a full polynomial (or similar) transformation creates a combinatorial explosion of new features — pairwise products, squares, etc. — that grows extremely fast with the number of original attributes.
Linear kernel — high-dimensional, sparse text data is often already close to linearly separable, and the linear kernel is far faster to train at that scale.
The boundary becomes highly localized and “wiggly,” fitting the training data very tightly. This increases the risk of overfitting (low bias, high variance).

10. Summary — the full SVM series

Part 1: SVM is a linear model that separates classes with a hyperplane: ∑(Wi·Xi) + W0 = 0.
Part 2: The Maximal Margin Classifier picks the hyperplane with the widest margin, defined by the closest points (support vectors) — but it is fragile on noisy, non-separable data.
Part 3: The Soft Margin Classifier tolerates some misclassification using slack variables (ε), controlled by the cost parameter C — a direct expression of the bias-variance tradeoff.
Part 4 (this topic): Kernels let SVM separate non-linear data by implicitly working in a transformed feature space, without ever computing that transformation explicitly. gamma controls how flexible/wiggly the resulting boundary is.

Together, these four ideas — hyperplanes, margins, soft margins, and kernels — are the complete foundation of how Support Vector Machines work.

Related reading: Bias and Variance (prerequisite), Naive Bayes — A Friendly Guide (a complementary classification algorithm often compared with SVM).