Naive Bayes Part 1: Probabilistic Classification Intuition

This is Part 1 of the Naive Bayes mini-series. The goal is to build the classifier from first principles, one step at a time.

• Part 1 (this topic): Probabilistic classification intuition
• Part 2: Deriving Bayes’ theorem from conditional probability
• Part 3: From Bayes’ theorem to the Naive Bayes classifier
• Part 4: Gaussian Naive Bayes from scratch in Python
• Part 5: Multinomial Naive Bayes
• Part 6: Bernoulli Naive Bayes

1. Classification is not about certainty — it’s about probability

Most real-world classification problems are not 100% obvious. When your email client decides whether a message is spam, it does not know — it makes a best guess based on the evidence inside the email.

Naive Bayes answers classification questions using probability:

Given the words/features I see, what is the probability that this email belongs to class spam? What about class not spam?

The class with the higher probability wins.

2. A concrete spam example

Imagine you receive an email with the subject:

“Congratulations! You have won a free iPhone.”

From experience, you know:

These numbers are conditional probabilities — the probability of seeing a specific word given the email type.

Naive Bayes combines these individual clues into a single verdict: spam or not spam. The key idea is to turn many small pieces of evidence into a total score for each class, then pick the class with the highest score.

3. The two probabilities we always need

For every class C (e.g. spam, not spam) and every new sample with features X, we need two things:

1. Prior probability: P(C) — how common the class is overall.
   • If 30% of your emails are spam, then P(spam) = 0.30.
2. Likelihood: P(X | C) — how likely the observed features are if the class is true.
   • If the email contains “free,” how likely is that word in spam emails vs. normal emails?

The final answer we want is the posterior probability:

P(C | X) = probability the class is C, given the features we observed

Bayes’ theorem (covered in Part 2) is the bridge that turns priors and likelihoods into posteriors.

4. The hard part: multiple features

Real emails contain many words, not just one. We need to handle many features at once:

X = [x1, x2, ..., xn]

where each xi could be a word count, a pixel value, a measurement, or a binary yes/no feature.

The direct approach would be to estimate the full joint probability:

P(x1, x2, ..., xn | C)

That is, how likely is this exact combination of features together, given the class. This is where the curse of dimensionality hits: as the number of features grows, the number of possible feature combinations explodes, and most combinations are never seen in the training data.

For example, with just 10 binary features, there are 2^10 = 1,024 possible combinations. With 100 features, there are 2^100 combinations — far more than any dataset could ever cover. Directly estimating the joint distribution becomes impossible.

5. The “naive” assumption that saves everything

Naive Bayes makes one bold simplifying assumption:

Once the class is known, every feature is independent of every other feature.

Formally:

P(x1, x2, ..., xn | C) ≈ P(x1 | C) × P(x2 | C) × ... × P(xn | C)

This is “naive” because features are rarely truly independent. In an email, the words “New York” appear together far more often than two random words. A house’s square footage and number of rooms are positively correlated. But the assumption turns an impossible high-dimensional joint-probability problem into many simple one-dimensional problems — one per feature.

In practice, Naive Bayes often works well even when the independence assumption is violated, because the model only needs to rank the classes correctly, not estimate the exact probabilities perfectly.

6. Decision rule: pick the class with the highest score

For each class, Naive Bayes computes a score proportional to the posterior:

score(C) = P(C) × P(x1 | C) × P(x2 | C) × ... × P(xn | C)

The predicted class is the one with the highest score:

predicted class = argmax_C score(C)

The argmax simply means: “choose the class C that makes the score as large as possible.” In code this is just comparing the scores for each class and returning the winner.

7. Practice questions

1. Why can’t we just estimate the full joint probability P(x1, x2, ..., xn | C) for real datasets with many features?
2. What is the “naive” assumption in Naive Bayes, and why is it unrealistic?
3. Why does Naive Bayes still work well even when the independence assumption is not exactly true?
4. What two quantities does Naive Bayes combine to decide the class of a new sample?

Answers:

1. The number of possible feature combinations grows exponentially with the number of features. With limited data, most combinations never appear, so direct estimation is unreliable.
2. The model assumes all features are independent once the class is known. It is unrealistic because real features often depend on each other (e.g. “New York,” house size and number of rooms).
3. Naive Bayes only needs to rank classes correctly, not produce exact probabilities. The class ranking is often still correct even when the independence assumption is only approximately true.
4. The prior probability P(C) and the likelihood P(xi | C) for each feature. These are combined into a score proportional to the posterior probability.

8. Summary

• Classification can be viewed as asking: “What is the probability of each class, given the evidence?”
• We need two ingredients: the prior P(C) and the likelihoods P(xi | C).
• Real datasets have many features, making direct joint probability estimation impossible due to the curse of dimensionality.
• Naive Bayes assumes feature independence, turning one hard high-dimensional problem into many easy one-dimensional problems.
• We pick the class with the highest score: argmax_C P(C) × ∏ P(xi | C).

Next topic: Deriving Bayes’ Theorem from Conditional Probability