Part 11: CNN Training Pipeline, Transfer Learning, and Visualization — A Friendly Guide

What this guide covers

You now understand how a CNN works and how it learns through backpropagation.

But understanding the architecture is just the beginning.

Before you can train a CNN on a real problem, you need to answer three questions:

1. How do I prepare my data?
2. Should I train from scratch or reuse an existing model?
3. How do I know what the model actually learned?

This guide answers all three.

1. The CNN model design pipeline

Designing a CNN for a real problem follows a clear sequence of steps.

Step 1: Explore the problem
Step 2: Collect data
Step 3: Annotate data
Step 4: Preprocess data
Step 5: Augment data
Step 6: Split into train / validation / test
Step 7: Train the model
Step 8: Evaluate and iterate

Each step matters. Skipping one usually causes problems later.

2. Step 1: Explore the problem

Before collecting any data, understand what you are trying to do.

Ask yourself:

Example:

Problem: classify vehicle images into car, bus, truck
→ supervised classification
→ 3 output neurons with softmax

You also need to ask: is my dataset diverse enough for this problem?

If the task is “classify all vehicles” but your dataset only has cars, buses, and trucks, the model is not ready for motorcycles or cycles even though they are also vehicles.

Diverse = enough coverage of all the variation in your real-world problem

3. Step 2: Collect data

A CNN is a parametric model — it has many learnable weights and biases inside its convolution layers and dense layers.

To learn all those parameters correctly, it needs enough examples.

A rough rule of thumb:

More complex model → more parameters → needs more data

For a simple CNN, a few thousand images per class may be enough.

For a deep CNN like VGG16, you may need tens of thousands per class if training from scratch.

Imbalanced data

Your dataset also needs to be balanced — roughly equal examples per class.

Example of an imbalanced dataset:

1000 car images
 900 bus images
  20 truck images

The model will see cars and buses hundreds of times per epoch, but trucks only 20 times.

It will learn cars and buses well but fail on trucks.

Options to fix imbalance:

• Collect more data for the minority class
• Use data augmentation to create more variety from the few examples you have
• Use class weights during training to penalize mistakes on rare classes more heavily

4. Step 3: Annotate data

Each image needs a correct label.

If an image of a cat is labeled as “dog”, the model is learning the wrong mapping.

wrong label → wrong gradient → wrong weight update → broken model

For classification:

• Each image gets one class label
• Labels must be consistent across the whole dataset

For detection or segmentation:

• Annotations can include bounding boxes or pixel-level masks
• These require more careful tooling

Memory trick: Garbage in, garbage out. No amount of architecture complexity fixes a badly labeled dataset.

5. Step 4: Preprocess data

Raw images collected in the real world often contain noise or distortion.

Common sources:

• Camera sensor noise during capture
• Compression artifacts during storage
• Network transmission errors

Preprocessing cleans up the data before giving it to the model.

Noise removal and enhancement

Salt-and-pepper noise is a common noise type where random pixels are set to either pure white or pure black — like a scattering of white and black dots across the image.

The median filter removes this well because the extreme values get replaced by the actual local neighborhood values.

Resizing

CNNs expect a fixed input size.

VGG16 expects: 224 × 224 × 3
ResNet50 expects: 224 × 224 × 3

All images in your dataset must be resized to match the expected input before training.

Normalization

Raw pixel values range from 0 to 255.

When different features have very different ranges, the model may learn them unevenly.

Normalization rescales values to a common range so the model treats all inputs fairly.

Min-max normalization

Rescales values to be between 0 and 1:

normalized value = (value - min) / (max - min)

Example for pixel values:

original pixel: 180
min: 0, max: 255
normalized: 180 / 255 = 0.706

Mean and standard deviation normalization (standardization)

Rescales values so the distribution has mean 0 and standard deviation 1:

normalized value = (value - mean) / std

This is the most common method for image inputs to CNNs.

ImageNet pretrained models typically use:

mean = [0.485, 0.456, 0.406]  (one per RGB channel)
std  = [0.229, 0.224, 0.225]

In PyTorch:

from torchvision import transforms

transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225]
    ),
])

Memory trick: Normalization is like converting everyone’s score to the same grading scale. A 90 out of 100 and a 900 out of 1000 are both 90%, so the model treats them the same.

6. Step 5: Augment data

See the full augmentation explanation in the Advanced CNN guide, Section 4.

Quick summary:

Original image → rotate, flip, crop, change brightness, add noise
→ multiple new training examples
→ model learns general patterns, not exact photos

Augmentation is applied only to the training set. Validation and test sets are not augmented.

7. Step 6: Train / validation / test split

Split your dataset into three separate parts before any training begins.

Full dataset
  ├── Training set   (~70–80%)  → used to update weights and biases
  ├── Validation set (~10–15%)  → used to monitor performance each epoch
  └── Test set       (~10–15%)  → used once at the very end to evaluate the final model

What happens each epoch

Epoch 1:
  → train on training set → update weights
  → evaluate on validation set → check loss

Epoch 2:
  → train on training set → update weights further
  → evaluate on validation set → is loss improving?

...

Epoch N:
  → validation loss stops improving → stop training
  → evaluate once on test set → final score

The test set represents the real world. The model has never seen it during training or validation.

Critical rule: Never mix test data into training. Split before training begins.

Memory trick: Training set is the textbook. Validation set is the practice exam after each chapter. Test set is the final exam you only sit once.

8. Training from scratch vs. transfer learning vs. fine-tuning

Once the data is ready, you need to decide how to train.

There are three strategies.

Strategy 1: Training from scratch

All weights start at random values.

All weights = random
→ model learns everything from your data alone

When to use:

• You have a very large, labeled dataset (tens of thousands of images per class)
• Your task is very different from anything in ImageNet
• You have the compute resources to run many epochs

Challenge: CNN models have millions of parameters. With too little data, the model cannot learn all of them correctly.

3×3×3 filter = 27 weights + 1 bias = 28 parameters per filter
VGG16 total  = ~138 million parameters

You need enough examples to give the model enough signal to update all those weights meaningfully.

Strategy 2: Transfer learning

Take a CNN already trained on a large dataset (such as ImageNet), freeze its weights, and attach a new classifier head for your task.

Pretrained CNN (frozen)     → acts as a feature extractor
New classifier head (trained) → learns your specific classes

Why this works:

Early CNN layers learn very general features — edges, corners, textures, color gradients.

These features are useful across many image tasks, not just the original task.

ImageNet-trained layer 1 → detects edges
ImageNet-trained layer 2 → detects corners and simple textures
ImageNet-trained layer 3 → detects more complex patterns

These learned features transfer to your task even if the classes are different.

When to use:

• Your dataset is small (a few hundred to a few thousand images per class)
• Your task is related to what ImageNet contains (natural images, objects, scenes)

When transfer learning may not work:

• Your task is very different from ImageNet (e.g., medical ultrasound, satellite imagery from unusual sensors)
• Even then, it is worth trying — it has worked surprisingly well in some unrelated domains

Strategy 3: Fine-tuning

Same as transfer learning, but you also unfreeze some of the later pretrained layers and train them alongside the new classifier.

Pretrained CNN early layers (frozen)  → keep general features as-is
Pretrained CNN later layers (unfrozen) → update slightly for your task
New classifier head (trained)          → learns your specific classes

Why fine-tune later layers and not early ones?

Early layers → general features (edges, corners) → useful everywhere → freeze
Later layers → task-specific features → may need small adjustment → fine-tune

When to use:

• Transfer learning alone gives good but not great results
• Your dataset is somewhat related to ImageNet but has its own specific variations
• You have some compute budget for the extra training

Decision guide

In PyTorch

import torchvision.models as models
import torch.nn as nn

# Load pretrained VGG16
model = models.vgg16(pretrained=True)

# Freeze all layers
for param in model.parameters():
    param.requires_grad = False

# Replace the final classifier for your number of classes
model.classifier[6] = nn.Linear(4096, 3)  # 3 classes: car, bus, truck

# Now only model.classifier[6] has requires_grad = True

For fine-tuning, unfreeze the last few layers after the initial transfer learning phase:

# Unfreeze last two feature blocks
for param in model.features[24:].parameters():
    param.requires_grad = True

9. CNN visualization: what did the model actually learn?

For a long time, CNNs were treated as black boxes.

You gave them input, they produced output, and nobody knew exactly what was happening inside.

In 2014, researchers published methods to visualize the intermediate layers of a trained CNN. This opened up the interpretation of what CNNs actually learn.

What each layer learns

When you visualize intermediate feature maps from a trained CNN, a clear pattern appears:

Early layers  → detect edges, lines, color boundaries
Middle layers → detect textures, corners, simple shapes
Later layers  → detect complex object parts (eyes, wheels, windows)

This is the hierarchical learning that makes CNNs so powerful — simple patterns combine into complex ones.

Testing phase: weights are fixed

Once a CNN is trained, the weights and biases do not change during inference.

Input image 1: car  → same weights → highest activation on car neuron
Input image 2: bus  → same weights → highest activation on bus neuron
Input image 3: truck → same weights → highest activation on truck neuron

The same fixed weights generalize to different inputs and produce the right output for each, in an ideal scenario.

Activation maps

After each layer, the output is called an activation map (or feature map).

The final activation map before softmax tells you how strongly each class is being activated for the given input.

Final activation map for a car input:
  car  → 0.91
  bus  → 0.06
  truck → 0.03

The model predicts “car” because that activation is highest.

Grad-CAM: visualizing where the model looks

Grad-CAM (Gradient-weighted Class Activation Mapping) is a technique that generates a heatmap showing which spatial regions of the input image the model focused on most for a given prediction.

Simple idea:

1. Run a forward pass → get prediction
2. Compute gradients of the predicted class score
   with respect to the last convolutional layer's activations
3. Use those gradients to weight the activation maps
4. Average the weighted maps → one heatmap
5. Resize and overlay on the original image

The result is a heatmap where:

• Red = high activation, model focused here
• Blue = low activation, model ignored this region

Example: give a CNN an image of a beach scene with people flying kites.

The Grad-CAM heatmap will show the highest activation around the kites and the people — the regions that drove the prediction.

This is possible because of the spatial structure of convolution. Each location in a feature map corresponds to a specific region in the input image. Grad-CAM traces the gradients back through that spatial correspondence.

Feature map location (x, y)
→ corresponds to receptive field at (x, y) in the input image
→ gradient tells us how important that location was
→ visualize as a heatmap

Why Grad-CAM matters

Without visualization, you cannot tell whether the model learned the right thing.

Examples of things Grad-CAM can reveal:

These are called shortcut features — the model learned a spurious correlation rather than the real signal.

Grad-CAM lets you catch this before deployment.

Simple Grad-CAM in PyTorch

import torch
import torchvision.models as models
import torchvision.transforms as transforms
from PIL import Image
import numpy as np

model = models.vgg16(pretrained=True)
model.eval()

# Hook to capture gradients and activations from last conv layer
activations = {}
gradients = {}

def save_activation(name):
    def hook(module, input, output):
        activations[name] = output
    return hook

def save_gradient(name):
    def hook(module, grad_input, grad_output):
        gradients[name] = grad_output[0]
    return hook

# Register hooks on last conv layer
model.features[28].register_forward_hook(save_activation('last_conv'))
model.features[28].register_backward_hook(save_gradient('last_conv'))

# Forward pass
output = model(image_tensor)
pred_class = output.argmax(dim=1).item()

# Backward pass for predicted class
model.zero_grad()
output[0, pred_class].backward()

# Compute Grad-CAM
grads = gradients['last_conv']         # shape: [1, C, H, W]
acts  = activations['last_conv']       # shape: [1, C, H, W]
weights = grads.mean(dim=[2, 3])       # average over spatial dims → [1, C]
cam = (weights[:, :, None, None] * acts).sum(dim=1)  # weighted sum → [1, H, W]
cam = torch.relu(cam)                  # keep positive values only

The resulting cam tensor can be resized and overlaid on the original image to produce the visualization.

10. Summary

The full pipeline in one line:

problem → data → clean → augment → split → train → visualize → evaluate

What to read next

• Part 9: Convolutional Neural Networks — CNN architecture from scratch
• Part 10: Advanced Convolutional Neural Networks — overfitting, dropout, batch normalization, residual connections, backpropagation
• Part 4: Data Handling with Dataset and DataLoader — how to build training pipelines in PyTorch