Introduction

This will probably be my last post about CS231n. After opening this GitHub blog, I began moving posts from my old Naver blog while comparing them with the original lecture notes. Looking back, I found many vaguely written passages and quite a few sections I had written without fully understanding them. It has only been a year, so I may not know much more now, but revisiting what I studied while reorganizing it seems effective for strengthening the fundamentals.
The material from here on involves less mathematical proof and derivation than the previous discussions of neural-network architecture, parameters, and functionality.

Visualizing what convolutional neural networks learn

After a CNN first won the ImageNet competition, deep learning—and neural-network-based algorithms in particular—began attracting attention.

XRCE, shown in the table, was not based on deep learning. Although errors before 2011 are omitted, the 2011 error rate was effectively regarded as the lowest attainable convergence without deep learning. AlexNet arrived as the game changer that overturned this assumption and the algorithmic landscape, improving performance by an astonishing $10\%$. From then on, deep-learning networks won every competition.
Even after deep CNNs won ImageNet, persistent questions and criticism emerged, largely from researchers who had used conventional computer algorithms and machine learning for low-level vision tasks. They agreed that deep learning transformed the field through major performance gains, but argued that it was impossible to explain rigorously how it worked. Earlier posts described architectures such as the MLP (Multi-Layer Perceptron) and explained learning in terms of forward propagation, an objective function, and backpropagation. Understanding how training optimizes a loss requires mathematical knowledge such as gradients and linear algebra. But for deep learning to be recognized as a research discipline, it also needed mathematical support explaining why learning this way produced such extraordinary performance gains.
This question became important in computer science because, unlike deterministic algorithms, data-driven algorithms do not readily reveal their principles or a clear direction for improvement. Critics found it insufficiently explanatory to say only that a perceptron imitating human neurons evolved into an MLP and then a CNN for computer vision, and happened to perform well.

The term “black box” came to describe this situation: the inputs and outputs of a neural network can be explained mathematically, but the basis on which its internal parameters generalize to predictions cannot. Apart from explicit inputs and outputs, the implicitly learned interior cannot be observed directly, so a neural network alone did not solve this black-box problem. Even AlexNet, the first ImageNet-winning network, was criticized because its paper could not describe these issues in detail.

Explaining network performance was necessary not only to persuade the academic community. Further deep-learning research required explainability to improve network design and understand samples that produced errors. Explainability was thus needed both to convince established researchers and as a foundation for advancing the work itself.

Researchers therefore began to inspect learning inside deep networks indirectly, through weight visualization and visualization of layer activations $(f_l \circ f_{l-1} \circ \cdots \circ f_1(x))$ for a particular input $x$. Such work sought to support explanatory accounts of performance.

Layer activations

One visualization technique displays a layer's activation for an input during the forward pass. In a network with ReLU, activations initially look blobby and dense, but become increasingly sparse and localized as training proceeds. I wrote the following quick code to inspect the result.

import torch
import torchvision
import torchvision.transforms as transforms

# Define CIFAR10 dataset
batch_size = 16

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,
                                          shuffle=True, num_workers=2)
testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size,
                                         shuffle=False, num_workers=2)

Training uses CIFAR-10. The network uses ReLU as its activation function and also includes batch normalization.

# Let's define simple CNN model
import torch
import torch.nn as nn
import torch.nn.functional as F

class SMPCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.feature1 = nn.Sequential(
            nn.Conv2d(3, 32, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(inplace=True)
        )
        self.feature2 = nn.Sequential(
            nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True)
        )
        self.feature3 = nn.Sequential(
            nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(inplace=True)
        )
        self.feature4 = nn.Sequential(
            nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(inplace=True)
        )
        self.feature5 = nn.Sequential(
            nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(inplace=True)
        )

        self.classifier = nn.Sequential(
            nn.Linear(256*2*2, 64),
            nn.Dropout(0.5),
            nn.Linear(64, 10)
        )

    def forward(self, x):
        f1 = self.feature1(x)
        f2 = self.feature2(F.max_pool2d(f1, kernel_size=2, stride=2))
        f3 = self.feature3(F.max_pool2d(f2, kernel_size=2, stride=2))
        f4 = self.feature4(F.max_pool2d(f3, kernel_size=2, stride=2))
        f5 = self.feature5(F.max_pool2d(f4, kernel_size=2, stride=2))
        f5 = f5.view(f5.size(0), -1)
        score = self.classifier(f5)

        return score, (f1, f2, f3, f4, f5)

To visualize each layer's output, I return it in addition to score. Training under the chosen configuration is implemented as a function.

# Let's define trainer function
from collections import defaultdict

def trainer(model, epochs, train_loader, val_loader, optimizer, criterion, device):
    features = defaultdict(list)
    model.to(device)
    for epoch in range(epochs):
        model.train()
        training_loss = 0.0
        training_acc = 0.0
        for i, (image, label) in enumerate(train_loader):
            optimizer.zero_grad()
            image, label = image.to(device), label.to(device)
            score, activations = model(image)
            loss = criterion(score, label)
            loss.backward()
            optimizer.step()
            _, prediction = torch.max(score, axis=1)
            accuracy = float(torch.sum(torch.eq(prediction, label)))/len(prediction)
            training_loss += loss.item()
            training_acc += accuracy

            if i%200 == 0:
                print(f"Epoch [{epoch+1}/{epochs}] (Iter [{i+1}/{len(train_loader)}]) ===> Training loss : {training_loss/(i+1):.6f}, Training accuracy : {100*training_acc/(i+1):.2f}%")
            
            if i%1000 == 0:
                features[epoch] += [activations]
      
        model.eval()
        with torch.no_grad():
            validation_loss = 0.0
            validation_acc = 0.0
            for i, (image, label) in enumerate(val_loader):
                image, label = image.to(device), label.to(device)
                score, _ = model(image)
                loss = criterion(score, label)
                _, prediction = torch.max(score, axis=1)
                accuracy = float(torch.sum(torch.eq(prediction, label)))/len(prediction)
                validation_loss += loss.item()
                validation_acc += accuracy                    

            print(f"Epoch [{epoch+1}/{epochs}] ===> Validation loss : {validation_loss/len(val_loader):.6f}, Validation accuracy : {100*validation_acc/len(val_loader):.2f}%")
    return features

The function saves the output feature maps at fixed iteration intervals and returns the stored features as a dictionary containing lists. As the code shows, training uses a batch size of 16 for 20 epochs, with a fixed learning rate of $10^{-3}$ and no scheduling.

# Train configurations
model = SMPCNN()
epochs = 20
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=5e-4)
criterion = nn.CrossEntropyLoss()
device = "cuda" if torch.cuda.is_available() else "cpu"

Since this code trains a classification task, criterion is cross-entropy loss. The trained model reaches $79.49\%$ accuracy on the test set and nearly $90\%$ on the training set. It has overfit slightly, but regularization such as data augmentation could readily improve that. Our concern is not network performance but feature-map activations during training, so I will move on. Visualization uses PyTorch's make_grid method. Rather than include the detailed code, let us analyze the results. The following three images show the mean feature map over a batch of 16 images for each layer immediately after training begins (epoch 0).

The early-training images look as shown above. As noted earlier, activations begin dense and blobby; the feature maps are not especially smooth overall. As training proceeds, they become increasingly sparse and localized, as the figures below show.

Because the difference was not obvious in the mean activation map, I separated the feature maps and visualized the properties captured by each channel.

The three figures above separate each layer's early-training activation map by channel. The channels fail to capture meaningful differences in features and look similar to one another; their prediction features still depend heavily on the training set. They effectively filter the raw contours of the image, extracting features that ordinary non-deep-learning algorithms could readily produce.

The activation maps after some training look quite different. They clearly show sparsity—meaningful features appear in only some channels—which aligns with ReLU learning not to emit signal values it considers irrelevant. This visualization also exposes a drawback of ReLU: dead activations, which prevent the network from using every designed filter channel meaningfully.

Visualize convolutional, fully-connected filters

Besides inspecting activations for an input, we can examine learned weights. Early convolutional layers are easy to interpret because their inputs are images made of raw pixels, but filter weights in deeper network layers can also be visualized. A well-trained network generally produces smooth patterns without noise. After slightly modifying and training the network above, the weights of the first and second convolutional layers in the final trained network look as follows.

Weight visualization really requires convolutional filters with large kernels. Modern networks often enlarge the receptive field through deeper layers instead, so directly visualizing weights is unlikely to reveal much. ZFNet, one of the earliest visualization studies, used visualization to improve AlexNet and won the ImageNet competition the following year.

Retrieving images that maximally activate a neuron

Another visualization technique passes a large collection of dataset images through the network and examines each neuron's activation. This reveals which part of the image each neuron attends to within its receptive field. The technique was introduced in R-CNN, an ancestor of deep-learning-based object detection.

The problem is that a ReLU neuron has no semantic meaning by itself. It is more useful to treat ReLU neurons across multiple layers as basis vectors of a space, where each axis acts as a switch that turns a signal on or off, forming a coordinate system for images. Put differently, these visualizations show elements on the edges or boundaries of a system of representations and explore only directions corresponding to filter weights. Since a convolutional neural network is linear with respect to its input and cannot explore trajectories, it is unable to search diverse directions in representation space (reference).

Embedding code with t-SNE

A convolutional neural network extracts a score map from which a linear classifier can separate an image into different classes. Since classification embeds a high-dimensional image into a lower-dimensional space, we can expect the low- and high-dimensional topologies to be roughly similar. Let us compare t-SNE results using the network trained above. First, the following code maps CIFAR-10 directly onto a two-dimensional manifold with t-SNE, without applying a deep network.

cls = []
embedding = []
for data in testloader:
    images, labels = data[0].to(device), data[1].to(device)
    embedding += images.view(images.shape[0], -1).cpu().numpy().tolist()
    cls += labels.cpu().numpy().tolist()

tsne = TSNE(n_components=2, random_state=0)
points = np.array(tsne.fit_transform(np.array(embedding)))
classes = np.array(cls)

plt.figure(figsize=(10, 10))
cifar = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']
for i, label in zip(range(10), cifar):
    idx = np.where(classes == i)
    plt.scatter(points[idx, 0], points[idx, 1], marker='.', label=label)

plt.legend()
plt.show()

As the resulting figure shows, simply embedding the high-dimensional images in two dimensions does not separate the classes well.

This time, the following code visualizes with t-SNE the final 256-dimensional feature map produced by the classification network.

cls = []
deep_features = []
model.eval() # resnet18
with torch.no_grad():
    for data in testloader:
        images, labels = data[0].to(device), data[1].to(device)
        _, features = model(images)
        deep_features += features[-1].cpu().numpy().tolist()
        cls += labels.cpu().numpy().tolist()

tsne = TSNE(n_components=2, random_state=0)
points = np.array(tsne.fit_transform(np.array(deep_features)))
classes = np.array(cls)

plt.figure(figsize=(10, 10))
cifar = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']
for i, label in zip(range(10), cifar):
    idx = np.where(classes == i)
    plt.scatter(points[idx, 0], points[idx, 1], marker='.', label=label)

plt.legend()
plt.show()

Unlike the previous visualization, this one separates the classes reasonably well. It shows that classification performance can be visualized directly even for a custom-designed network.

Occluding parts of the image

Another method occludes part of the input image and measures confidence—the probability assigned to the correct class. For this experiment, I use a ResNet-18 pretrained on ImageNet.

import torch
import cv2
import matplotlib.pyplot as plt
import torchvision.transforms.functional as TF
from torchvision.models import ResNet18_Weights
from tqdm import tqdm
import numpy as np

model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet18', weights=ResNet18_Weights.DEFAULT)
model.to("cuda")
model.eval()

A pretrained ResNet-18 can be loaded easily from torchvision. Move the model to the GPU and switch its layers to evaluation mode, since we are evaluating rather than training. The image used is shown below.

Unlike CIFAR, ImageNet contains 1,000 classes. It does not place every cat image in the same class, but can distinguish individual breeds. To feed the image above into the pretrained ResNet-18, preprocess it as follows.

test_img = cv2.cvtColor(cv2.imread("test_img.jpg"), cv2.COLOR_BGR2RGB)
resized = cv2.resize(test_img, (224, 224))
input_image = TF.to_tensor(resized)
input_image = TF.normalize(input_image, [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
input_image = input_image.unsqueeze(0)

Since ImageNet models are trained at a resolution of $224\times224$, resize the image accordingly to match the classifier dimensions. Convert the NumPy image array to a tensor and normalize it at the same time.

A prediction is not meaningful unless we know which class its index represents, so I use a file that maps indices to class names (reference).

with open("idx2label.txt") as f:
    idx2label = eval(f.read())

with torch.no_grad():
    output = model(input_image.to("cuda"))
    _, prediction = torch.max(output, axis=1)
    print(idx2label[prediction.item()])

After converting the model's scores to softmax probabilities, the index with the largest value becomes the predicted class. The prediction correctly maps to “Siamese cat, Siamese”. We can now perturb the original image and see how the prediction changes.

masked_img = []
mask_size = 32
for i in range(224):
    for j in range(224):
        left = int(max(0, j-mask_size//2))
        right = int(min(224, j+mask_size//2))
        top = int(max(0, i-mask_size//2))
        bottom = int(min(224, i+mask_size//2))
        
        masked = resized.copy()
        masked[top:bottom, left:right, :]=0
        masked_img.append(masked)

confidences = np.zeros(224*224)
correct = 0
total = 0

loading = tqdm(enumerate(masked_img))
for i, img in loading:
    input_image = TF.to_tensor(img)
    input_image = TF.normalize(input_image, [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    input_image = input_image.unsqueeze(0)
    with torch.no_grad():
        output = model(input_image.to("cuda"))
        output = torch.softmax(output, dim=1)
        _, estimate =  torch.max(output, axis=1)
        total += 1
        if estimate.item() == 284:
            correct += 1
        confidences[i] = output[:, 284].item()
        loading.set_description(f"Accuracy : [{correct}/{total}]")

The mask size is $32\times32$. Confidence is assigned to the mask's center coordinate. Across masks applied at every position in the $224\times224$ image, 43,348 positions preserve the original class prediction. The confidence heatmap is shown below.

Prediction confidence falls when the central region containing the Siamese cat is masked. The binary map below marks only locations that cause a wrong prediction—that is, locations whose masking lowers confidence in the correct class until another class exceeds it.

The prediction becomes incorrect when the mask center lies in a black region on the left. Mapping those regions back onto the original image provides an explanation: the network consults the cat's face when making its prediction.

Transfer learning and Fine-tuning

Studying AI inevitably involves reading many papers. Material such as CS231n is really only an overview of AI. Deep learning contains a huge variety of specialized tasks, and each task uses different methods and research directions, making it difficult to follow the trends. Regardless of the task, however, beginning research requires understanding transfer learning and fine-tuning. These concepts are used almost universally.

ImageNet-winning networks such as AlexNet, VGGNet, and ResNet were trained on enormous datasets for days or even weeks. ImageNet contains 1.2 million images, while the larger ImageNet-22k contains 14,197,122. To adapt a network to a desired task, there is no need to train it from scratch. A network that has learned meaningful representations from a large dataset can be applied to a new task without another long training period. Transferring a trained model's representations to a new task is called transfer learning. Fine-tuning is a transfer-learning method that adjusts the parameters. One may use the trained model as is or add a task-specific architecture to its backbone, then separate frozen layers from trainable ones and modify them for the desired objective.

Using a Convolutional Neural Network as a Fixed Feature Extractor

Suppose the ConvNet is AlexNet pretrained on ImageNet. Its final fully connected layer is a classifier that produces scores for 1,000 classes. Unless the new task is another 1,000-class classification problem, this architecture is unnecessary. Removing it leaves only feature extraction. Freeze those parameters and train a new classifier on the new dataset. As discussed earlier, that classifier could be a linear SVM (Support Vector Machine) or a softmax classifier. This corresponds to the third diagram above. R-CNN, for example, takes a model trained on ImageNet and uses several fine-tuning steps, including SVM training, to transfer its representation to object detection.

Fine-Tuning a Convolutional Neural Network

The convolutional feature extractor need not remain fixed; its weights can also be adjusted through backpropagation. To avoid overfitting the new dataset, some layers may stay frozen while only others are updated. The first and second diagrams above illustrate these cases.

How to fine-tune?

Fine-tuning requires a strategy for deciding how much of the network to adjust, but there is no fixed rule for choosing layers for a given task. In practice, people usually consider the size of the available dataset and its distributional difference from the data used to train the source network. The following guidance applies in common cases, but is not always correct.

Small New Dataset, Similar to the Original Dataset

Fine-tuning the convolutional network risks overfitting because there is little data. Since the new and original data are similar, however, their high-level features (coarse feature maps) are likely similar. It is therefore best to freeze the feature extractor and train only the linear classifier.

Large New Dataset, Similar to the Original Dataset

This is the most favorable case. The whole model can be trained, or some layers can be fine-tuned to reuse existing weights. With enough data, overfitting is less concerning, so fine-tuning most of the network can raise performance and produce task-specific behavior.

Small New Dataset, Very Different From the Original Dataset

This is the hardest case. As in the first case, the dataset is so small that only the classifier should be fine-tuned. But because the datasets differ, we cannot assume their feature maps are similar. Rather than training a classifier on high-level feature maps to replace the network head, it may help to train an SVM on early-network feature activations. Performance varies greatly with the chosen method in this setting, so there is no definitive answer.

Large New Dataset, Very Different From the Original Dataset

If the dataset is entirely different, training from scratch may seem preferable. In practice, performance comparisons and convergence still show that adjusting a pretrained model's weights is more effective. Since plenty of data is available, fine-tuning the entire network can learn useful high-level features.

Fine-tuning involves several considerations. A pretrained model constrains the architecture, making it harder to build a network optimized for a different metric or task. And because training moves an existing optimization point instead of learning from scratch, fine-tuning uses a small learning rate. A typical rate is $1/100$ or $1/1000$ of what would be used to train a network from scratch.