This post covers several ways to train deep-learning networks. Because it discusses methods applicable across a variety of settings rather than training for a single task, it spans a broad range of topics. Before beginning in earnest, here is a brief overview.

1. Transfer learning: DL optimization across different tasks
2. Knowledge distillation: Increasing representation generalization with soft labels
3. Continual learning: Learning diverse representations without losing performance on previous tasks
4. Self-supervision: Learning representations with little or no supervision

What does it mean to train a deep-learning model?

Deep learning proposed gradient-based methods for training deeper layers of Neural Networks, one family of machine-learning techniques, using large datasets. Beginning with the basic classification models that won ImageNet, it has since developed rapidly.
The point is that training a deep-learning model requires a defined task, a modality-specific dataset for solving that task, a loss function to optimize according to the form of the dataset, and an optimization method for gradient-based learning.

Can we build a universal AI?

Tasks must ultimately be defined one by one, and a deep-learning model can be optimized for each. But can we construct a network that, like a human, speaks well, distinguishes objects accurately, answers questions about a scene reasonably, and responds to perception in real time?
“AI that draws” and “ChatGPT” have recently become famous and attracted widespread interest. Generative AI performance is indeed growing at a frightening pace, and network architectures capable of handling many modalities have advanced considerably. Even so, we have not yet developed deep learning that can efficiently solve every real-world problem. Let us therefore examine methods for applying deep learning optimized for one task to a different task.

Transfer learning

Transfer learning, which is also introduced in cs231n, is used throughout modern deep-learning research.
Its central question is whether knowledge or skills learned by deep learning on a “previous task” can be used or applied to a “novel task.”
Consider a human example. A strong chess player will likely adapt more easily to similar games such as checkers or Go.

Likewise, a person who is good at mathematics may find it easier to study AI or learn programming, and a strong tennis player may quickly pick up table tennis or badminton.
The simple point behind this long introduction is whether parameters learned by a network optimized for one task can be applied to a “similar but slightly different task.”

A network trained on general ImageNet images can extract meaningful image features with its learned filters. To apply this network to classification as a more specialized downstream task, we can modify the parameters of the existing feature extractor. Its pretrained representations can provide a more stable starting point than training from scratch. Another option uses a support vector machine (SVM): preserve the existing feature extractor and train a task-specific head at the end. Strategies for transfer learning or fine-tuning vary with the relationship between the existing task and the new task. Dataset size is another important criterion.
The reference network used in this way is called a “pretrained model.” Accessing and using a pretrained model can broadly be divided into three steps:

• Select source model: Choose the architecture most applicable to the problem from the available networks. Platforms such as Papers with Code now provide pretrained networks released by many research organizations, including Facebook AI and Google, while PyTorch and TensorFlow ship with their own pretrained models. The pool of available networks has become extensive.
• Reuse model: Use the selected pretrained network as the basis of training. In a sense, the pretrained network becomes the “initialization” for applying the model to a new task rather than starting entirely from scratch. The training strategy depends on whether the modeling approach uses part or all of the network.
• Tune model: When necessary, tune the input-output relationship of the network for the new task of interest. Replacing the classifier with an SVM or training at a low learning rate are examples of this stage.

When training a network on a new task, we must distinguish which parts and methods are to be trained. In terms of whether gradients optimize each layer, the options are

• Re-train: Discard all previously learned weights and replace them with newly learned ones.
• Fine-tune: Keep the learned weights as the starting point and adjust their parameters using the training data.
• Freeze: Preserve the learned weights exactly, including throughout training.

Fine-tuning generally uses a learning rate one-tenth or one-hundredth of the value used for the original training. The learning rate determines how much each weight is updated in response to the error between predictions and ground truth. Reducing it helps preserve the existing weights to some degree.

Transfer learning by dataset similarity and size

As discussed above, after selecting a pretrained model we can retrain certain layers from scratch, fine-tune their weights with a low learning rate, or leave them frozen. The strategy can be divided into four cases according to the similarity between the pretrained and new tasks—the dataset similarity—and the size of the new dataset used for training.

Large dataset, low similarity to the existing task (Quadrant 1)

In this case, training the full network may improve performance more than preserving representations from the existing task—the implicit mapping encoded in its learned weights. The assumption is that the dataset is large enough to train all network parameters without overfitting, while weights optimized for the old task are unlikely to transfer well to a dissimilar new task.

Large dataset, high similarity to the existing task (Quadrant 2)

This is the most favorable situation, although it presents a dilemma. If the dataset is sufficiently large, training from scratch poses no major problem, just as in Quadrant 1. But if the representations in the existing parameters provide useful supervision for the new task, the optimizer can find rapidly converging parameters more quickly, without falling into a poor local optimum early or wasting effort on meaningless exploration. Training in this case usually fine-tunes or retrains the lowest layers—the later feature-extraction layers and classifier—and freezes the other layers to benefit from their learned parameters.

Small dataset, low similarity to the existing task (Quadrant 3)

This is the most difficult of the four quadrants. There is no single clearly correct solution: the dataset is too small to generalize easily to the new task without overfitting. As in Quadrant 2, usually only selected layers are trained, but the reason for freezing differs.
In Quadrant 2, selected layers are trained to increase optimization speed or improve performance by using the learned weights. In Quadrant 3, training every layer to make the network specific to the new dataset forces the whole network to learn from a relatively small dataset, creating a risk of overfitting.

Small dataset, high similarity to the existing task (Quadrant 4)

As in Quadrant 3, this case requires attention to overfitting during training. Although the dataset is small, when the pretrained network’s representation mapping appears applicable to the novel task—a similar task—there is no need to train anything beyond the classifier. The classifier generally has relatively few parameters and can be fine-tuned in a task-specific way, making it easier to train on a small dataset.

Knowledge distillation

The next subject is knowledge distillation. Transfer learning uses an existing network to tackle a new task. Knowledge distillation is slightly different: a high-performing pretrained model optimized for one task—the teacher network—helps train a shallower, lighter-weight student network. Knowledge distillation uses a hyperparameter called temperature ($T$) to construct higher-entropy soft labels that are more informative than one-hot labels. Training with these soft labels rather than hard labels can improve the optimization performance of a lightweight model.

In other words, the student network learns to follow the output of the more capable teacher network. The method is broadly applicable to any kind of deep network, not only convolutional neural networks. It can be used in Transformer-based approaches such as DeiT. That paper applies knowledge distillation by transferring learned knowledge from a CNN while training a Transformer, whose weaker inductive bias gives it poorer inference ability than a CNN when little data is available.
The optimization loss is as follows. Given a training dataset and hard labels $(X,~Y)$, a student prediction $S(\cdot, T)$ at temperature $T$, and teacher network $R(\cdot, T)$,

$$\mathcal{L} = T^2 \lambda \mathcal{L_t} + (1-\lambda) \mathcal{L_s}$$

where the student and teacher loss terms are

$$\mathcal{L_s} = \mathcal{L}_{CE}(Y, S(X, 1))$$ $$\mathcal{L_t} = \mathcal{L}_{CE}(S(X, T),~R(X, T))$$

The temperature-based prediction turns a hard label into a soft label by replacing ordinary softmax with temperature-normalized softmax:

$$q_i = \frac{exp(z_i/T)}{\sum_j exp(z_i/T)}$$

Here $z_i$ is the logit predicted by the network.

Why knowledge distillation?

Why does knowledge distillation help train a network? In ordinary classification, it may seem that aggressively training against the original hard labels should ultimately produce better performance.
Before explaining the effect of soft targets, let us revisit why knowledge distillation was proposed. We want to use deep learning to build optimized networks that are practical even in mobile or embedded environments. Applying deep learning across edge devices such as phones, cameras, and game consoles requires lightweight models. The earlier post on MobileNet provides useful background. Many approaches for lightening networks were introduced, but simply applying the training recipe of a large, high-performing network to a small one often fails to achieve good performance.

Attribute-based soft labels

List the attributes of a cat in a photo: unless it is a Sphynx, it is furry and cute; the cat in the example above is wearing a “hat” and has large, bright eyes. Many cat photos contain some of these fine-grained attributes. Discarding them all and training a low-parameter network with a simple all-or-nothing target can impair its ability to generalize. The teacher network’s predictions are therefore smoothed so that the logits assigned to other classes by a well-trained network can also contribute to learning. Cross-entropy with a one-hot label normally ignores predictions for the other classes; knowledge distillation asks, can we use them too? From another perspective, the soft target represents the network’s prior knowledge—the output of the pretrained network’s implicit function—so $S(X, T) \rightarrow R(X, T)$.

Using a pretrained model

In conclusion, both transfer learning and knowledge distillation use a pretrained network. The difference is that transfer learning applies a single network to a different task, whereas knowledge distillation uses two networks—a deep network and a shallow network—to improve performance and assist training on one task.

Continual learning

Suppose we have a well-trained autonomous-driving model, but its training dataset contained no examples of “a water deer suddenly appearing” or “a kangaroo suddenly appearing.” In other words, abnormal situations the vehicle must handle do not include an animal unexpectedly entering the road.
We need to update the in-vehicle system to train on this additional dataset. One might think that simply adding the new data to the existing network’s training would suffice. But because the abnormal case is a special situation, even a pretrained network provides no guarantee that its representations will perform satisfactorily, and the amount of data available for training will certainly be limited.
We want the existing deep-learning model to handle the new task—vehicle control for suddenly appearing animals—without degrading its original performance in ordinary driving. This is the starting point of continual learning.
Successful continual learning should meet the following requirements:

• The network should perform well on the new task.
• The network should preserve performance on the existing task.
• The network should not require the old task’s dataset, providing memory efficiency.
• Training and inference should have reasonable time complexity (FLOPs).
• The network should have reasonable storage complexity—the scope of its trainable representations.

Regularization-based methods

Let us examine several continual-learning methods and approaches. Recall transfer learning, which uses a “pretrained representation.” If a training method applies to a new task without substantially changing the existing network’s parameters, it may preserve performance on the old task while optimizing for the new task. This is called a regularization-based method. The name comes from preserving old-task performance by regularizing a complex model with a low learning rate. Fine-tuning is a representative example.

Rehearsal-based methods

Another approach records samples of the old task’s data in a buffer and uses them alongside the dataset for the current task. This continues to provide supervision from the old task while optimizing for the new one. Learning without Forgetting (LWF) is a representative method.

Architecture-based models

A third method relies on the training architecture. To learn a new task while perfectly preserving performance on an existing task, we might simply expand the network or train with auxiliary heads. This resembles the rehearsal-based approach, but its main objective is to adapt the network structure to multiple tasks, so it may also require the old task’s dataset. Multitask learning is a representative example.

Feature extraction

We have briefly reviewed several approaches to continual learning. Feature extraction, the simplest baseline underlying these methods, is straightforward. First, let us define the notation.

• $\theta_s$: Shared parameters
• $\theta_o$: Task-specific weights for previously trained, or old, tasks
• $\theta_n$: Randomly initialized task-specific parameters for new tasks

Feature extraction preserves the shared parameters from the old task—usually all convolutional layers—and attaches a new head trained on the new task. In the terminology of transfer learning, the convolutional network used as the feature extractor ($\theta_s$) is frozen while the new-task parameters ($\theta_n$) are optimized. Since the heads differ, this does not affect the old-task framework ($\theta_s + \theta_o$), preserving its old-task performance. The disadvantage is that the feature extractor cannot be optimized for the new task, limiting new-task gains.

Fine-tuning

As the name suggests, fine-tuning uses a small learning rate to make small optimizations to existing parameters. It can optimize more parameters than feature extraction and therefore improve performance on the new task further. However, if the new task differs from the old one, performance on the old task can decline sharply. Feature extraction freezes all parameters $\theta_s + \theta_o$. Fine-tuning freezes only $\theta_s(:k) + \theta_o$, where $k$ is the first convolutional-layer index to be fine-tuned, so all remaining weights may change. Of course, fine-tuning only the fully connected layer is equivalent to the feature-extraction method above.

Multitask learning (MTL)

Multitask learning can be used when datasets are available for every task to be trained. Unlike the preceding methods, it jointly optimizes $\theta_s,~\theta_o,$ and $\theta_n$. The feature extractor is generally shared, while separate heads are adapted to each task.

The objective of multitask learning is to combine related, domain-specific information from several training tasks to improve single-task generalization. In autonomous driving, for example, the primary task is deciding which way to turn the steering wheel in each situation. Reading roadside signs and signals, observing traffic, and deciding whether to press the accelerator or brake are auxiliary tasks. When auxiliary and primary tasks are trained together, L2 regularization keeps their weights from diverging excessively. Beyond improving one task, the structure can raise performance across all auxiliary and main tasks simultaneously.
The variants of network sharing available to MTL are as follows. When training to generalize across multiple tasks, the feature-extraction parameters do not necessarily have to be shared, as the figure below illustrates.

The arrangement above does share every parameter and is called hard parameter sharing. Since one feature-extraction network learns across every task, it has less risk of overfitting. It also uses $N$ times fewer parameters than maintaining a separate parameter set for every task, providing a corresponding regularization effect. Another arrangement gives each task different network parameters, as in the next figure.

Unlike the preceding design, Tasks A, B, and C pass through separate networks. A regularization term called cross-talk, implemented through the L2 regularization described above, keeps their weights similar. Each task follows a separate optimization process overall, but the parameters of corresponding layers remain similar.

There is also a cross-stitch method in which activation maps from different tasks intersect during training. As the paper’s name suggests, the two activation maps are crossed and multiplied by the input activation maps.

$$\begin{bmatrix} \tilde{x_A}^{ij} \\ \tilde{x_B}^{ij} \end{bmatrix} = \begin{bmatrix} \alpha_{AA} & \alpha_{AB} \\ \alpha_{BA} & \alpha_{BB} \end{bmatrix} \begin{bmatrix} x_A^{ij} \\ x_B^{ij} \end{bmatrix}$$

This form of MTL is memory intensive, because every network and dataset must be loaded on the GPU simultaneously for training. It also offers the benefit of optimizing multiple tasks at once.
The multitask-learning loss sums the losses of all tasks. Easier tasks tend to produce smaller gradients, however, which causes a training imbalance. A weighted-sum loss is used to control this appropriately.

$$L_{total} = \sum_i w_i(t) L_i$$

Here the $t$ term denotes the epoch. Since tasks converge at different speeds, it describes a loss whose weights are allocated dynamically. A simple weighted loss can be viewed as a specialized version of this loss.

Learning without Forgetting (LWF)

What if training every dataset simultaneously is impractical, but we still want to preserve old-task representations? LWF, or Learning without Forgetting, was proposed from this perspective. It adapts better to a new task than feature extraction or fine-tuning while retaining performance on the old task. MTL is slow because it must continue optimizing the existing dataset as well; LWF avoids that requirement and therefore trains faster. Above all, its training framework is comparatively simple.
Dividing the training procedure into stages gives the following:

• Pre-train network $F_{\theta_s,\theta_o}$ with the old task. $\theta_s$ is a parameter trainable in the shared network, and $\theta_o$ is a parameter trainable in the classifier specialized for the old task.
• Generate soft label $y'$ from pretrained network $F_{\hat{\theta_s},\hat{\theta_o}}(x_n)$. It can be generated as $y_o' = \frac{(y_o^{(i)})^{1/T}}{\sum_j (y_o^{(j)})^{1/T}}$ with $y_o^{(i)} = F_{\hat{\theta_s},\hat{\theta_o}}(x_o^{(i)})$.
• Freeze $\hat{\theta_s},~\hat{\theta_o}$ and train new-task classifier $\theta_n$ with new-task supervision $(x^i_n,~y^i_n),~i=1,~\cdots,~l$ and cross-entropy criterion $CE(y_n, F_{\theta_s,\theta_n}(x_n))$.
• Jointly train $\theta_s,~\theta_o,~\theta_n$ at a low learning rate with old-task soft-label supervision and new-task supervision. The training criterion is $\lambda_o\mathcal{L_{old}}(y_o', \hat{y_o'})+\mathcal{L_{new}}(y_n, \hat{y_n})+\lambda_r \mathcal{L}_2$.

In simpler terms, suppose we have a network pretrained on an old task. We use it to extract soft labels for the new task dataset $x_n$. Those soft labels preserve the representations of the network optimized for the old task. During training, we compute both the loss between the soft labels and predictions and the loss between the actual ground truth and predictions, then optimize their weighted sum. L2 regularization helps the weights train stably.

Limitations of each approach

We have discussed three broad approaches. First, fine-tuning as a regularization method optimizes the new task effectively, but almost entirely forgets performance on the old task. In particular, it cannot solve a complex multitask problem.
Second, the architecture-based MTL (Multi-Task Learning) approach requires task identity during training. This limits it in class-incremental settings, where the number of classes to distinguish grows, and in task-agnostic settings, where performance falls when jointly trained tasks differ. Memory also remains a persistent issue.
Finally, the rehearsal-based method introduced as LWF cannot operate with a tiny buffer—it needs old-representation labels for the new task—and cannot be used when the dataset has security or privacy constraints.

Self-supervised learning

Think back to studying your major. If you are taking circuits or electromagnetics and an example problem has no solution, you immediately start searching Chegg and Google. Sometimes no answer can be found at all; on other occasions the results are full of strange answers that are not the one you need. Surely that is not just me.

Unless we can recognize that the answer is wrong or solve the problem ourselves, we will learn a nonsensical answer—or never learn the answer and bomb the exam.
Deep learning is similar. Every problem defined so far assumed a desired output for each input. If we have a large collection of images and a person must classify and label every one, the work is harder than it sounds. We build AI to make life easier, yet create the irony that humans must labor to train it.

Have you ever bought clothing from Musinsa? Among online fashion retailers, it is not only a sanctuary for university students but a platform used by people of all ages. Customers can leave reviews of their purchases. Full-body photos are required, and depending on the review conditions the company must identify submissions ineligible for reward points. Checking every review by hand is laborious, but labeling every example to train a deep-learning model is also burdensome. The Musinsa Data Solution team wrote an article on this subject, which I recommend if you are interested.
The point of this sudden mention was not to advertise Musinsa, but to show that obtaining supervision for deep learning is often difficult in industry. Self-supervised learning was proposed to learn useful representations of a modality without human intervention. We need to train a model to extract useful features from images without labels, using only the image rather than the usual (image, label) pair. The proxy objective is called a pretext task, and its learned representations can then support many downstream tasks.

Examples of pretext tasks

The simplest method rotates an image as augmentation and trains the model to predict its rotation angle. Recognizing the angle requires understanding the object or using image features; perhaps training this way can produce useful representations for arbitrary images.

Another task predicts patch locations like a jigsaw puzzle. Choose one random image patch, define a $3 \times 3$ grid around it, and take the other eight patch positions. The model learns representations while solving the task of predicting the locations of those surrounding patches.

What is contrastive learning?

Some self-supervised-learning papers use non-contrastive approaches such as clustering.

Instead of obtaining supervision, these algorithms solve the task from relationships between data points. Since that approach has limitations, the remainder of this post focuses on self-supervised learning through contrastive methods.

The idea of contrastive learning is simple: bring similar samples together and push different samples apart. The figure above makes this intuitive. Samples of the same class—a positive pair—are pulled closer, while samples of different classes—a negative pair—are pushed apart. Contrastive learning therefore defines the concepts of positive, anchor, and negative. An anchor is a sample query. Given one image of a bird, for example, another image of a bird is a positive, while an airplane image outside the class is a negative. A positive pair is an (anchor, positive) pair, and a negative pair is an (anchor, negative) pair. As everyone knows, however, self-supervised learning does not know the label of each sample. Constructing contrastive image pairs seems to require knowledge or supervision about whether they are similar or different, which creates another contradiction. The solution is as follows.
Given $N$ images, treat the $N$ different images as $N$ separate classes. Triplets—positive and negative pairs—can then be constructed for every sample.

The figure shows a triplet. We can construct its three elements—negative, positive, and anchor—for every sample. A negative pair can simply combine different images, but how do we construct the positive pair?

Data-augmentation pipeline

This is where data augmentation becomes useful. An augmented view of an image differs from the original but contains information about the same object, so their encoded representations should be similar.

The well-known SimCLR paper describes the arrangement above. A stochastic augmentation module $A(x)$, or $T$, sits in the middle, and the differently augmented images $x_i$ and $x_j$ are trained to produce similar encodings. SimCLR argues that training the encoder itself to align object pose and alignment restricts the diversity of samples available for representation learning. It therefore introduces an additional linear layer—a representation-extraction projection—and trains its outputs $z_i,~z_j$ to be similar. In the figure, using similarity between $h_i,~h_j$ as the metric would make the preceding encoder behave like a spatial transformer network, making it harder to generalize to downstream tasks. The representation-extraction component at the end can instead be viewed as a kind of affine filter, like an STN.

Contrastive Predictive Coding (CPC) uses the augmentation shown above. After applying familiar transformations such as color filtering, random grayscale, and random flipping, it divides the image into overlapping subpatches. One patch becomes the anchor; another patch from the same image forms the positive pair; and a patch generated from a different image forms the negative pair.

Most other well-known papers—including AMDIM, SimCLR, and MoCo—use augmentation like that shown above. They likewise apply conventional transformations such as jitter and flipping. Rather than dividing the image into patches, however, two different augmentations of the same image form a positive pair, while augmented views of two different images form a negative pair.

Every image in the pairs described above is mapped into latent space, and the resulting feature maps are used to optimize a contrastive loss.

CPC is described as predicting the future in latent space. It treats image patches as a timeline, ordered from top-left to bottom-right, and models them autoregressively.

AMDIM generates two augmented samples and compares their feature maps from the same encoder at different feature levels. It therefore has the advantage of making comparisons across spatial scales.

Loss functions with similarity measures

There are several ways to measure similarity between multidimensional vectors such as latent representations and feature maps. One can use cosine similarity or an inner product. Since cosine similarity is a normalized inner product, the two metrics can be treated similarly.

$$\begin{aligned} \cos \theta =& \frac{\vec{a}\cdot \vec{b}}{\parallel \vec{a} \parallel \parallel \vec{b} \parallel} \\ \parallel \vec{a} \parallel =& \sqrt{a_1^2 + a_2^2 + \cdots + a_n^2} \\ \parallel \vec{b} \parallel =& \sqrt{b_1^2 + b_2^2 + \cdots + b_n^2} \end{aligned}$$

The resulting similarity can be interpreted, over a particular range, as something like the probability that the two vectors are similar, and optimized directly through negative log likelihood. The resulting expression is the Noise Contrastive Estimation loss, or NCE loss.

$$NCE_{Loss} = -\log \left( \frac{\exp (sim(g(x), g(x^+)))}{\exp (sim(g(x), g(x^+))) + \sum_{k=1}^K \exp (sim(g(x), g(x^-_k)))} \right)$$

The learned representations are then fine-tuned for downstream tasks such as classification and detection, as shown above. SimCLR, however, uses NT-Xent—Normalized Temperature-Scaled Cross-Entropy Loss—rather than the NCE loss above.

$$1_{i,j}= -\log \frac{\exp (sim(z_i, z_j)/ \tau )}{ \sum^{2N}_{k=1} 1_{(k \neq i)}\exp (sim(z_i, z_j) / \tau)}$$

SimCLR also reports that random cropping plus random jitter substantially improves representation learning in self-supervised learning. As discussed above, other helpful scaling strategies include spatial transformation through nonlinear projection, increasing the number of negative samples in a batch, using a larger network, and training for more epochs.