The Evolution of Convolutional Neural Networks

Most network architectures used in conventional deep learning were based primarily on multilayer perceptrons (MLPs) or convolutional neural networks (CNNs). Unlike an MLP, which simply stretches a modality into a $1 \times N$-dimensional vector for computation, convolutional neural network architectures proved their performance on modalities such as images, and a wide range of research followed. Convolutional neural networks had many advantages over MLPs. First, they required fewer parameters to train when constructing a deep neural network with the same number of hidden layers as an MLP, and despite having fewer trainable parameters, the representations they could learn generalized well with respect to the modality.
Thanks to this performance, research in NLP (natural language processing) and audio-related deep learning also began using CNN architectures, and this can be attributed to "inductive bias." Inductive bias is the set of all "assumptions" that can be supplied during training or inference so that a model can make estimates about modalities it has not observed. To illustrate what this means, suppose a deep learning network has learned to classify cats without a dataset containing cats "looking to the left." Inductive bias can be viewed as a kind of constraint that enables the network to recognize that the object is a cat when it is given an image of a cat "looking to the left" during inference. Of course, this example is somewhat imperfect, but because the term inductive bias is essential to understanding the concept behind the transformer model, I wanted to establish a simple understanding of it first.

The figure above provides a simple way to explain inductive bias. Looking at how convolution is performed, a filter multiplies all of its parameters by the feature values within a particular region (for example, $3 \times 3$) and then sums the results. We can therefore make the following two assumptions.

• For an object in an image, the relative positions of the pixels representing that object are preserved (localization).
• When an object in an image moves, the feature output for that object also moves within the image (translation equivariance).

The first condition does not cause a major problem, but the second one does. Translation equivariance was a problem that MLPs could not handle either, and if changing only the position of the same object changes the form of the feature map, there is no guarantee that the same prediction will be made for that object. Unlike an MLP, however, a CNN does not alter the object's shape; only its overall localization in the feature map changes while its relative positions are preserved.
Yet it was found that a CNN could produce the same prediction from a feature map with preserved localization by using filtering modules such as max pooling, which reduce feature values within the same kernel to a single value, together with probability calculations through softmax. In other words, the feature extraction component is the process of retaining a meaningful feature form for an object based on translation equivariance and localization, while the classifier component uses translation invariance to make consistent predictions from the extracted feature map.

These very advantages allowed convolutional neural networks to learn feature representations for the same class with fewer parameters, which led to an enormous body of research following their first success on ImageNet. Above all, CNN architectures began to be used widely to extract temporal information from sequential data in NLP, audio, video, and other domains.

This naturally led to the emergence of RNNs (recurrent neural networks) for processing sequential data, along with efforts to learn the global contextual meaning between inputs and outputs by using long-term memory modules such as LSTMs and GRUs.

RNNs in Machine Translation and Their Limitations

Suppose, for example, that there is a machine translation task that uses machine learning or deep learning to translate a Korean sentence meaning "I decided to name my cat Epoch" into the English sentence "I decided to name my cat Epoch." If we approach this task simply from the perspective of an RNN, it can be expressed as follows:

Here, $E_\theta, D_\phi$ denote the encoder and decoder, respectively. The sequence-to-sequence model is a representative RNN with this kind of encoder-decoder architecture.

The first and largest problem is computation speed. Deep learning could use deep stacks of layers while still computing quickly because tensor operations could be processed in parallel on GPUs. Since feature-map values at the same level are independent, all convolution operations can be carried out simultaneously, without having to proceed sequentially, which provides an architectural advantage. An RNN, however, cannot do this. As its structure makes clear, each LSTM calculation (an LSTM can be regarded as a hidden layer) requires the result of the preceding LSTM. Even if parallel processing accelerates the computation within a single LSTM, the sentence length $N$ still creates a bottleneck. The second problem is that the context vector used during computation has a fixed size. This also becomes a serious problem as a sentence grows longer, because in an RNN architecture the only feature embedding that the translation process can refer to is the output of the encoder's final hidden layer. Consequently, with complex sentences or difficult translations, the encoder becomes less able to properly reference the sentence as its length grows. The attention mechanism emerged directly from this problem and became a foundational technology for transformers.

Attention Mechanism

As mentioned above, RNNs have two major problems. First, computation is too slow for long sentences. Second, as a sentence grows longer, performance is constrained by the fixed length of the context vector. Solving these problems required reasoning over the outputs of every LSTM composing the RNN, which led to the proposal of a module called "attention." An attention module extracts a tensor whose weights express how strongly the inputs—whether a single input or multiple inputs—are related to one another. Consider, for example, the following sentence.

$$\text{I decided to name my cat Epoch}$$

After tokenizing it word by word,

$$\text{(I, decided, to, name, my, cat, Epoch)}$$

we map every token to a value according to the embedding scheme, and then try to measure the similarity between the word "name" and the other words in the same sentence. We can first infer that the word most closely related to "name" is "Epoch," and that the next most important word may be "cat."

$$\text{(I, decided, to, name, my, cat, Epoch)} \rightarrow (0, 0, 0, 0.3, 0, 0.1, 0.6)$$

This is an extreme simplification, but the idea is to map the similarity between each input embedding to a weight and use the attention value obtained from the softmax probabilities of those weights. Attention makes it possible to capture global correlations or long-range interactions across an entire sentence without relying on a long-term module such as an LSTM.
Yet even sequence-to-sequence models with attention leave one problem unresolved: how to accelerate the RNN's "sequential computation" for extracting the feature vector used by the decoder. An unavoidable trade-off between training performance and inference time remains. We can now finally introduce the transformer, which proposed a new paradigm in which meaningful machine translation could be performed without relying on an RNN architecture whose bottleneck could not be eliminated.

Attention Is All You Need!

Transformer appeared with this title and made an enormous splash. The paper's main idea was that attention already enables immediate global reasoning without having to extract global context through a recurrent architecture such as an RNN, so perhaps a deep neural network that performs this attention operation repeatedly would actually be better suited to tasks such as machine translation. It demonstrated its performance while reducing computation time relative to existing RNNs, and this architectural change became a cornerstone for changes throughout NLP and many other fields of research.

We will now examine self-attention, multi-head attention, positional encoding, and masked attention, which are among the key elements of the transformer architecture.

Self-Attention

In an RNN, a feature vector is extracted through an encoder architecture. The RNN architecture assumes that this feature vector is learned to contain context for all input tokens; the conventional approach additionally considers prior RNN feature vectors through attention values. A transformer, which uses only the attention mechanism, removes this dependency and extracts contextual embeddings simply by applying attention repeatedly within the same sentence. Let us first review the terminology used in attention.

• Query ($Q$): Literally means a "question" or "inquiry." In the process of inferring how each token in a sentence relates to the other tokens, it refers to "each token" under consideration. For example, it is the word "name" when, as in the example above, we try to determine its similarity to the words in the same sentence (I decided to name my cat Epoch).

• Key ($K$): Operates as a pair with a query and serves as the item that can supply an answer when the query asks for the value of a particular key. For example, these are the words in the sentence (I, decided, cat, and so on) when we try to determine their similarity to "name" in the example above.

• Value ($V$): The value corresponding to a key. Rather than trying to understand query, key, and value separately, it is much simpler to think about the relationship among all three.

Suppose there is an input sentence $S$, which an embedding function $\mathcal{E}$ converts into an embedding tensor $X$. For this embedding tensor, the query, key, and value are linearly projected according to their respective weights.

$$\begin{aligned} Q =& W_1X \newline K =& W_2X \newline V =& W_3X \end{aligned}$$

For example, suppose a sentence is divided into a total of $n$ tokens, each of which is replaced by a tensor of dimension $e$. For dimension $d$ of $Q,~K,~V$,

$$\begin{aligned} X \in & \mathbb{R}^{n \times e} \newline K \in & \mathbb{R}^{n \times d},~W_1 \in \mathbb{R}^{e \times d} \newline Q \in & \mathbb{R}^{n \times d},~W_2 \in \mathbb{R}^{e \times d} \newline V \in & \mathbb{R}^{n \times d},~W_3 \in \mathbb{R}^{e \times d} \end{aligned}$$

This can be represented as above. Among the various ways to calculate attention values, the scaled dot-product procedure used in the transformer works as follows.

First, calculate scores between different input embeddings: $S = Q \cdot K^\top$. Each row vector of $Q$ is the value for a token embedding, and each column vector of $K^\top$ is the key for a token embedding. Taking their inner products can be expressed as follows for row vectors $r^q_i,~r^k_i \in \mathbb{R}^d$ $(i = 1,~2,~\cdots,~n)$ of $Q,~K$.

$$QK^\top = \begin{bmatrix} r^q_1 \newline r^q_2 \newline \vdots \newline r^q_n \end{bmatrix} \cdot \begin{bmatrix} {r^k_1}^\top & {r^k_2}^\top & \cdots & {r^k_n}^\top \end{bmatrix}$$

The resulting values consist of inner products between row vectors, and their magnitude tends to increase as the row-vector dimension $d$ grows.

$$S = QK^\top = \begin{bmatrix} {r^k_1}^\top r^q_1 & {r^k_2}^\top r^q_1 & \cdots & {r^k_n}^\top r^q_1 \newline \vdots & \vdots & \ddots & \vdots \newline {r^k_1}^\top r^q_n & {r^k_2}^\top r^q_n & \cdots & {r^k_n}^\top r^q_n \end{bmatrix}$$

For stable training (that is, to keep the gradients properly scaled), divide the scores above by the square root of the dimension.

$$S_n = \frac{QK^\top}{\sqrt{d}} = \begin{bmatrix} ({r^k_1}^\top r^q_1)/\sqrt{d} & ({r^k_2}^\top r^q_1)/\sqrt{d} & \cdots & ({r^k_n}^\top r^q_1)/\sqrt{d} \newline \vdots & \vdots & \ddots & \vdots \newline ({r^k_1}^\top r^q_n)/\sqrt{d} & ({r^k_2}^\top r^q_n)/\sqrt{d} & \cdots & ({r^k_n}^\top r^q_n)/\sqrt{d} \end{bmatrix}$$

Apply softmax to convert the resulting scores into probabilities. The softmax function uses the exponential function to normalize values into the range $0 \sim 1$ and, most importantly, ensures that they sum to $1$ along a given dimension.

$$\begin{aligned} \text{Let }z_{ij} =& ({r^k\_j}^\top r^q_i)/\sqrt{d}, \newline softmax(z_{ij}) =& \frac{e^{z_{ij}}}{\sum\_{i=1}^n e^{z_{ij}}} \newline P =& softmax(S_n) \end{aligned}$$

Finally, multiplying this by the values corresponding to the keys gives the weighted value matrix we want.

$$Z = V\cdot softmax(\frac{Q \cdot K^\top}{\sqrt{d}})$$

Masked Attention in the Decoder

The attention operation is almost identical to what was described above. When the decoder attends to the encoder, however, the encoder's results are treated as the keys and values, while the decoder's output is used as the query. During training, because the entire tokenized sentence is provided as input unlike in an RNN, a causality problem arises. The problem is as follows.
When the decoder emits a translated result for an input sentence, it follows a recurrent structure. This is the same for both an RNN-based sequence-to-sequence model and a transformer. Therefore, even if the "translation result" is known during training, it must not be used improperly.

Consider this figure. Its task depicts the process of translating the sentence "I want to buy a car" into another language. The decoder first predicts the first word ("Ich") based on BOS (a token meaning Beginning of Sequence) and the encoder output. It then predicts the second word ("will") from the predicted "Ich" and the encoder output. Next, it predicts the third word ("ein") from the predicted "Ich, will" and the encoder output. The process continues in this fashion.
Although the paper does not state it here, what matters in actual training is that the ground-truth word is used rather than the word predicted by the model; in natural-language research this is called teacher forcing. For example, if the translation result is already known, there is no need to use the words predicted up to position $n-1$ when predicting the $n$th word; instead, the already known $n-1$ ground-truth tokens can be used. This approach can make attention training on the encoder side more stable. Early in training, before the model has converged, the decoder will produce irrelevant words, and using them directly as attention queries for decoder prediction would make training unstable.

Of course, the fact that the word currently being predicted cannot refer to a word that will be predicted later always holds during "inference" (testing). During training, however, the full translated result is already known and is used as supervision, so the decoder must be prevented from referring to later words during training.

The computation is summarized above. Starting from the top row, the diagram shows attention over the decoder input tokens. The shaded cells in the mask have value $1$ and the unshaded cells have value $0$, so the $n$th embedding can refer only to attention weights at positions less than or equal to $n$.

Multi-Head Attention

The concept of an "ensemble" in machine learning and the number of kernel channels used in deep convolutional neural networks serve similar functions: they allow the model to learn multiple representations of the same feature map. CNN architectures already have a property called network width that can provide this capacity, but a transformer faces the problem that an attention layer cannot itself have multiple channels. Multi-head attention was introduced to solve this problem.

The concept is quite simple: increase the number of linear layers that can perform attention to match the number of heads $h$. The model calculates attention weights with multiple heads, concatenates the resulting values along the head axis, and uses a linear operation to restore the original dimension. Using the same notation as above, for query, key, and value linear operators $W_h^Q,~W_h^K,~W_h^V$ at head index $h$, and multi-head linear operator $W_o \in \mathbb{R}^{hd \times d}$,

$$\begin{aligned} Q_h =& W_h^QX,~K_h = W_h^KX,~V_h = W_h^VX, \newline Z_h =& V_h\cdot softmax(\frac{Q_h \cdot K_h^\top}{\sqrt{d}}) \in \mathbb{R}^{n \times d}, \newline Z_{concat} =& \text{Concat}(Z_1;~Z_2;~\cdots;~Z_h) \in \mathbb{R}^{n \times hd}, \newline Z_{output} =& W_o \cdot Z_{concat} \in \mathbb{R}^{n \times d} \end{aligned}$$

the structure is calculated as shown above.

Positional Encoding

The causality issue absent from RNN architectures still exists here. An even greater problem, however, is that the distance between embeddings also contributes to interpreting context. An RNN that processes each token sequentially does not need to worry about this, but a transformer needs a way to impose that constraint. Consider the following sentence, for example:

$$\text{I have a dog named Adam and my friend has a cat named Epoch}$$

In other words, I have a dog named "Adam", and my friend has a cat named "Epoch." The important point is the distance within the sentence between the subjects who own the dog and the cat. If the model learns the relationship between a particular name and an animal regardless of token position, the following disaster could occur.

"I have a dog named Adam and my friend has a cat named Epoch"
"I have a dog named Adam and my friend has a cat named Epoch"

If the bold portions receive high attention values with one another, the names of my dog and my friend's cat may be swapped. The translation might even say that my friend owns the dog and I own the cat. Therefore, to provide some inductive bias during training—similar to the properties in CNNs that objects occupy nearby pixel positions or are invariant to translation—a method was devised that adds an embedding based on each token's position.

The positional encoding uses the familiar sinusoidal functions.

$$\begin{aligned} PE_{(pos,~2i)} =& \sin (pos/10000^{2i/d_{model}}) \newline PE_{(pos,~2i+1)} =& \cos (pos/10000^{2i/d_{model}}) \end{aligned}$$

Here, $i$ is the axis along the embedding dimension, and $pos$ represents the position of each token. The mottled pattern on the left is an actual visualization of this encoding. As the equations show, $d_{model}$ is the same as the embedding dimension $d$ mentioned earlier, and as $i$ increases, the frequency of the sinusoidal function gradually decreases. As can be seen from a periodic function, if we follow the y-axis and inspect the function values, the indices with the same function value are identical regardless of the reference position. In other words, the y-axis distance between one dark-blue value and another dark-blue value (the difference in pos), or the y-axis distance between one dark-red value and another dark-red value (the difference in pos), remains constant. The functions above can also add a different encoding at every embedding position. Their values repeat because the functions are periodic, but no two encoded embeddings (the values parallel to the x-axis) are identical; each is unique. Thus, regardless of how long the sentence becomes, every token can be mapped to a unique value, making it possible to distinguish every token regardless of sentence length.

All that remains is to generate the encoding for each token at the appropriate dimension and add it to every embedding.

Training a Transformer

Because supervision is available, training is straightforward and can optimize cross-entropy loss. The model maps the index with the maximum value in its softmax output to a word, and it is trained so that the probability assigned to the correct word approaches 1.

What About Today?

It is no exaggeration to say that transformers are among the most actively researched neural networks today, because they can be applied effectively across such a wide variety of fields. Since the transformer was first proposed for machine translation in 2017, many NLP-related models such as BERT and GPT-3 have appeared. In particular, the emergence of the vision transformer in computer vision made a multimodal approach possible. The idea is that if every modality can be encoded with a consistent architecture, multiple modalities can be used together as supervision on that basis.

Vision Transformer

We previously examined the architecture and training method of the transformer in machine translation. Although the results are not included separately here, transformers performed extremely well in practice, and numerous NLP technologies subsequently advanced through a variety of transformer-based approaches. After the "Attention Is All You Need" paper, researchers in vision also began actively investigating whether the transformer architecture used in NLP could be applied there.
Unlike natural language or audio, however, which can be tokenized relatively clearly (for example, by words or syllables), images consist of more continuous signals. This made it unclear exactly how an image should be tokenized, and conventional embedding techniques such as word2vec could not be used.

The first paper to demonstrate the performance of a vision transformer in this setting was ViT: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. A brief look at the paper reveals a remarkably simple approach.

The image is divided into a total of $n$ patches. As the paper's title indicates, if an image has size $256 \times 256$, it is divided evenly into patches of size $16 \times 16$. To illustrate this more simply, the figure above divides $H,~W$ into three parts each to create nine samples. Each separated patch is flattened into a one-dimensional tensor and sent into the embedding space through a linear projection. A class token is added at the beginning of this patch sequence (remember it as a reference point for global inference when determining an image's class). As with the transformer described earlier, a positional embedding is also added. Because the sinusoidal embedding used for natural-language embeddings does not have the same effect for images, a learnable positional embedding is used instead. Once all attention operations have finished, the class token is taken from the encoder output. Through repeated attention operations, this class token has now completed a global reference to the 16 patch embeddings. The class token has size $B \times D$ and, rather than treating it as immediately usable for class prediction, it helps to think of it as "summarizing information from the other patches" as attention proceeds. After the class embedding has been extracted, the following operation is performed.

Because there are 16 patches, for transformer-model dimension $d$ of $Q,~K,~V$, the output is extracted as a tensor of size $(16+1) \times d$, including the class token. A linear operation in the MLP head then emits one output per class ($d \rightarrow C$, where $C$ is the number of classes), and the layer-normalization result is used as the score map for class prediction. In other words, applying softmax to this as the logits finally produces a class prediction from the class-token output.

When the dataset is not sufficiently large, however, a vision transformer cannot be expected to perform as well as CNN-based models such as BiT (a ResNet-based transformer). The reason is related to the absence of the inductive bias explained above. A CNN-based network can generalize and optimize quickly even from a small dataset thanks to the inductive bias inherent in its architecture, whereas a vision transformer, whose architecture divides an image into patches and computes attention for each one, has no such dependency at all.
This gives it a property that is both a disadvantage and an advantage. In a CNN, the representation map or feature map that can be extracted through predetermined operations eventually converges after sufficient training because of the inductive bias. In a vision transformer, by contrast, attention performance can continue to improve as the model is trained on ever larger datasets. This was interpreted to mean that it can keep advancing rather than converging to a form dictated by a particular architecture.
Accordingly, a vision transformer can achieve better performance by first pretraining on the massive JFT dataset ($3 \times 10^8$ samples) and then fine-tuning it for the desired downstream task.

Another problem is that attention operations can involve far more parameters and a much larger overall network than convolution operations. Even though VGG-16 is far from a lightweight model, it has 175.12M parameters and is therefore much lighter than ViT-Large or ViT-Huge. Even ResNet-152, one of the deepest ResNet networks, has only 60.34M parameters, which makes the excessive scale of transformer networks unmistakable.

Why Could the Vision Transformer Beat CNNs?

The simplest explanation is that the mechanism of self-attention makes it possible to reason over the entire input from the earliest layers. For example, for a CNN using a $3 \times 3$ kernel to obtain a receptive field of size $32 \times 32$,

$$3+2 \times (n-1) \ge 32,~n > 14$$

more than 14 layers are required. If filtering such as max pooling is added to reduce computation along the way, the model may have to perform image classification before it can even recognize global fine detail in some parts of the image. Put differently, a ViT can obtain a uniform representation map at every layer and use self-attention to compress global information more quickly. It can then share this rapidly summarized global information with each patch across multiple layers, conveying a well-learned representation. Its constraints, however, are invariably that it requires a massive dataset (roughly one billion samples) and correspondingly requires many parameters. In addition, for high-resolution images, calculating attention scores for $Q,~K,~V$ creates the major problem that the computation required for multi-head self-attention grows in proportion to $(H \times W)^2$. From here, we will introduce research that attempts to solve each of these problems.

How to Use Datasets Efficiently

First, let us examine DeiT: Training data-efficient image transformers & distillation through attention, a paper that explored how to train without using enormous datasets.

DeiT proposed training the same transformer architecture as ViT only on the ImageNet dataset, without pretraining it on a massive dataset such as JFT. I previously wrote a short introduction to knowledge distillation (reference). The idea here is that a convolutional neural network can serve as a teacher and provide considerable assistance through distillation when a transformer struggles. There is no need to depend on a convolutional neural network architecture; one need only add a single token to which distillation loss can be applied.

There are two choices at this point: whether to use soft-label distillation from the teacher network's predictions as the loss, or hard-label distillation in which the maximum teacher prediction is one-hot encoded.

$$\mathcal{L}\_{\text{global}} = (1-\lambda) \mathcal{L}\_{\text{CE}}(\psi(Z\_s), y) + \lambda \tau^2 \text{KL}(\psi(Z_s/\tau),~\psi(Z_t/\tau))$$

One approach adds a weighted soft-distillation loss by appropriately adjusting the temperature value $\tau$ and the corresponding $\lambda$ value, while the other uses hard distillation to predict a hard target, as follows:

$$\mathcal{L}\_{\text{global}}^\text{hard Distill} = \frac{1}{2} \mathcal{L}\_{\text{CE}}(\psi(Z\_s), y) + \frac{1}{2} \mathcal{L}\_{\text{CE}}(\psi(Z_s),~y_t)$$

Here, $y_t$ is expressed as $\arg \max_c Z_t(c)$. In other words, it is supplied as a one-hot encoding in which the maximum teacher prediction is 1 and all others are 0. The DeiT paper introduced the latter method and optimized this loss term. This network, however, has a very clear limitation: the CNN's performance becomes an upper bound. After ViT was proposed, the analysis behind transformer-based vision approaches was that they could outperform CNNs if they could be trained using a massive dataset such as JFT. Yet in trying to use data efficiently, DeiT used a CNN as the teacher network and thereby returned to the starting point.

Efficient Computation for High-Resolution Images

The next issue is computation on high-resolution images, which the vision transformer did not solve. In the vision transformer's computation process, an image is divided into equal-sized patches, linearly projected into embeddings, and then passed through a series of attention operations. Thus, if the image resolution doubles, the cost of linear projection increases by the square of that factor, or fourfold, and calculating the attention weights through $Q,~K,~V$—which is essential for attention—again squares that amount, increasing it sixteenfold.

Most importantly, if the number of tokens is limited because of computation, it becomes difficult to extract features at multiple scales as CNNs do. Accumulating global information rapidly is an advantage, but the accumulated data is ultimately limited to a single scale. A vision transformer should be extended so that it can be applied to high-level vision tasks such as segmentation and image restoration, yet the current architecture does not appear capable of this. This is where hierarchical feature extraction through Swin-Transformer, which we will now discuss, emerged.

Instead of extracting a fixed number of patches from a high-resolution image, what if we perform progressive operations on patches and then add a merging process? The paper asks whether this might operate like a CNN, which captures fine detail in its early stages and coarse features in its later stages.

The network architecture is shown above. Looking for the major differences from ViT, the patch-partition step initially produces $\frac{HW}{4^2}$ patches (this refers to the number of patches), then gradually increases patch size until the final stage uses patches corresponding to $\frac{HW}{32^2}$. Patch size is increased by joining adjacent patches.

1. Through patch partitioning, the initial patch has size $4 \times 4 \times 3$. To avoid confusion about the values in the figure above, $\frac{H}{4} \times \frac{W}{4}$ is the size of one patch, and each patch can be thought of as containing pixels with RGB values arranged in a total of $4 \times 4$ units.

2. In Stage 1, linear embedding increases the number of patch channels from $4\times4\times3$ to $4\times4\times C/16$. This can be regarded as the stage that extracts an embedding for each patch. The paper actually uses C = 192, so although it begins with three RGB channels, the number is increased fourfold to $192/16 = 12$.

3. Beginning with Stage 2, the familiar attention operation is performed. Here, however, patch merging also occurs. When patch merging takes place, four adjacent patches are joined into one larger patch. Computation decreases at the same time, so the saved capacity is used to increase the number of channels. In other words, when four patches of size $\frac{H}{4} \times \frac{W}{4}$ are joined, the size of each patch becomes $\frac{H}{8} \times \frac{W}{8}$. Because the number of subdivisions inside a patch remains constant (this is the most important part!), the amount of attention computation decreases with each merge layer. Why? The number of subdivisions within each patch on which attention is performed remains constant, while the total number of patches gradually decreases through merging. The number of channels is therefore doubled.

4. For reference, attention is performed only within each window. A window here effectively means a patch. Strictly speaking, the concepts of a window and a patch should be distinguished, but this paper appears to use the terms interchangeably. Most reviewers probably became confused at this point. The number of subdivisions in each window is constant (this has the same meaning as the statement in item 3 that "the number of subdivisions inside a patch remains constant"—it bears repeating to prevent confusion).

5. This process proceeds through the attention blocks. The concepts of W-MSA and SW-MSA will be explained below.

In short, that was a lengthy explanation of performing attention while adjusting patch sizes in various ways. This part is virtually impossible to understand from the wording alone until you work it out yourself, because the terms "patch" and "window" overlap so much in the paper. Strictly speaking, they are not entirely different and do overlap to some extent, but it feels slightly mischievous to confuse the reader this way for the sake of the shifted-window method. In any case, let us now look at W-MSA and SW-MSA.

W-MSA is attention over the patches inside a window. In the paper's implementation, a single window probably contains a $7 \times 7$ grid of embeddings, and the operation is performed among the elements contained in that one window. If computation is performed only in this way, however, there are cases in which embeddings belonging to different windows cannot obtain information from one another. I drew the following example.

Although the blue patch and red patch are clearly adjacent in space, they cannot attend to each other until the very last attention layer. This divided-country problem can be fatal for images, where locality matters, and it also prevents the rapid acquisition of global information—one of ViT's advantages. The following strategy is therefore used. Attention within a window is calculated as depicted above. Shifted-window attention, however, uses a strategy that moves the window as follows. To make it easier to understand, I treated the entire image as a single window and divided its interior into $4 \times 4$ patches.

When the window is shifted in this way, a region formerly contained in the original window falls outside by the amount of the shift. A simple "cut-and-paste" method fills it back in. More precisely, this is a cyclic shift. It may help to recall the method used in chemistry to count components in molecular structures (body-centered cubic structures and the like), where the problem becomes easier by assuming that the entire material has a repeating structure.

After shifting the window and performing attention, the following figure shows that attention can now be calculated with patches from other windows as well.

Instead of simply using the previous scaled dot-product self-attention equation, this method adds a bias reflecting relative position to the score.

$$\text{Attention}(Q,~k,~V) = softmax(QK^\top / \sqrt{d}+B)V$$

Unlike the earlier positional-embedding approach, which added absolute coordinates (sinusoidal embeddings), this attention-embedding method reflects the positions of individual patches. In simple terms, if the current query patch is at the upper left and the patch to be attended to is at the lower right, a bias is added in the direction in which both $x, y$ increase. If their positions are reversed, a bias is added in the direction in which both $x, y$ decrease.

Let us use the figures above as an example. If one window in which attention is performed contains a total of nine patches (the regions numbered 1 through 9), we can express an $x$-axis matrix that represents differences in row indices between these patches and a $y$-axis matrix that represents differences in column indices.

The following operations are then applied to reflect the window size.

# Add (window size - 1) to each matrix
y_axis_matrix += window_size - 1
x_axis_matrix += window_size - 1

# Scale by 2M-1 and then add
x_axis_matrix *= (2 * window_size -1)
relative_position_M = x_axis_matrix + y_axis_matrix

With this scaling, the relative matrices, which originally ranged from $-M-1 \sim M-1$, are scaled to $0 \sim 2M-1$ and then multiplied together, allowing them to be represented over the range $0 \sim (2M-1)^2$.

How Can CNNs and ViTs Be Combined Effectively?

As we saw above, Swin-T, which uses the shifted-window method, has major advantages: it enables much more efficient computation; its hierarchical architecture provides the flexibility across multiple scales that CNNs possess; and its computation time remains linear, rather than quadratic, with respect to image size. This scale flexibility also makes the architecture generally applicable to a range of vision tasks beyond classification, including detection and segmentation.
The drawback of Swin-T, however, is that it remains difficult to train from scratch on a small dataset, a problem that had gone unsolved since ViT. The data-efficient transformer introduced earlier used a CNN as a teacher model for knowledge distillation, but its performance was ultimately determined by that of the CNN. Could there be a way to combine the advantages of CNNs and ViTs, obtaining both efficient representation learning through inductive bias (data efficiency) and the use of global information?
Convolutional vision Transformer (CvT) emerged from precisely this question. It sought to integrate the many advantages of CNNs into the ViT architecture. For example, a CNN can handle shifts, scaling, and even a tolerable amount of distortion in an object invariantly. ViT, by contrast, offers dynamic attention, global information, and better generalization.
The two principal architectural changes in CvT are as follows.

1. It introduces a new embedding method called convolutional token embedding for the transformer's hierarchical architecture.
2. The convolutional transformer block produces the convolutional projections that will be computed.

To begin with the conclusion, DeiT reused the architecture as-is with a smaller dataset but could neither reduce the parameter count effectively nor improve performance beyond its CNN teacher network. CvT, however, demonstrated that it could use parameters more effectively while raising performance beyond ViT. Looking at the overall network architecture,

the core idea is to perform token embedding on the input image or an intermediate feature map through a convolution operation, then apply attention to those tokens. A convolution is then performed again on the resulting lower-resolution feature map, followed by attention over its tokens, and this process repeats.

For output $x_{l-1}$ from layer $l-1$, let the new token map obtained as the output of the convolution in layer $l$ be $f(x_{l-1})$. Here, $f$ denotes a filtering function. In the official code, the convolutional embedding component is implemented as follows:

class ConvEmbed(nn.Module):
    """ Image to Conv Embedding
    """

    def __init__(self,
                 patch_size=7,
                 in_chans=3,
                 embed_dim=64,
                 stride=4,
                 padding=2,
                 norm_layer=None):
        super().__init__()
        patch_size = to_2tuple(patch_size)
        self.patch_size = patch_size

        self.proj = nn.Conv2d(
            in_chans, embed_dim,
            kernel_size=patch_size,
            stride=stride,
            padding=padding
        )
        self.norm = norm_layer(embed_dim) if norm_layer else None

    def forward(self, x):
        x = self.proj(x)

        B, C, H, W = x.shape
        x = rearrange(x, 'b c h w -> b (h w) c')
        if self.norm:
            x = self.norm(x)
        x = rearrange(x, 'b (h w) c -> b c h w', h=H, w=W)

        return x

According to the default settings, patch_size = 7, stride = 4, and padding = 2. Therefore, for input-image spatial resolution $H_i \times W_i$, the output-embedding resolution $H_o \times W_o$ is

$$\begin{aligned} H_o = \left( \frac{H_i + 2 \cdot 2 - 7}{4} + 1 \right) \newline W_o = \left( \frac{W_i + 2 \cdot 2 - 7}{4} + 1 \right) \end{aligned}$$

as shown above. An embedding is extracted while reducing the dimensions in this way, and the result is then passed through layer normalization. Attention is now performed over the extracted tokens. Unlike conventional attention, the convolutional projection differs from the preceding projection method as follows.

The leftmost diagram, (a), shows the linear-projection method ViT uses to map tokens to query, key, and value. They can be produced simply by multiplying by weights $W^Q,~W^K,~W^V$. Convolutional projection, in contrast (diagram (b) in the middle), reshapes and pads the tokens to create a window structure on which convolution can be performed, then applies convolutional projection to produce the query, key, and value. The following excerpt from the official code provides a simple illustration of the computation process.

def forward_conv(self, x, h, w):
    # Separate the class token; convolution is applied only to the image tokens
    if self.with_cls_token:
        cls_token, x = torch.split(x, [1, h*w], 1)

    # Use einops.rearrange to reshape the HW * C input into C * H * W
    x = rearrange(x, 'b (h w) c -> b c h w', h=h, w=w)

    # Every conv_proj_ operation is a convolution with the embedding dimension as both channel input and output
    # The convolution uses (kernel=3, stride=1, padding=1), preserving the spatial dimensions
    if self.conv_proj_q is not None:
        q = self.conv_proj_q(x)
    else:
        q = rearrange(x, 'b c h w -> b (h w) c')

    if self.conv_proj_k is not None:
        k = self.conv_proj_k(x)
    else:
        k = rearrange(x, 'b c h w -> b (h w) c')

    if self.conv_proj_v is not None:
        v = self.conv_proj_v(x)
    else:
        v = rearrange(x, 'b c h w -> b (h w) c')

    # Once every operation is complete, rearrange flattens the values again as before
    if self.with_cls_token:
        q = torch.cat((cls_token, q), dim=1)
        k = torch.cat((cls_token, k), dim=1)
        v = torch.cat((cls_token, v), dim=1)

    # Reattach the class token separated earlier to obtain the query, key, and value needed for attention
    return q, k, v

For reference, the convolutional block used above employs depthwise + pointwise convolution.

def _build_projection(self,
                    dim_in,
                    dim_out,
                    kernel_size,
                    padding,
                    stride,
                    method):
    if method == 'dw_bn':
        proj = nn.Sequential(OrderedDict([
            ('conv', nn.Conv2d(
                dim_in,
                dim_in,
                kernel_size=kernel_size,
                padding=padding,
                stride=stride,
                bias=False,
                groups=dim_in
            )),
            ('bn', nn.BatchNorm2d(dim_in)),
            ('rearrange', Rearrange('b c h w -> b (h w) c')),
        ]))
    elif method == 'avg':
        proj = nn.Sequential(OrderedDict([
            ('avg', nn.AvgPool2d(
                kernel_size=kernel_size,
                padding=padding,
                stride=stride,
                ceil_mode=True
            )),
            ('rearrange', Rearrange('b c h w -> b (h w) c')),
        ]))
    elif method == 'linear':
        proj = None
    else:
        raise ValueError('Unknown method ({})'.format(method))

    return proj

This is evident from the internal function that builds the projection. The paper itself describes the sequence as

• Depth-wise convolution 2d
• BatchNorm2d
• Point-wise convolution 2d

but looking at the actual implementation, the first two stages are as described, while linear projection appears to provide correlation across channels in place of pointwise convolution.
Ultimately, CvT achieved higher accuracy than transformer-based models with fewer parameters and fewer FLOPs. Because it does not rely on MLPs for attention operations, it requires fewer parameters to emit the same projection. At the same time, unlike CNN-based models, it demonstrated the high performance associated with ViT-based networks.

Multimodality Using Transformers

We have reviewed more papers than one might have expected. We began with CNNs and RNNs, then moved to sequence-to-sequence models, followed by the attention mechanism and the development of the attention-only network (transformer) built upon it. We also examined the proposal of ViT, which extended this success in NLP to vision tasks, and methods designed to overcome its various limitations (DeiT, Swin-T, and CvT). To conclude this article, let us briefly introduce the specific ways in which the transformer, which helped open the horizons of multimodality, can be used across diverse research areas.

Because the range of phenomena that modality can encompass is so broad, the term can be explained in many ways. From the perspective of "deep learning," which is our focus here, it can be defined as follows: a "modality" is an individual communication channel, such as vision, audio, or language, that can be acquired through a particular sensor or method of observation. The terminology may feel unfamiliar, but if a device such as a camera or LiDAR sensor can collect information in a particular form, that form can be described as a modality. A thermal image acquired through a thermal sensor is another modality, and a medical image acquired through a CT or MRI machine is yet another.
Accordingly, the deep learning task called multimodality, or multimodal learning, aims to perform representation learning by meaningfully using datasets acquired through different means, including vision, text, sound, and other forms of data, together.

The terms multimodal and cross-modal sometimes overlap, but the difference between them is as follows. Multimodal learning is a newly proposed class of deep learning algorithms that train by using multiple modalities together. Humans, for example, use both sight and hearing to identify a person or a particular object, and multimodal learning attempts to give computers this same ability. Cross-modal learning, by contrast, takes a multimodal deep learning approach in which information from one modality is used to improve the performance of another modality. If you saw an image of a cat, hearing a meow could lead you to conclude, "Ah, this must be a picture of a cat."
An AI system that can operate jointly across different modalities is called multimodal, while cross-modal learning uses knowledge shared across different tasks. As an analogy, multiple startups working in the same shared office, creating synergies that help all of their businesses succeed, could illustrate cross-modal learning. By contrast, multiple departments within a single startup working together, each diligently carrying out its own role to grow the company, could illustrate multimodal learning.

Representative multimodal methods that combine text and vision include captioning techniques such as video captioning and reasoning techniques such as video question answering, as shown above. Other applications include retrieval tasks that locate a particular part of a video or other media from a text description, technologies that generate video or images from text, and technologies that create video from audio.
From precisely this perspective, the "transformer" is the network architecture best suited to approaching multimodality.

Put simply, as long as a method can be found to tokenize every form of input, a transformer can embed it into a representative space and train it with attention. The embedding process is not complicated, and most importantly, it applies to "diverse forms of data." In a multimodal transformer, cross-modality interactions such as fusion and alignment also occur automatically through attention. Interested readers may find this survey paper helpful because it organizes the topic well.

If we regard purple and green as embeddings from two different modalities, the approaches above can be interpreted as follows. The embeddings may be added together before attention is performed (a), or they may be concatenated along a dimension without being added (b). Unlike those methods, separate transformer layers may be used and their outputs subsequently combined through a single transformer layer (c), or the modalities may initially be learned in one layer and then split into multiple layers (d). They may also be learned through separate layers whose queries cross over to attend to the other layer (e), or the results may be concatenated after this cross-attention (f). All of these methods appear conceptually plausible, and the architecture can be differentiated further according to relationships among the modalities and the nature of the problem being solved.
There is no need to know in detail which task uses which method. The key implication of this figure is simply that "the transformer architecture offers an extremely diverse set of strategies for jointly learning multiple modalities."

CLIP: Learning Transferable Visual Models From Natural Language Supervision

We will now explain CLIP, one of the most famous and widely cited papers to emerge from this multimodal perspective.

The training method is simple. Each image is paired with a text prompt describing it, and each is converted into an embedding through a transformer encoder. By training positive pairs between image and text embeddings to be close (the diagonal entries) and negative pairs to be far apart (the remaining entries), the model can ultimately learn text-driven image classification. Following the first contrastive-pretraining stage, the diagram shows a dataset classifier being created from label text. Because text descriptions were used during training, a similarly formed description—such as "This is a photo of [class]"—is also supplied during classification.
The key result of the paper was that after training on descriptions of countless images, the model could also perform well on classification images that had not been used during training. It has since been used in many forms and can be regarded as the hottest baseline in the current multimodal landscape.

The green regions indicate datasets on which zero-shot CLIP performed better than a fully supervised ResNet. Achieving higher accuracy than a ResNet trained with supervision on every dataset, despite having no access whatsoever to their training samples, is genuinely revolutionary. The CLIP paper contains many experimental details that deserve close examination, so I plan to review it separately in a future post.