Introduction

The preceding posts took us on a long journey through neural networks. Deep learning began as research that extended neural networks—a machine-learning methodology—to train models with deeper stacks of layers. That research has advanced so far that AI now underpins much of the business world. When I first began studying deep learning, I sometimes wondered whether it was even a field I should pursue. As I moved toward more recent papers and research, I often questioned how much I could still gain through graduate study in this area. Even so, I am frequently grateful that I began studying it before it was too late, and since choosing this career path I do not think I have ever regretted the decision. In any case, this post examines CNNs, the foundation of the deep-learning advances that accelerated in the early to mid-2010s, and organizes the reasons this architecture attracted so much attention in computer vision.

Convolutional neural network

The network architecture known as a CNN is widely used in deep learning. But how, exactly, did the convolutional neural network become so useful? Before looking at CNNs themselves, let us first examine the mathematically defined convolution operation.

Convolution in a $1D$ signal

Given two functions $f$ and $g$, convolution is the term for their convolution product, denoted mathematically by $f \ast g$. The convolution operation reflects one of the two functions about the $y$-axis, shifts it, multiplies it by the other function, and integrates the result. This is difficult to convey in words, so in equation form:

$$f \ast g(t) = \int_{-\infty}^\infty f(\tau) g(t - \tau) d \tau$$

If we regard the convolved function as a new function $h(t)$ of $t$, then $t$ represents the distance translated along the $x$-axis. Convolution is commutative, so the following identity holds:

$$f \ast g(t) = \int_{-\infty}^\infty f(t - \tau) g(\tau) d \tau$$

The figure also defines autocorrelation and cross-correlation. Both measure the relationship between functions: cross-correlation compares two different functions, while autocorrelation measures a function against itself. Autocorrelation uses the same equation as cross-correlation and can be understood as the special case in which the two functions whose correlation is measured are identical. For continuous, real-valued signals $f$ and $g$,

$$f \star g(t) = f \ast g(-t) = \int^\infty_{-\infty} f(\tau) g(t+\tau) d \tau$$

and

$$g \star f(t) = g \ast f(-t) = \int^\infty_{-\infty} g(\tau) f(t+\tau) d \tau = \int^\infty_{-\infty} g(t - \tau) f(t) d \tau$$

Unlike convolution, this operation is not commutative. Autocorrelation, of course, retains commutativity. With the mathematical meaning of one-dimensional convolution in mind, we can now see how a CNN differs and in what sense it came to be called a convolutional neural network.

Returning to CNNs

A convolutional neural network differs from the neural network architectures discussed so far in its operations, but its overall logic is similar. The perceptron-based MLP introduced earlier had the following structure.

A deep-learning network was built from perceptrons as its basic unit, with each layer containing multiple nodes and the full structure containing many such layers. In short, it had trainable weights and biases—the parts marked in red. Each neuron receives input, computes with its weight and bias parameters, and then passes the result through an activation function, represented as a nonlinear gate. The full network is a differentiable function that outputs scores and is trained to minimize a loss function such as SVM or softmax loss. A convolutional neural network works through the same mechanism if we regard a convolutional layer as a kind of neuron containing weights and biases. The difference is that a convolutional neural network (CNN) adds the explicit assumption that the input is an image. This assumption gives the architecture useful properties and makes it possible to build a network that can achieve high performance with fewer parameters.

Neural network architecture

A conventional neural network receives its input as a single vector and transforms it through several hidden layers that expand and contract its dimensions. Each hidden layer consists of many neurons, and every neuron is fully connected to all neurons in the preceding layer. Nodes within a layer do not share computations; each performs an independent operation. The last layer is the output layer, which emits task-appropriate scores.
This MLP, or regular neural network, scales poorly as dimensionality grows. CIFAR-10, for example, has relatively low-resolution images ($32 \times 32 \times 3$), yet the first hidden layer must already receive $3072$ input dimensions. This number alone may not seem computationally burdensome, but an ImageNet image with resolution $224 \times 224 \times 3$ requires computing over $150,528$ input weights. Remember that this is only the input dimension; accounting for the output dimension causes the number of parameters needed for the weights and biases to rise dramatically. Deep learning requires multiple layers, and learning sufficiently expressive representations requires assigning enough nodes to each layer. A conventional network made from fully connected layers is therefore computationally expensive, and its dense connectivity also creates a high risk of overfitting. From this perspective, CNNs—neurons with three-dimensional structure—benefit from the explicit assumption that the input is an image and from constraining the architecture to a particular form. Unlike a conventional network, each layer has three dimensions independent of a flattened input: width, height, and depth. For a CIFAR-10 input image, for instance, the width, height, and depth are $32$, $32$, and $3$. The idea is that every CNN layer can likewise retain information about these natural dimensions of the image modality. Consequently, CNN operations do not involve every part of the input in the way the preceding neural-network operations did. Moving away from a fully connected structure helps prevent overfitting to image computations. Because computation no longer proceeds in a fully connected manner and can be applied independently of the overall input dimensions, the parameter count also decreases. The final output, of course, must still have the same task-dependent size as a conventional neural network’s output.

How does a CNN calculate output features for each hidden layer?

The following illustrates a 2D convolution. Suppose that in a cat-classification task we have an RGB cat image of size $H \times W \times 3$.

The conventional approach expands this RGB image data into a one-dimensional vector and then passes it through several hidden layers, as in the image on the left.

A convolutional layer, by contrast, applies a convolutional kernel (filter) directly to the original image dimensions without flattening. The enlarged RGB pixels appear in the figure on the right. Since the size of the filter used for convolution is independent of the input size, it can be applied directly to an image represented as a three-dimensional tensor. The figure above depicts the kernel applied at each pixel. Each kernel has a corresponding number of trainable parameters.

The figure above shows the CIFAR-10 example in three dimensions. The pale-pink $32 \times 32 \times 3$ tensor is the input image, and the smaller dark-pink three-dimensional tensor within it is the convolution kernel. The convolution kernel performs cross-correlation over the input region covered by the filter—we will discuss the precise computation later—and the results for every kernel are arranged to form the blue output tensor on the right. The way convolution proceeds, and why we have called it cross-correlation rather than convolution, can be explained as follows.

Convolution has three basic hyperparameters: kernel size, stride, and padding. The figure above uses padding $= 1$, stride $= 1$, and kernel size $= 3$. The teal grid is the output produced by convolution, and the blue grid below is the input. The dotted region outside the blue grid is the padding.

Kernel

The kernel specifies the region of the input to which the convolution filter is applied; “kernel” and “filter” are therefore used to mean essentially the same thing here. In the figure, the moving gray area is the kernel, with a kernel size of $3$. A filter’s kernel need not be spatially square—the width and height do not have to match.

Padding

As already noted, the outer region around the input, drawn with dotted lines, is the padding. Padding may use a constant value, as in zero-padding, or follow any of several other methods, such as extrapolated padding. Padding $= 1$ means increasing the input’s spatial dimensions by one cell on every side, which is visible in the figure. As with kernel size, the padding need not be identical at the top, bottom, left, and right.

Stride

The kernel moves by the stride as it computes across the input, including the added padding region. In the example, a filter with a $3 \times 3$ kernel moves one cell at a time because stride $= 1$.

One point deserves attention: unlike the 1D convolution introduced above, this operation does not reverse the input and filter relative to one another. Their inner product is computed while both retain the same orientation. And although the example appears to operate on a plane, the filter is not applied to just one channel; a single filter spans every channel.

The actual computation therefore looks like the figure above. The black block is a filter with a $3 \times 3$ kernel, moving with stride $= 1$. Since a $3 \times 3$ kernel is applied to each channel, the filter is actually a $3 \times 3 \times 3$ tensor that covers all RGB channels. We previously said that each neuron in a convolutional neural network has width, height, and depth. Here the kernel size represents the width and height, while the channel dimension of the input image or feature map is the filter’s depth. Consider the following concrete computation.

Both the RGB values and filter values are arbitrary. The filter is multiplied with each channel, and an inner projection is computed by summing the results. The channel-wise sums are then added together, so the filter’s output at this location is $7 + 15 + 4 = 26$. If a bias is present, the output becomes $26 + \text{bias}$. To visualize what a filter actually does, I wrote some toy code in Colab. First, upload any image you like. I used the cat image below, which was easy to find on Google.

I imported the modules needed to load the image as a numpy array and to visualize it, then normalized the image values from the original range of $0$ to $255$.

import cv2
from google.colab.patches import cv2_imshow
import numpy as np

image = cv2.imread("cat.jpg")
image = image/255.0 # Normalize image to 0 ~ 1

To run this locally or in Jupyter Notebook rather than Google Colab, I recommend using the cv2.imshow method instead of google.colab.patches. I then implemented convolution over a NumPy array as follows.

def conv2d(image, out_channels, kernel, padding=0, strides=1):
    # Build kernel according to input image size
    image_height, image_width, image_channel = image.shape
    if type(kernel) == int:
        kernel_channel, kernel_height, kernel_width = image_channel, kernel, kernel
    elif type(kernel) == tuple or type(kernel) == list:
        kernel_channel, kernel_height, kernel_width = image_channel, kernel[0], kernel[1]

    kernel = np.random.randn(kernel_channel, kernel_height, kernel_width)

    # Calculate output shape according to input image size and filter size
    output_height = int(((image_height - kernel_height + 2 * padding) / strides) + 1)
    output_width= int(((image_width - kernel_width + 2 * padding) / strides) + 1)
    output_channel = out_channels
    output = np.zeros((output_height, output_width, output_channel))

    # padding(zero-padding) on input
    if padding != 0:
        image = np.pad(image, ((padding, padding), (padding, padding), (0, 0)), 'constant', constant_values=0)
    
    # calculate 2d convolution
    for c in range(output_channel):
        output_per_channel = np.zeros((output_height, output_width))
        for h in range(output_height):
            if (h * strides + kernel_height) <= image.shape[0]:
                for w in range(output_width):
                    if (w * strides + kernel_width) <= image.shape[1]:
                        output_per_channel[h][w] = np.sum(
                            image[h*strides : h*strides + kernel_height, w*strides : w*strides + kernel_height, :] * kernel
                        ).astype(np.float32)

            output[: ,:, c] = output_per_channel

    
    return output

This is a simple implementation without exception handling. Every convolutional layer is initialized from a normal distribution, and I designed a network with three convolutional hidden layers. Between convolutions, I used the following leakyrelu method as the activation function. Leaky ReLU scales values below $0$ by $0.1$.

def np_leakyrelu(image):
    return np.where(image<0, 0.1*image, image)

I computed the output of each hidden layer and normalized it for visualization.

hidden1 = conv2d(image, 3, 3, 1, 1)
hidden2 = conv2d(np_leakyrelu(hidden1), 3, 3, 1, 1)
hidden3 = conv2d(np_leakyrelu(hidden2), 3, 3, 1, 1)

hidden1 = (hidden1 - np.min(hidden1))/(np.max(hidden1) - np.min(hidden1))*255.0
hidden2 = (hidden2 - np.min(hidden2))/(np.max(hidden2) - np.min(hidden2))*255.0
hidden3 = (hidden3 - np.min(hidden3))/(np.max(hidden3) - np.min(hidden3))*255.0

Visualizing hidden1, hidden2, and hidden3 produces the following results.

cv2_imshow(hidden1)
cv2_imshow(hidden2)
cv2_imshow(hidden3)

Even with completely untrained, randomly initialized filters, the network extracts object contours reasonably well. If the filters were optimized for a task, I would expect them to extract better image feature maps than those shown above.

Why is it convolution, not correlation?

Suppose that $x$ and $h$ are discrete signals. We use discrete rather than one-dimensional continuous signals here because convolutional neural networks are applied to digitized image datasets. Let the height dimension follow index $i$, and the width dimension follow index $j$.

$$y(m, n) = x(m, n) \ast h(m, n) = \sum_{j = -\infty}^\infty \sum_{i = -\infty}^\infty x(i, j) \cdot h(m-i, n-j)$$

Suppose that the first image is the input and the second contains the kernel values. If we regard the center of the kernel as $0$, then computing $y(-1, -1)$, for example, requires

applying the kernel after flipping it, as shown. Actual CNN computation, however, does not take an inner projection with a flipped filter; it simply multiplies and sums values in the same orientation. The NumPy conv2d example I implemented above likewise just multiplies by the filter. Under the formal definition, the operation should therefore be written $x(m, n) \ast h(-m, -n)$ rather than $x(m, n) \ast h(m, n)$. As noted in the one-dimensional case, it corresponds more closely to cross-correlation, which is not commutative. This distinction is not particularly important in practice, but I wanted to mention it because the term “convolution” can naturally prompt the question.

Calculating the output tensor shape

To build a network using convolution, we need to know the output dimensions produced by each convolutional layer. The PyTorch documentation gives a general formula (reference); stated in more detail here, suppose the input tensor has shape $H_\text{in} \times W_\text{in} \times C_\text{in}$ and the output tensor has shape $H_\text{out} \times W_\text{out} \times C_\text{out}$.
Regardless of each convolution filter’s kernel size, the output channel dimension must be $C_\text{out}$, so there must be $C_\text{out}$ filters. Likewise, every filter must cover all channels of the input tensor regardless of its spatial size, so each filter has depth $C_\text{in}$. Excluding the as-yet unspecified spatial kernel size, the tensor shape ($H \times W \times C$) and count of the convolutional filters are therefore

$$(\text{kernel size} \times \text{kernel size} \times C_\text{in}) \times C_\text{out}$$

The term in parentheses is the shape of one filter, and the multiplier outside the parentheses is the total number of filters. This is also the number of parameters in a bias-free convolutional layer. If a bias is included for every channel, the parameter count becomes

$$(\text{kernel size} \times \text{kernel size} \times C_\text{in} + 1) \times C_\text{out}$$

So far, we have discussed the parameter count of a convolutional layer without fixing the kernel size. The output tensor shape after applying a filter is

$$\begin{aligned} H_\text{out} =& \left( \frac{H_\text{in} + 2p - k}{s} + 1 \right) \\ W_\text{out} =& \left( \frac{W_\text{in} + 2p - k}{s} + 1 \right) \end{aligned}$$

This equation ignores special forms of convolution such as dilation. In the notation, $s$ is the stride, $p$ is the padding, and $k$ is the kernel size.

Other modules in a CNN

Because convolution itself is a linear operation, a convolutional neural network also uses supporting modules such as nonlinear gates and pooling layers. The following are several of the most common.

Nonlinearity

When AlexNet won the ImageNet competition with a CNN architecture, its authors noted that applying ReLU instead of sigmoid produced faster convergence.

Using a nonlinearity such as ReLU lets the computation extract features through a greater number of effective nodes.

Pooling layer

A pooling layer reduces the spatial dimensions. It is usually placed after convolution operations to reduce $H$ and $W$. For understanding-based tasks such as image classification, absorbing the information from the entire image into low-resolution semantic features is both computationally efficient and beneficial to performance. More generally, convolutional neural networks tend to increase the number of filters to gain representational power because each filter references a smaller region of the input than a fully connected layer. If the spatial dimensions remain unchanged, computation grows rapidly with every layer, eliminating the parameter-efficiency advantage over an MLP. Even segmentation, which uses high-level spatial features rather than only classification features, therefore downsamples via pooling to preserve computational efficiency and later upsamples. Such architectures inevitably lose some image signals that coarse features cannot reconstruct—a form of aliasing—and several methods have been proposed to compensate. Those methods fall outside the subject of this post, so we will not discuss them here.

Fully connected layer head

To apply the same score-function losses in a convolutional neural network, the final feature map may be flattened and passed through a fully connected layer, or global average pooling may be followed by a fully connected layer.

The figure above shows an approximate CNN architecture containing the modules just introduced. The convolutional and fully connected layers of a CNN contain trainable parameters; ReLU and pooling layers do not.

Max pooling vs. average pooling

As the figure below shows, pooling reduces the spatial dimensions of an output feature map. Because of the nature of two-dimensional signals, pooling is similar to low-pass filtering. Image frequency is related to resolution, so greater downsampling produces coarser features rather than fine-grained ones.

Pooling requires no parameters, as explained above, leaving two obvious options: take the maximum over each pooling region, or take its average. Anyone who has taken a course on low-level image signal processing will likely recognize that the choice of pooling method is related to the method used to reduce image resolution.

Max pooling is generally used between convolutional layers. One explanation is that, instead of averaging, it preserves the dominant or highlighted features. Details vary by task, but after a learned filter is applied, feature maps extract everything from the broad contours of an object to its detailed outlines. For downstream tasks including classification, object detection, and segmentation, the most useful signal values are the salient responses, not the surrounding noise near the maximum.

Useful baseline models

ResNet, which achieved state-of-the-art performance during the rise of CNN-based deep learning, remains highly popular: extensions of its architecture, including ResNeXt and ConvNeXt, are used as backbones across many tasks. I also recommend reading the papers on other important foundational models, including AlexNet, VGGNet, and GoogLeNet, which helped launch CNN-based computer vision. There is also LeNet, an even earlier foundational paper.
After reading both CNN and Transformer architecture papers, you begin to notice that many of them share broadly similar structures. Other worthwhile papers include EfficientNet, which sought to find efficient network configurations; deformable convolution, which addresses the limitations of fixed-shape convolutional receptive fields; atrous spatial pyramid pooling networks; and many other architectural modules.