Introduction

In the previous post, we looked at the perceptron, which lived through the long stagnation of deep learning, the multilayer perceptron architecture proposed to overcome its limitations, and backpropagation, which made it possible to train that architecture. This post examines the same general subject—neural networks—but whereas the discussion so far focused mainly on methodology, this time we will pay more attention to their architecture and structure.

Neural network

You can understand neural networks without any background in neuroscience or anatomy. That is because the perceptron is designed in an intuitive, mathematical way. In linear classification, we computed a score for each class for a given image (with the score function $f$), and defined the weight parameter $W$ that constitutes the score function of a single perceptron. For example, with the CIFAR-10 dataset, the input was a column vector of size $32 \times 32 \times 3$, and the weights were arranged as a $10 \times 3072$ matrix so that the score function could compute scores for the ten classes.

That was the case with a single parameter. Now consider a neural network with a slightly more complex structure. Suppose its score function consists of two connected perceptrons:

$$s = W_2 \max(0,~W_1x)$$

The result of this operation is no longer affine—that is, it is nonlinear. If $W_1$ is a $100 \times 3072$ matrix, the operation $W_1 \cdot x$ produces a 100-dimensional hidden feature vector. Before the operation with $W_2$, the $\max$ function applies a nonlinear operation to that 100-dimensional vector. A function that performs such a nonlinear operation is called an activation function. It is commonly used to separate computational nodes in architectures that connect multiple perceptrons, as well as in many other neural network structures. There are many activation functions, and we will introduce several basic ones later. The $\max$ function above maps values below $0$ to $0$ and leaves larger values unchanged; it is called the Rectified Linear Unit (ReLU).

Consider a neural network built simply by stacking layers. If we apply linear operations across $n$ layers,

$$y = x \bigodot_{i=1}^n W_n = x \odot W$$

this is equivalent to first computing $\bigodot_{i=1}^n W_n = W$ and then applying a single linear operation. In other words, merely increasing the number of weights does not increase the computational complexity of the neural network. Activation functions are therefore essential to constructing a deep neural network. As we saw in the previous post, the process used to train the resulting full network is backpropagation.

Modeling a neuron

Neural networks began with the goal of implementing biological neural systems in a computing environment. Although the ultimate engineering objective eventually became achieving better performance, the research can be understood as having started from an attempt to reproduce the body's neural system—however approximately—in computational or logical form. The fundamental unit through which our brain performs computation is the neuron. Roughly $86$ billion neurons are used in the human brain, interconnected through approximately $10^{14}$ to $10^{15}$ synapses.

Without turning this into a biology lesson, the basic operating principle of the human nervous system is as follows. Each neuron receives input signals through its dendrites and emits an output signal through its axon. Along each branch, axons connect through synapses to the dendrites of other neurons. In a perceptron that implements this arrangement computationally, signals traveling through axons are represented as the input $x$. As shown in the figure, each dimension of $n$-dimensional data connects to the dendrites through a different synapse. The weight $W$ multiplied at this point represents the strength of the interaction between the signal entering through each synapse and each dendrite of the neuron. A generalized network like the one above takes a weighted sum of the signals received by the dendrites. We can consider the case where the output for an input is a single scalar value—an $n$-to-$1$ function.

$$\sum_{i} w_i x_i + b$$

If the combined interaction of all $W$ and $x$ values exceeds a certain threshold, the neuron fires (activates) and transmits a signal through another axon. In humans, the timing of synaptic transmission and neuron activation plays an important role in overall biological activity. In a computational setting, however, these events are synchronized in the simplest formulation, so spike timing is not important. What matters for a perceptron, rather than a biological neuron, is how strongly it fires (activates) in response to the incoming signal. This is called the “firing rate of the neuron,” and the function that implements it is function $f$, or the activation function.

$$y = f \left( \sum_{i} w_i x_i + b \right)$$

Thus, the firing and activation process of a human biological neuron is implemented in a computational perceptron as the activation function, which provides the nonlinearity discussed above. Declaring the perceptron described so far as a Python class gives the following code. The forward method applies the neuron’s computation to inputs ($x$) and returns the desired firing rate ($0 \sim 1$). This has the effect of normalizing the firing rate, which we will discuss further below.

class Neuron(object):
    def forward(self, inputs):
        """ assume inputs and weights are 1-D numpy arrays and bias is a number """
        cell_body_sum = np.sum(inputs * self.weights) + self.bias
        firing_rate = 1.0 / (1.0 + math.exp(-cell_body_sum)) # sigmoid activation function
        return firing_rate

A single neuron as a linear classifier

The expression represented by a single neuron was defined in the previous example as $\sigma \left( \sum_i w_i x_i + b \right)$, which is ultimately the class probability for a given weight $W$ and input $x$. Suppose this problem has two classes ($y$ is either $0$ or $1$). A single value normalized to the range $0 \sim 1$ can be interpreted as the probability of the corresponding class. For input $x_i$, the probability that the class is $1$ is $p(y_i = 1 \vert x_i;~w)$, while the probability that it is class $0$ is $1 - p(y_i = 1 \vert x_i;~w) = p(y_i = 0 \vert x_i;~w)$.
That is the intuitive account; let us now examine a concrete example. The point is to connect the use of a single perceptron for classification with the linear classification using softmax that we introduced earlier. For a particular input $x_i ~ (i = 1,~2,~\cdots,~n)$, suppose that

$$\sigma \left( \sum_i w_i x_i + b \right) = 0.2$$

As discussed, the sigmoid function $\sigma$ can normalize any input value to the interval $0 \sim 1$. The result can therefore express the probability of class $0$ or $1$ as follows:

$$\sigma \left( \sum_i w_i x_i + b \right) = p(y_i = 1 \vert x_i;~w) = 0.2$$

As the output approaches $0$, the probability of class $1$ decreases; as it approaches $1$, that probability increases. Conversely, the probability of class $0$ is

$$1 - \sigma \left( \sum_i w_i x_i + b \right) = 1 - p(y_i = 1 \vert x_i;~w) = 0.8$$

as shown above. This example used linear classification with softmax. If we instead used an SVM (Support Vector Machine), the activation function and the form of the neuron would change.

$$R(W) = \sum_k \sum_l W_{k,~l}^2$$

Among the concepts introduced in the linear-classification post was the regularization loss shown above, in addition to the data loss. Interpreted in terms of biological neurons, this regularization loss amounts to “gradual forgetting,” because it encourages the weights corresponding to synaptic strengths toward $0$ while optimizing the perceptron’s parameters.

---

Activation functions

We introduced activation functions—nonlinear functions—as a structural mechanism for implementing the firing rate in a perceptron modeled after the human nervous system. Let us look at the various activation functions proposed as neural networks developed.

Sigmoid ($\sigma$) function

The first function to introduce is sigmoid. Its historical popularity arose from the following properties. It can accept any real number as input—its domain is the full set of real numbers—and, much like a neuron's firing rate, maps a very small negative number to $0$ and a large positive number to $1$. Sigmoid nevertheless has several disadvantages.
The first problem is vanishing gradients: because the sigmoid graph saturates when $x$ is too small or too large, its gradient converges to zero. Recall that backpropagation trains the full network by multiplying each local gradient by the upstream gradient passed to a gate’s output. If the local gradient of a sigmoid gate converges to $0$, it becomes difficult to train all the weights before that gate. If training begins with a poor weight initialization, the unfortunate result may be that the network barely learns at all from the outset.
The second problem is that the midpoint of the sigmoid function is not $0$. When an activation value cannot be negative, all weights may update in the same direction during parameter updates, severely slowing training. Consider an example. Suppose that when an input ($x$) enters a neuron, every component of that input is positive.

$$f = \sigma \left( W^\top x + b \right) = \sigma \left( y \right)$$

The local gradient of the sigmoid function is

$$\frac{d \sigma(y)}{dy} = \left( \frac{1+e^{-y}-1}{1+e^{-y}} \right)\left( \frac{1}{1+e^{-y}} \right) = (1-\sigma(y))\sigma(y)$$

Because the sigmoid function’s output lies between $0$ and $1$, the gradient passed through a sigmoid gate is positive for every input. Applying the chain rule to weight $W$ gives

$$\begin{aligned} \frac{\partial f}{\partial W} =& \frac{\partial f}{\partial \sigma} \cdot \frac{\partial \sigma}{\partial y} \cdot \frac{\partial y}{\partial W} \\ =& (1-\sigma)\sigma x > 0 \end{aligned}$$

as above. Because every gradient points in the same direction, training becomes difficult.

The figure above depicts parameter updates in a coordinate system whose axes are weight columns. If weights can update only in the same direction, learning cannot proceed directly as shown on the left—decreasing $W_2$ while increasing $W_1$—and instead follows the zigzag path on the right.

Hyperbolic tangent ($\tanh$) function

The function above is the hyperbolic tangent, expressed as $\tanh(x) = 2\sigma(2x) - 1$. Its local gradient is

$$\frac{d}{dx} \tanh(x) = (1 - \tanh(x))(1 + \tanh(x))$$

which has the immediate advantage of being larger than the gradient of the sigmoid function introduced earlier. Because $\tanh$ can take negative values, it also resolves the sigmoid’s problem of causing weight parameter updates to follow a zigzag path. However, inputs that are too large or too small still cause gradient saturation, so nodes before the activation function can still die during neural-network training.

Rectified Linear Unit (ReLU) function

This is probably familiar to most people who have encountered deep learning. Its form is simply a hinge loss with a threshold of $0$. The activation function appears in the AlexNet paper. The graph below shows that training with ReLU converges faster than training with sigmoid: the solid line represents a ReLU neural network, and the dashed line represents the convergence of a sigmoid neural network.

ReLU has the following advantages and disadvantages. Vanishing gradients affected both of the preceding functions—sigmoid and hyperbolic tangent—but with ReLU the gradient remains $1$ no matter how large $x$ becomes. Sigmoid and hyperbolic tangent also require computing exponentials, which is computationally expensive, whereas ReLU simplifies computation by emitting only values greater than zero. However, ReLU is vulnerable to “dying”. When a large gradient flows through a ReLU gate and updates its weights, the gate can move into a region where it will never activate again. Some deactivation is useful because ReLU naturally excludes unnecessary weight computations. But if the weight parameters learn in the wrong direction, none of the network’s nodes can optimize the parameters at that position. Learning rate, weight initialization, and similar choices therefore become important, making robust training more difficult.

Leaky ReLU and Parametric ReLU

Alternatives to conventional ReLU have therefore been proposed, including Leaky ReLU, which maps values below zero with a linear function having a small slope, and Parametric ReLU, which allows the network to learn an appropriate slope $a$ ($y = ax$) during training. Subsequent studies have proposed periodic activation functions, GeLU, SiLU, and other activation functions suited to particular tasks.

Neural network architectures

Graph structure and the number of layers

A neural network is commonly modeled as a directed graph in which signals flow in one direction between neurons. Put simply, the output of one neuron becomes the input of another, and this order cannot be exchanged (the operations are not commutative). Cycles—where a node’s output eventually loops back to become its own input—are not allowed in this kind of graph, because such a structure is an IIR (infinite impulse response) system.

A neural network with the structure above is described as a “network with $3$ layers.” When defining a neural network with $N$ layers, the input layer is generally not counted, because it simply represents the input signal—with a modality such as image, speech, or text—itself.
Thus, a single-layer neural network has no hidden layer; logistic regression and SVMs, for example, are called single-layer neural networks. The linear classifier discussed earlier is likewise a single-layer neural network. Networks of this sort are referred to as ANNs (Artificial Neural Networks) or MLPs (Multi-Layer Perceptrons). Perceptrons responsible for individual layers are connected to form a neural network with depth. The output layer sometimes has no activation function, particularly when it emits scores used as the basis for classification or when the task performs regression without normalization.

Neural network size

The size of a neural network is often defined by its number of parameters. The number of trainable parameters is determined by the number and dimensionality of its layers, as follows.

Although the layers are drawn as nodes (circles), the actual layer operations are performed along the edges connecting the nodes (arrows). For example, assuming biases are included, the number of parameters in the network on the left is

$$\begin{cases} \text{input layer} \rightarrow \text{hidden layer 1}, & W \in \mathbb{R}^{3 \times 4},~b \in \mathbb{R}^{1 \times 4} \\ \text{hidden layer 1} \rightarrow \text{hidden layer 2}, & W \in \mathbb{R}^{4 \times 4},~b \in \mathbb{R}^{1 \times 4} \\ \text{hidden layer 2} \rightarrow \text{output layer}, & W \in \mathbb{R}^{4 \times 1},~b \in \mathbb{R}^{1 \times 1} \end{cases}$$

so the total number of trainable parameters—the sum of all parameter dimensions—is $41$. Applying the same calculation to the structure on the right gives

$$\begin{cases} \text{input layer} \rightarrow \text{hidden layer}, & W \in \mathbb{R}^{3 \times 4},~b \in \mathbb{R}^{1 \times 4} \\ \text{hidden layer} \rightarrow \text{output layer}, & W \in \mathbb{R}^{4 \times 2},~b \in \mathbb{R}^{1 \times 2} \end{cases}$$

for a total of $26$ trainable parameters.

Representational power

Viewing neural networks as fully connected layers means treating the weights that constitute the network as parameters and defining a function from them. We can define the representational power of this family of functions and examine how it is modeled by a neural network. We defined a neural network with at least one hidden layer as an MLP; such a network has a structural scale of at least two layers. A neural network of this kind is a universal approximator (reference). Briefly, consider a complex function from the real world that we would like to describe:

We want a neural network to predict the function value $f(x)$ for a particular input $x$. Although we are assuming one-dimensional (scalar) function estimation for simplicity, multidimensional function estimation for an $n$-dimensional modality is also possible. In other words, given any continuous function $f$, a neural-network function $g$ can approximate it. This lets us treat the neural network itself as a continuous function.

$$\begin{aligned} \text{if any }f(x) \text{ is continuous and some }\epsilon > 0, \\ \exists~g(x)~\text{s.t. }\forall x, \vert f(x) - g(x) \vert < \epsilon \end{aligned}$$

The reference above provides examples in which a neural network with one hidden layer serves as a universal approximator. Although it is mathematically true that a two-layer neural network is a universal approximator, using one in practice is difficult when the real world contains complex, multidimensional functions.

$$g(x) = \sum_i c_i 1(a_i < x < b_i)$$

A function like the one above can also serve as a universal approximator, for example, but functions of this form are never used in machine learning. To make neural networks useful in practice, we seek function forms that reflect the statistical properties of the data and fit the structures encountered in the real world. Finding those forms requires optimization with algorithms such as gradient descent. Deep networks—models with many layers—are more applicable to real-world data because multiple layers increase the network’s representational power. Ultimately, representational power encompasses the variety and complexity of functions that a universal approximator can express.

Why deep learning?

In general, a three-layer neural network has greater representational power than a two-layer network and therefore performs better on real-world tasks. Yet blindly adding layers is not necessarily the key to improving every performance metric. This differs somewhat from a Convolutional Neural Network, where recognition performance tends to improve as depth increases—although even for CNNs, making a network arbitrarily deep is not always beneficial. Images possess a hierarchical feature structure, from object outlines to detailed textures, and multiple layers can separate that hierarchy to extract or recognize features. Returning from that detour, the point is that choosing a neural network’s number of layers requires an appropriate trade-off.

The figure above visualizes how the number of hidden layers affects the decision regions—the boundaries—for a binary classification task that separates red and green data. Larger neural networks have greater representational power and can therefore express more complex function forms, but this can also cause them to overfit the training dataset.

Overfitting means that the network needlessly learns even the noise in the training data. In the preceding figure, the 20-layer classifier learned representations capable of separating every training sample, creating isolated regions of red within green and vice versa. As a result, its accuracy on the actual test dataset may decrease. Reducing the number of layers is not the only way to alleviate this problem. Other methods include the $L_2$ regularization introduced for linear classification and dropout, which ignores selected nodes (weights) during training.

If $\lambda$ denotes the importance assigned during optimization to the regularization loss, in addition to the classification loss, the continuous function learned by the network becomes progressively simpler as the regularization loss grows. The earlier post on regularization may be difficult to recall, but its point was that regularization encourages a more even distribution of weight parameters in a classification task.
Models with fewer layers usually have fewer local minima. Greater representational power increases the complexity of the functions a model can express, but it also makes the loss surface over the weights more complicated. The minima of a smaller network are easier to reach, yet they can lie far from the true global minimum—if each trainable weight is regarded as an axis, the feasible region is smaller—so the resulting performance may be poor. A larger model has many more local minima, but can still produce better results in practice.
Small networks are much easier to optimize because the number and depth of local minima are markedly lower. However, this can create large performance variation—or a large gap between optimized and maximum performance. Large networks contain many local minima, making the global minimum difficult to find, but they often perform well despite this variance. The initialization of their weights has less influence on the final result, which makes generalization easier.
This extended discussion provides one interpretation of why deep learning can work so well while preventing overfitting and learning representations. Rather than training fewer layers, training a deep network with regularization to prevent overfitting provides more stable learning during initialization and optimization.

Conclusion

A model with shallow layers learns a simple function and therefore has the advantage of easier convergence. It has fewer local minima and is more likely to converge to the global minimum, but the minimum it reaches may deliver performance well below the network’s potential maximum. A shallow model is consequently highly sensitive to weight initialization and may fail to generalize across real-world datasets whose full loss-function structure is unknown. This is why deeper models are used, although—as discussed above—increasing representational power also creates a greater risk of overfitting. The answer is therefore to use regularization, rather than merely reduce the number of layers. During representation learning in a deep neural network, many local minima exist and the point reached during training may not be the global minimum of the entire function. Yet many of these minima still perform well, which makes the model easier to generalize to a wide range of real-world settings.