Introduction

The previous article introduced batch normalization and implemented its deep-learning computations in code. This time, we will examine several other forms of regularization. Unlike batch normalization, which addresses shifts in layer activation statistics, these methods primarily seek to prevent overfitting. We will then review loss functions for the two most representative supervised-learning tasks: classification and regression.

Most of these ideas appeared briefly in earlier discussions of perceptrons and linear classifiers. This article gathers them in one place.

Regularization

Before considering its deep-learning meaning, look at the general idea of regularization. Across mathematics, statistics, economics, and computer science, regularization is used when we want the solution to be simple.

Regularization can be divided broadly into two kinds:

• Explicit regularization adds an explicit term to the optimization problem. It may take the form of a prior, penalty, or constraint. A regularization or penalty term assigns a cost to an optimization objective and can help select a unique optimum.
• Implicit regularization covers regularizing effects not expressed as an added objective term. Examples include early stopping—ending training after an appropriate fit has been reached—and using a robust loss or stochastic training procedure.

The $L_1$ and $L_2$ methods below are commonly applied as explicit constraints, often controlled through a weight-decay coefficient. Dropout changes the training procedure and is commonly described as an implicit form of regularization.

As discussed in the article on linear classifiers, if a particular $W$ satisfies the SVM margin, then $\alpha W$ also satisfies it for every sufficiently large $\alpha$. This creates non-unique solutions, which can slow or destabilize convergence. With softmax, increasing $W$ spreads the node outputs farther apart; an expressive network can consequently fit an overly complex function and overfit. Controlling the size of $W$ is therefore one useful form of explicit regularization.

L2 Regularization

L2 is the most common regularizer. It penalizes the squared magnitude of every parameter by adding a term to the objective, encouraging smaller weights. For a network containing many parameterized layers, the term can be written with weight $\lambda$ as

$$\frac{\lambda}{2} \lVert W \rVert_2^2.$$

The factor $1/2$ cancels the factor of two produced by differentiation. A full loss then takes the form

$$\text{Cost} = \frac{1}{n} \sum_{i=1}^n L(y_i, \hat{y}_i) + \frac{\lambda}{2} \lVert W \rVert_2^2.$$

A regression model with L2 regularization is called ridge regression. The squared penalty assigns a disproportionately large cost to peaked parameter values and is less tolerant of outliers than L1. Weight decay prevents the model from learning weights tailored too narrowly to its training set. If training performance can improve only by making the value of one particular node extremely large, an unregularized parameter may grow accordingly. When this happens across a model, it can produce a highly complex fitted function.

Generalization requires learning about the broader distribution that contains the training set, rather than overfitting to the training samples. This is why regularization matters.

L1 Regularization

L1 regularization uses the L1 rather than L2 norm:

$$\text{Cost} = \frac{1}{n} \sum_{i=1}^n L(y_i, \hat{y}_i) + \lambda \lVert W \rVert_1.$$

A regression model with L1 regularization is called Lasso regression. Compared with the quadratic L2 penalty, L1 does not amplify the penalty on unusually large values as strongly, making it more tolerant of outliers. If preserving rare signals in a modality matters, L1 may be useful; if smooth, broadly distributed weights are preferred, L2 is often the better choice.

Consider two vectors and their norms:

$$\begin{aligned} v_1 = (0.5,~-0.5,~0) \newline v_2 = (0.3,~0.3,~-0.4) \newline \lVert v_1 \rVert_1 = \lVert v_2\rVert_1 = 1 \newline \lVert v_1 \rVert_2 = \sqrt{0.5} \newline \lVert v_2 \rVert_2 = \sqrt{0.34}. \end{aligned}$$

Different parameter vectors can have the same L1 norm, while their L2 norms distinguish how concentrated the values are. Under a fixed L1 magnitude, L1's diamond-shaped constraint tends to select solutions on coordinate axes and therefore drives some parameters exactly to zero. In practice, L1 regularization is commonly used to encourage a sparse weight vector, while L2 regularization generally produces a dense one.

Elastic-Net Regularization

L1 and L2 can be used together in elastic-net regularization:

$$\text{Cost} = \frac{1}{n} \sum_{i=1}^n L(y_i, \hat{y}_i) + \lambda_1 \lVert W \rVert_2^2 + \lambda_2 \lVert W \rVert_1.$$

Max-Norm Constraints

The infinity norm is defined by the largest absolute parameter value. It can therefore constrain the maximum magnitude any weight may take:

$$\text{Cost} = \frac{1}{n} \sum_{i=1}^n L(y_i, \hat{y}_i) + \lambda \lVert W \rVert_\infty.$$

Dropout

The methods above add an explicit penalty or regularization term to the objective function. Regularization can also arise from the training procedure rather than a newly defined objective. The classic and best-known example is dropout.

Deep neural networks can overfit because their representational power allows them to describe highly complex functions. Greater capacity is necessary for many real-world tasks, but the same expressive power can fit the training data too closely.

Dropout indirectly simplifies the network during training by updating only some nodes. On every step, it turns off a random subset with probability $p$, effectively “dropping out” parts of the network.

In the ordinary multilayer perceptron on the left, every node is connected by edges and the complete network participates in computing the output. On the right, some nodes are switched off so that only a subset is trained during each training step.

The important point is that this stochastic masking applies only during training. Batch normalization likewise behaves differently during training and inference. Because dropout modifies the network to resist overfitting, random node selection is valuable while parameters are being learned. Once training is complete and the parameters are fixed, there is no reason to keep discarding capacity; inference uses the complete network with the appropriate activation scaling.

Loss Functions

Regularization controls model complexity or adds penalties so that a model generalizes rather than overfits its training set. The explicit $L_1$, $L_2$, and max-norm terms above are added to a task-specific data loss. Until now, that loss was written abstractly as

$$\frac{1}{n}\sum_{i=1}^n L(y_i,\hat{y}_i),$$

without specifying its form for classification or regression. Here $n$ is the number of training samples, or the number of samples in a mini-batch during batched training.

To define concrete losses, write a neural network $f$ with layer weights $W_k$ as a composition over each input $x_i$:

$$\begin{aligned} f(x_i; W_k),~(i =& 1,~2,~\cdots,~n\text{ and }k = 1,~2,~\cdots,~l) \newline \text{Let }&k^{th}\text{ layer's function be } f_k(\cdot,~W_k), \newline \newline f(x_i; W_k) =& f_l \circ f_{l-1} \circ \cdots \circ f_1(x_i; W_1). \end{aligned}$$

Keep this notation in mind for the equations that follow.

Classification

Classification is the fundamental task introduced repeatedly in the earlier linear-classifier articles. Each sample in a dataset has a label drawn from a predefined set. Two familiar classification objectives are SVM and softmax. For an SVM with score margin $\Delta=1$,

$$L_i = \sum_{j \neq y_i} \max \left( 0, f_j - f_{y_i} + 1 \right).$$

The hinge is not differentiable exactly at the margin, so some applications use squared hinge loss:

$$L_i = \sum_{j \neq y_i} \max \left( 0, f_j - f_{y_i} + 1 \right)^2.$$

Squaring changes the penalty and therefore the resulting model, but squared hinge loss performs better for some tasks.

Instead of applying hinge loss directly to scores $f$, a softmax classifier converts them into a normalized probability distribution:

$$L_i = -\log \left( \frac{e^{f_{y_i}}}{\sum_j e^{f_j}} \right).$$

Ordinary classification works well when the number of classes is manageable. In NLP, where the vocabulary may contain an enormous number of words, every incorrect class becomes a distractor in the full softmax and the calculation itself carries a large computational cost.

Some tasks therefore use modified classification objectives. Hierarchical softmax, introduced for NLP, organizes words into a tree (reference). Every label is represented by a path through the tree, and the classifier learns left-versus-right decisions at each branch instead of one flat softmax over all labels. A full softmax over a vocabulary of size $W$ requires work proportional to $W$, whereas following a balanced binary tree requires only $\log_2(W)$ decisions.

More formally, let $n(w,j)$ be the $j$th node on the path from the root to word $w$, and let $L(w)$ be that path's length. Then $n(w,1)=\text{root}$ and $n(w,L(w))=w$. For an internal node $n$, let $\text{ch}(n)$ be one designated child, and let $\langle x\rangle$ return $1$ if $x$ is true and $-1$ otherwise. Hierarchical softmax is

$$p(w \vert w_I) = \prod_{j = 1}^{L(w) - 1} \sigma \left(\left\langle n(w,j+1) = \text{ch} (n(w, j)) \right\rangle \cdot {v_{n(w, j)}^\prime}^\top v_{w_I} \right).$$

As the objective shows, each word requires only the sigmoid decision at every child node along its path.

Attribute Classification

SVM and softmax above assume one correct answer $y_i$ for every sample $x_i$. What if $y_i$ is instead a binary vector indicating the presence of multiple non-exclusive attributes? An Instagram image, for example, may be labeled with several hashtags from a very large set. A straightforward solution is a separate binary classifier for every attribute:

$$L_i = \sum_j \max (0, 1-y_{ij} f_j).$$

This sums a binary hinge loss over all attribute categories $j$. Here $y_{ij}=+1$ if sample $i$ contains attribute $j$ and $-1$ otherwise. Score $f_j$ represents the predicted presence of that attribute. A sign opposite to $y_{ij}$ accumulates loss.

Alternatively, each attribute can use logistic regression. Binary logistic regression uses class values zero and one, with the probability of class one given by

$$P(y = 1 \vert x;~w,b) = \frac{1}{1 + e^{-(w^\top x + b)}} = \sigma (w^\top x + b).$$

The probability of class zero is one minus this value. The default decision threshold is therefore $0.5$, equivalently $\sigma(w^\top x+b)>0.5$ or $w^\top x+b>0$.

The binary cross-entropy, interpreted as negative log-likelihood, is

$$L_i = -\sum_j \left[y_{ij} \log \sigma(f_j) + (1-y_{ij})\log(1-\sigma(f_j))\right].$$

Because $y_{ij}$ is zero or one, optimization pushes each sigmoid probability in the appropriate direction. The gradient with respect to attribute logit $f_j$ has the particularly simple form

$$\partial L_i / \partial f_j = \sigma (f_j) - y_{ij}.$$

Regression

Unlike classification, regression estimates real-valued quantities. Examples include predicting a property's sale price from its size, location, and facilities, or estimating the length of an object in an image. There are no predefined classes or discrete attributes; the model predicts a continuous value from the features.

The loss therefore measures the difference between the prediction and the true value, commonly with a squared L2 or L1 norm:

$$L_i = \lVert f-y_i \rVert_2^2.$$

Squaring is monotonic in the magnitude of the error, so it does not change which prediction is optimal and gives a simple gradient. An L1 loss is

$$L_i = \lVert f-y_i \rVert_1 = \sum_j \vert f_j - (y_i)_j \vert.$$

Unlike the L2 gradient, its magnitude does not depend on the size of $f-y_i$, only on its sign. Let $j$ index an output dimension and write the prediction errors for sample $i$ as

$$f - y_i = (\delta_{i1},~\delta_{i2},~\cdots,~\delta_{ij}).$$

Then the local gradient is

$$\partial L_i / \partial f_j = \operatorname{sign}(\delta_{ij}).$$

Why Regression Can Be Difficult to Optimize

L1 and L2 regression require a network to predict an accurate real value. Softmax, by contrast, can normalize imperfect raw outputs into a probability distribution, giving optimization a wider region in which the correct class wins. Direct regression can therefore have a narrower route to an acceptable optimum.

Squared L2 loss is also sensitive to large errors and, like L2 regularization, is not robust to outliers. These properties can make it difficult to stabilize a regression output within a desired range.

Suppose we want to predict a movie's public rating from one to five stars. Replacing direct regression with a five-class classification task provides more than single-output supervision: it also exposes the distribution the network assigns across possible ratings, which can serve as a confidence measure.

Closing

Many objectives also predict complicated structures such as graphs and trees. Covering their details would move well beyond the scope of a deep-learning introduction, so we will stop here. The central lessons are that regularization can prevent overfitting, and that even the same task can be optimized with several different forms of loss.