Introduction

The previous post covered parts of a neural network that remain fixed—that is, things that do not change during training. We might choose either L1 or L2 regularization, for example, but once selected for a task, the objective function itself is fixed. We also examined network architecture and data preprocessing.
This post focuses on the parts that can change during training: methods for learning weight parameters and manually searching for hyperparameters.

Learning

A gradient check compares an analytic gradient, obtained by differentiating a differentiable function, with a numerical gradient, which approximates the derivative from the ratio between a small input change $h$ and the output change $f(x+h)-f(x)$.

Use the centered formula

$$\begin{aligned} \frac{df(x)}{dx} =& \frac{f(x+h) - f(x)}{h}~\text{(bad)} \newline \frac{df(x)}{dx} =& \frac{f(x+h) - f(x-h)}{2h}~\text{(use instead)} \end{aligned}$$

For very small $h$, the numerical gradient follows the ordinary definition of a derivative shown above. As in elementary calculus, the derivative is defined as follows.

$$f^\prime (x) = \frac{df(x)}{dx} = \lim_{h \rightarrow 0} \frac{f(x+h)-f(x)}{h}$$

If $f^\prime(x)$ is difficult to derive analytically because the function is too complex, differences in function values can instead produce a value that approximates the derivative.
We therefore choose very small $h$ and compute the numerical gradient as above. The figure below explains why the centered formula is preferable.

The black $f^\prime(x)$ is the true analytic gradient. We approximate it from the ratio of function-value change over a small interval $h$. A one-sided difference like the red line produces a slope error when curvature is high and the function quickly departs from its tangent. The centered formula instead places $f(x)$ in the middle, as shown in green, and follows the true derivative—the tangent slope—more closely.

Use relative error for comparison

The method above obtains a more accurate numerical gradient. We now need a fair comparison between numerical and analytic gradients.
Suppose we take an absolute or squared difference and call it large when it exceeds a fixed threshold. A difference of $10^{-4}$ is small if both gradients are near $1.0$, but enormous if they are near $10^{-5}$ or smaller. A relative measure, the relative error, is therefore better than an absolute threshold.

$$\frac{\vert f_a^\prime - f_n^\prime \vert}{\max (\vert f_a^\prime \vert,~\vert f_n^\prime \vert)}$$

If gradient difference is the absolute difference, dividing it by the larger gradient magnitude reveals whether that difference is actually a large error. Network depth matters as well: errors accumulate through deeper networks, so the same observed error means something different in a ten-layer network than in a one-layer network.

Single precision and double precision

This section is not specific to deep learning, but floating-point representation matters when discussing numerical error.
Single and double precision, commonly encountered in C as float and double, differ in how many bits represent a value.
Computers use floating point to approximate the infinitely many real values after the decimal point, storing the decimal position separately rather than fixing it. This covers a wider range than fixed point, but values are approximate and arithmetic is slower. Integer and fractional parts are not sharply separated, and the number of significant digits is finite.
Early computers used different formats, but nearly all now use the IEEE 754 standard for compatibility.

The mantissa range differs between 32-bit and 64-bit representations. A 32-bit value is single precision; a 64-bit value is double precision. To reduce error, double precision is preferable because it uses more bits.

Kinks in the objective

When measuring gradient error, differentiable and nondifferentiable functions must be distinguished. A “kink” is a nondifferentiable point, such as those in ReLU or SVM hinge loss.
Consider the ReLU slope at $x=-10^{-6}$. ReLU is defined as follows.

$$\text{ReLU}(x) = \begin{cases} 0, & \text{if}~~x < 0 \newline x, & \text{otherwise} \end{cases}$$

The function is nondifferentiable at $x=0$, so no derivative exists over the entire domain, but we can compute subgradients on its individual differentiable regions.

$$\frac{d}{dx} \left( \text{ReLU(x)} \right) = \begin{cases} 0, & \text{if}~~x < 0 \newline 1, & \text{otherwise} \end{cases}$$

Thus, the analytic ReLU gradient at $x=-10^{-6}$ should be $0$. Now suppose we instead compute a numerical gradient using small $h$.

$$\left. \frac{d}{dx} \left( \text{ReLU(x)} \right) \right\rvert_{x = -10^{-6}} \approx \frac{\text{ReLU}(-10^{-6}+h) - \text{ReLU}(-10^{-6})}{h}$$

If $h<10^{-6}$, the error from the analytic gradient is zero; above that, an error of $h-10^{-6}$ appears. To determine whether an evaluation crosses a kink, inspect both $f(x+h)$ and $f(x-h)$. For a function such as $\max(x,y)$, check which input is the winner on each side. If the winner changes—ReLU changes behavior at zero when $h>10^{-6}$—the interval crossed a kink and the numerical gradient is unreliable.

Other tips for gradient checking method

Use only a small dataset for gradient checking. More data points increase the chance that the loss includes a kink, and checking many samples is slow and inefficient.
Step size $h$ must also be chosen carefully. If it is too small, computational precision limits reduce accuracy; if too large, the approximation departs from the derivative definition. For objectives with kinks, choose $h$ small enough to avoid crossing many of them.
Gradient checking generally occurs at one particular point in parameter space. Passing at that point does not guarantee that every operation in the network is correct; it is necessary, not sufficient. Random initialization can also place the check away from characteristic points where gradient errors would become visible. If SVM weights are initialized extremely small, nearly every data point receives a near-zero score and similar gradient pattern, even when the gradient implementation is wrong. Once one score grows sufficiently larger than the others, that apparent correctness may fail to generalize. For stable checking, run a short burn-in period and check after the loss has begun learning. Checking at the first iteration examines an edge of the learned parameter space and can be inaccurate.
As discussed previously, explicit regularization adds a regularization loss weighted by $\lambda$ to the data loss to limit overfitting and parameter variance. If regularization loss dominates data loss, its gradient also dominates, obscuring the procedure for verifying the data-loss gradient. Check data loss with regularization removed. To check the regularization gradient, remove the data-loss contribution or increase $\lambda$ until regularization is too large to ignore.
Implicit regularizers such as dropout and augmentation must also be disabled during gradient checks. Any stochastic operation reduces reproducibility and therefore checking accuracy. Either fix the random seed while computing $f(x+h)$ and $f(x-h)$ or remove every such regularizer.
Networks commonly contain millions of parameters, making checks over every dimension impractical. Large models therefore check selected parameters and assume the rest are correct. Selection should cover every parameter type or network component: do not sample parameters blindly at random, but choose them deliberately as though designing a test of the full network.

Sanity check

Before optimization, a sanity check can verify that the network is configured correctly. A softmax classifier on CIFAR-10 uses cross-entropy; with random initial weights, every class probability should approach $0.1$ and initial loss should be about $-\log(0.1)=2.302$. For a Weston–Watkins SVM with margin $1$, every margin is initially violated because scores are near zero, so initial loss should be $9$. A different result may indicate incorrect initialization. Regularization can also be sanity-checked: increasing regularization strength $\lambda$ should increase total loss.
Another useful practice is to train on a tiny dataset before using the full one and verify that loss decreases. If the network cannot overfit even a small dataset, the code or implementation is probably wrong.

Learning process

After designing the network and completing sanity checks, monitor training continuously. Learning rate, optimizer settings, and layer details all require hyperparameter optimization even after the initial setup.

Loss function

First inspect the loss-function value, measured as each batch passes forward through the network. Every objective is optimized downward. If the learning rate is poorly chosen, training may be extremely slow or the loss may diverge, as shown below.

The learning rate determines how strongly each layer's local loss gradient updates its parameters. Too large a rate causes divergence or convergence at a poor loss; too small a rate slows training and risks getting trapped in local minima. With a rate suited to the model, dataset, and task, loss decreases cleanly. The graph on the right plots every batch loss by epoch and shows a downward trend. If the curve wiggles excessively, increasing batch size can reduce the noise.

Train/Validation accuracy

Decreasing loss is not always good: if the model is overfitting the training dataset, the training loss alone will not reveal it. We therefore reserve a validation set to evaluate generalization before final inference. Deep-learning data is divided into training, validation, and test sets.

1. Training dataset: Data used to train weight parameters with gradient descent.
2. Validation dataset: Evaluation data used during development without gradient updates.
3. Test dataset: Real-world data given to the model after training—the inference stage.

The graph distinguishes training accuracy from validation accuracy. Test data used at inference often lacks ground truth, so its accuracy cannot be measured directly. Comparing training and validation accuracy reveals whether gains from fitting the training set also transfer to unseen validation data, thereby measuring generalization.

Parameter updates

Everything learned so far about loss functions, backpropagation, and gradients serves to train weight parameters. We now examine several ways to update them: the concept represented by an optimizer in PyTorch and several variants of gradient descent, the foundation of neural-network optimization.

Vanilla update

The simplest method computes the gradient at each data point and updates parameters by a fixed learning rate in the opposite direction. A function's gradient gives the direction of steepest increase at a point, so its negative gives the direction of steepest decrease.

# Vanilla update
x -= learning_rate * dx

Momentum update

This method reaches the global optimum for a convex objective, but problems arise when convexity is not guaranteed. Consider the following figure.

The figure illustrates a convex function. Although a complete account also requires convex sets and convex hulls, a convex function can be described simply as one that curves upward throughout its domain. Thus, for every real $0 \leq \theta \leq 1$,

$$f(\theta x_1 + (1-\theta)x_2) \leq \theta f(x_1) + (1-\theta) f(x_2)$$

the inequality holds. For a multidimensional function, this corresponds to a positive-definite Hessian. A key advantage in optimization is the existence of one unique global optimum.
Suppose $f$ is minimized at $x^\ast$. The second-order sufficient conditions at an interior point are $\left. \nabla_x f(x) \right \rvert_{x = x^\ast}=0$ and $\left. \nabla_x^2 f(x) \right \rvert_{x = x^\ast}>0$. Now suppose another point $\hat{x}$ also minimizes the function, contradicting uniqueness. By the definition of convexity,

$$\begin{aligned} \text{For }\forall x& \in \left(0,~1 \right), \newline f(\theta x^\ast + (1-\theta)\hat{x}) &\leq \theta f(x^\ast) + (1-\theta) f(\hat{x}) \end{aligned}$$

Because both $f(x^\ast)$ and $f(\hat{x})$ are global optima, let their minimum value be $m$. The expression becomes

$$f(\theta x^\ast + (1-\theta)\hat{x}) \leq m$$

which says every point on the path between $x^\ast$ and $\hat{x}$ has value no greater than $m$, contradicting the condition that the two are distinct unique global optima.
This proof matters because ordinary loss functions cannot generally be expressed as simple convex functions. They can contain many local optima with zero gradient, making convergence by plain gradient descent difficult.

Momentum supplements gradient information with inertia that carries optimization through a local minimum. Think of a metal ball continuing to roll under gravity.

Descending against the gradient resembles a metal ball rolling down a mountain. Unlike a real ball with momentum, plain gradient descent can stop where the gradient reverses and fail to cross a ridge.

If downward velocity is preserved and the local optimum is not too deep, momentum can carry the ball over the ridge.

In code:

v = mu * v - learning_rate * dx
x += v

Velocity $v$ receives an additional force opposite the gradient. A gradient aligned with the current direction increases speed; one opposing it decreases speed. $\mu$ controls how much velocity from the previous batch affects descent on the next batch. Stable training generally uses a large value, commonly $0.9$.

Nesterov momentum

Standard momentum adds the gradient at the current location to momentum and uses the resulting velocity to update weight parameters.

Nesterov momentum instead moves by momentum first and then takes a gradient step. Because it evaluates the gradient after moving ahead, it is called a look-ahead method.

x_ahead = x + mu * v
v = mu * v - learning_rate * dx_ahead
x += v

This creates a causality issue because it requires a new gradient at the future, velocity-shifted location. To express it using the gradient at the current location, modify the update as follows.

v_prev = v # back this up
v = mu * v - learning_rate * dx # velocity update stays the same
x += -mu * v_prev + (1 + mu) * v # position update changes form

Annealing learning rate

Keeping one learning rate throughout optimization may be undesirable. Early on, far from the minimum, a large step accelerates progress; near the optimum, it can make loss oscillate and prevent convergence. Learning-rate annealing gradually reduces it. In PyTorch, this is handled by a scheduler.

Step decay

One pass of optimization over the full training set is an epoch. Step decay reduces the learning rate every few epochs. With step size $5$ and factor $0.1$, it becomes one-tenth as large every five epochs.

Exponential decay

$$\alpha = \alpha_0 e^{-kt}$$

This schedule exponentially decays initial learning rate $\alpha_0$ at epoch $t$ according to hyperparameter $k$. In practice, one usually specifies the decay factor $e^{-k}$ rather than $k$ directly. A factor of $0.1$, for example, reduces the rate tenfold each epoch.

$1/t$ decay

$$\alpha = \frac{\alpha_0}{1+kt}$$

This schedule follows the inverse curve $y=\frac{1}{x}$, whose asymptote is $y=0$.

Tasks may use many other schedules or loss terms designed to prevent overfitting.

Second-order methods

Optimization is not limited to gradient-based methods. Moving a fixed step opposite the gradient corresponds to a first-order Taylor approximation. Let $f \in \mathcal{C}^n$ be continuous and $n$ times differentiable in input $x$. Around a known value $f(a)$, we can approximate $f(x)$ as follows.

$$f(x) = f(a) + \frac{(x-a)}{1!}f^{(1)}(a) + \frac{(x-a)^2}{2!}f^{(2)}(a) + \cdots + \frac{(x-a)^n}{n!} f^{(n)}(a) + o((x-a)^n)$$

Here, notation $o(g(x))$ means

$$\lim_{x \rightarrow 0,~x \in \Omega} \frac{\vert\vert f(x) \vert\vert}{\vert g(x) \vert} = 0$$

A first-order approximation of $f(x)$ is

$$f(x) \approx = f(a) + (x-a)f^\prime (x)$$

A linear approximation has no minimum around the current function value, so it gives no explicit criterion for choosing step size $\alpha$. Consider the figure below.

The gradient identifies a descent direction, but because a linear function has no minimum, it cannot determine the step size. Knowing the Hessian—the second derivative—lets us approximate $f(x)$ quadratically and potentially optimize more efficiently. Newton's method uses

$$x^{(k+1)} = x^{k} - {\Delta f(x)}^{-1} \nabla f(x)$$

Here, $\Delta$ denotes the second-order gradient $\nabla^2$. The update finds where the derivative of the second-order Taylor approximation equals zero. It has two major problems.
First, the computation is difficult. For multidimensional $f(x):\mathbb{R}^n\rightarrow\mathbb{R}$, the Hessian is

$$\nabla f(x) = \begin{bmatrix} \frac{\partial f}{\partial x_1}(x) \newline \frac{\partial f}{\partial x_2}(x) \newline \vdots \newline \frac{\partial f}{\partial x_n}(x) \end{bmatrix}$$ $$\Delta f(x) = \nabla \left( \nabla f(x) \right) = \begin{bmatrix} \frac{\partial^2 f}{\partial x_1^2}(x) & \frac{\partial^2 f}{\partial x_1 \partial x_2}(x) & \cdots & \frac{\partial^2 f}{\partial x_1 \partial x_n}(x) \newline \frac{\partial^2 f}{\partial x_2 \partial x_1}(x) & \frac{\partial^2 f}{\partial x_2^2}(x) & \ddots & \frac{\partial^2 f}{\partial x_2 \partial x_n}(x) \newline \vdots & \vdots & \ddots & \vdots \newline \frac{\partial^2 f}{\partial x_n \partial x_1}(x) & \frac{\partial^2 f}{\partial x_n \partial x_2}(x) & \cdots & \frac{\partial^2 f}{\partial x_n^2}(x) \end{bmatrix}$$

Deep-learning functions have enormous input and output dimensions. A $1024 \times 1024$ image already has roughly one million input dimensions, requiring a Hessian matrix with about $10^{12}$ entries. Second, the global optimum of the Hessian-based quadratic approximation may point away from the original function's optimum. A first-order approximation at least guarantees that a sufficiently small negative-gradient step moves toward a better point.

The optimum of a second-order approximation may instead be a maximum, and poor initialization can make the loss diverge. For these reasons, deep learning generally uses gradient descent rather than higher-order Taylor optimization.

Various methods of optimization

Mini-batch stochastic gradient descent is the classic method applicable to nearly every network, but it is often inefficient. Simple image-classification or regression losses may be easy to optimize, but special modalities, generative networks, difficult parameter or hyperparameter landscapes, and multimodal objectives are harder. Plain SGD can be slow and makes extensive ablations with multiple regularization terms difficult. In short, robust optimization is hard.

Per-parameter adaptive learning rate methods

Optimizer research therefore became a branch of deep learning. Conventional updates compute an output gradient and move by one learning rate in the opposite direction. Momentum helps avoid local minima but does not fundamentally change the method. Manually assigning a learning rate to every parameter would require too much hyperparameter search and is not efficient. The following optimizers adapt how gradients apply to each parameter.

Adagrad

# Assume the gradient dx and parameter vector x
cache += dx**2
x += - learning_rate * dx / (np.sqrt(cache) + eps)

Let $dx$ be the gradient for parameter vector $x$. As its name suggests, Adagrad adapts gradients by accumulating squared $dx$ values in a cache. During each update, it divides $dx$ by the square root of that cache plus $\epsilon$ to prevent division by zero.
This reduces the effective learning rate for large gradients, common early in training, and increases it relatively for small gradients. Because the cache grows continuously, however, learning can eventually stop.

RMSprop

cache = decay_rate * cache + (1 - decay_rate) * dx**2
x += - learning_rate * dx / (np.sqrt(cache) + eps)

RMSprop uses a decay rate to reduce the influence of old gradients in its normalization. Since the effective learning rate does not decrease monotonically as in Adagrad, training does not terminate prematurely.

Adam

m = beta1*m + (1-beta1)*dx
v = beta2*v + (1-beta2)*(dx**2)
x += - learning_rate * m / (np.sqrt(v) + eps)

Adam adds momentum to RMSprop. m tracks momentum, while v acts as the cache. The most common defaults are beta1 = 0.9 and beta2 = 0.999, though SGD with Nesterov momentum can outperform Adam for some networks.
Adam also uses a bias-correction mechanism. In the first few steps, initialized vectors m and v are biased toward zero; the following code corrects that bias.

# t is your iteration counter going from 1 to infinity
m = beta1*m + (1-beta1)*dx
mt = m / (1-beta1**t)
v = beta2*v + (1-beta2)*(dx**2)
vt = v / (1-beta2**t)
x += - learning_rate * mt / (np.sqrt(vt) + eps)

Hyperparameter optimization

Parameters update by the chain rule while training optimizes the loss, and among many parameter optimizers, Adam is the most widely used. Hyperparameters cannot be optimized in the same training loop and must be adjusted manually. Common hyperparameters include:

1. Initial learning rate
2. Learning rate policy
3. Regularization strength(loss penalty strength)

Hyperparameter sensitivity differs by task, and exhaustively evaluating every value is impractical, so efficient search matters.

Implementation

Large networks take so long to train that hyperparameter tuning alone can require days or weeks. The methods below depend on code design and network architecture and do not apply universally.
A common setup runs a training worker that repeatedly samples hyperparameters and optimizes a model. At every epoch, the worker evaluates validation performance and saves either the latest checkpoint or the network with the best metric, such as loss or accuracy. PyTorch users commonly save .pth or .pt files with torch.save().

best_loss = 1e9
for epoch in epochs:
    for iteration in training_dataloader:
    # Optimize network parameters on the training dataset
    # Run validation after one training epoch
    avg_loss = 0.0
    with torch.no_grad():
        for iteration in validation_dataloader:
        # Compute the trained network's average loss on the validation dataset
        if avg_loss < best_loss:
            best_loss = avg_loss
            torch.save(model.state_dict(), "best_model.pt")

The PyTorch example above saves the network when its validation loss is smallest. Although criteria vary by task, most workflows save parameters with the best performance metric, while some also save the latest epoch. In K-nearest neighbors, we could choose hyperparameter $k$ by rotating training and validation subsets and averaging results. Deep learning cannot freely do this: once a sample participates in parameter training, it loses its meaning as an unseen validation sample for measuring generalization. Given enough data, a deep-learning algorithm therefore uses one fixed validation set for performance evaluation.

Hyperparameter ranges

Unlike parameter optimization, hyperparameter search has no differentiable objective and therefore no direct target value. We must define a feasible set of candidate solutions, since testing every real number is impossible.

learning rate = 10 ** uniform(-6, 1)

The learning rate controls how far each parameter moves opposite its gradient; the goal is to find a value that reduces loss efficiently.
The code searches on a logarithmic scale, since optimal rates commonly lie between $10^{-6}$ and $10^1$. In practice, learning rates must change by orders of magnitude to produce meaningful differences.

Research suggests that random search can be more effective for important hyperparameters than log search, a form of grid search over fixed intervals (reference). Some hyperparameters matter far more than others, and random sampling covers more values of those important dimensions.
This does not mean using arbitrary values without limits. Search should remain within a range whose feasibility can be evaluated, as explained next.

Careful with best values on border

The chosen search range itself may be poor. For example:

learning_rate = 10 ** uniform(-6, 1)

Suppose this range produces the following loss curve over candidate values.

Because $10^{-6}$ has the smallest observed loss, one might call it optimal, but a best value on the boundary suggests that the true optimum lies outside the range. Real curves are rarely this simply monotonic, but always verify that the search interval covers the relevant space.

Stage search from coarse to fine

Imagine playing a higher-or-lower game in which a friend chooses an integer from $1$ through $100$. Unless you can read minds, you would not begin with a boundary guess such as $2$ or $99$, though the answer could be $1$.
When the optimum is unknown, begin with a coarse search to locate its region, then search that region more finely. Guess $50$ first; if the answer is “higher,” guess $75$ and narrow the interval from $(0,100)$ to $(51,100)$. Hyperparameter search works the same way: after finding a promising region, progressively narrow the search within it.

Hyperparameter tuning need not train on the full dataset. Use fewer epochs or a subset of the training data to check feasibility first.

Bayesian hyperparameter optimization

Algorithms for systematic hyperparameter optimization continue to be studied. Their main idea is to balance exploration of a search space that grows with the number and range of hyperparameters against exploitation of promising regions. Libraries include Spearmint, SMAC, and Hyperopt, though carefully designed random search has reportedly remained competitive or better in final performance.

Evaluation with Ensembles

One way to improve performance and generalization is to train multiple networks and average their predictions at test time, an ensemble. Performance generally rises with the number of models, though not indefinitely, and ensembles benefit more from diverse models than from nearly identical ones. Several construction strategies follow.

Same model, different initialization

Train the same architecture with the same optimal hyperparameters but different parameter initializations. The drawback is that all model diversity depends on initialization; if training converges to similar parameters anyway, the ensemble gains little.

Top models discovered during cross-validation

Select the top $n$ models found across hyperparameter configurations. This guarantees more diversity, but includes suboptimal rather than strictly optimal models. It is easy to implement, yet a weak model can distract from the others.

Different checkpoints of a single model

If training is expensive, ensemble checkpoints from different stages of one strong model. This provides limited diversity because every member comes from the same trajectory, and some checkpoints may be incompletely optimized.

Running average of parameters during training

Maintain an exponentially decaying average of network parameters across iterations. This smooths parameters and is effective when a bowl-shaped convex objective makes the final network oscillate around the optimum.

Ensembling is powerful, but its fatal drawback is slow inference on test data. One alternative is “dark knowledge,” which distills ensemble knowledge into a single network.

Closing Thoughts

This post examined the parts of deep-learning training that must change: loss, parameters, and hyperparameters. Apart from task-specific mathematics and domain knowledge, this completes most of the foundational material needed for deep-learning research.
I wrote it by revisiting an old CS231n review from my Naver blog and the CS231n course notes. At the time, I thought I understood the material very well; looking back, I can see many places where I merely pretended to know more than I did. Perhaps when I reread this post later, it too will reveal an embarrassingly incomplete understanding.