On Efficient Deep Neural Networks

Efficient Networks

When building a deep learning network, we first consider the size and properties of the dataset and, ultimately, the task we want to solve. Even after all that, one final question remains: can the model actually be trained with the resources I have, such as my GPUs?

Suppose someone commissions us to build a model. If capable training GPUs are available and the DNN is not intended for a low-capacity mobile or embedded device, scaling up the network architecture may be necessary to achieve better performance. The idea of an efficient network emerged from exactly this perspective: let us increase network size efficiently. The research defines three variables affecting the storage and computational complexity of a CNN.

1. Resolution: image dimensions, or feature-map size
2. Width: the number of network channels
3. Depth: the number of network layers

EfficientNet v1

EfficientNet v1 first designs a new baseline architecture through NAS, an AutoML technique. Put simply, NAS automates the search for a well-performing deep learning network and the fine-tuning of its hyperparameters. EfficientNet then scales up that baseline. Its basic unit is the MBConv, or mobile inverted bottleneck convolution block, used in MobileNet-v2.

The diagram above is a useful reference. I have a separate post on lightweight models as well. In any case, EfficientNet begins with the following baseline network.

Depth Scaling ($d$)

Of the three scaling variables, depth is the most straightforward: it scales how deep the network is. Since the arrival of ResNet, it has become fairly intuitive that deeper networks have larger receptive fields and a better chance of capturing relationships among different image features.

As seen in ResNet and VGG, however, making a network excessively deep does not produce much further improvement in accuracy. Even with more parameters, the dataset may not be large enough to optimize all those representations. Or the growing distance between input and output may simply make the optimal solution harder to reach. ResNet-1000, for example, has accuracy similar to ResNet-101. The graph also shows accuracy converging from about $d=8.0$. Its horizontal axis is FLOPS, proportional to the amount of computation, and its vertical axis is ImageNet Top-1 Accuracy.

Width Scaling ($w$)

Another option is to keep the depth fixed and make the network wider. Wider here does not refer to the spatial dimensions; it means increasing the channel count.

A network with more channels can capture fine-grained features more effectively, and because it remains shallow, it suffers less from the optimization problems of excessive depth.

Even so, a network that is too shallow ($d\downarrow$) and excessively wide ($w\uparrow$) saturates easily. As the graph suggests, performance converges at roughly $w=5$.

Resolution Scaling ($r$)

Resolution increases the spatial dimensions. As with the previous factors, a high-resolution image is advantageous for extracting fine-grained features. Yet FLOPS rises with resolution, and performance is reported to converge around $r=2.5$.

Observations

The experiments lead to two conclusions.

1. Increasing any of the factors—depth, width, or resolution—improves performance, but the marginal improvement gradually shrinks.
2. When efficiency constraints such as FLOPS and model capacity are considered, it is important to balance every dimension of the network.

This yields the following objective:

$$\begin{aligned} &\max_{d, w, r} Accuracy(\mathcal{N}(d, w, r)) \\ &\text{s.t.}~\mathcal{N}(d, w, r) = \bigodot_{i=1 \cdots s} \hat{F_i}^{d \cdot \hat{L_i}}(X_{(r \cdot \hat{H_i}, r \cdot \hat{W_i}, w \cdot \hat{C_i})}) \\ &\text{Memory}(\mathcal{N}) \le \text{target\_memory} \\ &\text{FLOPS}(\mathcal{N}) \le \text{target\_flops} \end{aligned}$$

In simple terms, $\mathcal{N}$ represents the family of CNN models being compared. The unfamiliar symbol in the second line treats every layer of a CNN scaled by $(d,w,r)$ as a function and interprets the output as a composition of operations on the input. Thus $\mathcal{N}(d,w,r)$ maps input to output, and we want that output to maximize accuracy on a supervised task, without exceeding the target memory or FLOPS.

Stated this way, though, it is only a wish to build such a model rather than a numerical formulation. We still need to quantify how $d$, $w$, and $r$ affect memory and FLOPS, and how to search within the resources available to us.

$$\begin{aligned} \text{depth: }&d = \alpha^\phi \newline \text{width: }&w = \beta^\phi \newline \text{resolution: }&r = \gamma^\phi \newline \text{s.t. }&\alpha \cdot \beta^2 \cdot \gamma^2 \approx 2 \newline &\alpha \ge 1, \beta \ge 1, \gamma \ge 1 \end{aligned}$$

Because the search scales the model upward—as motivated by the first observation—every factor must be at least one. The compound coefficient $\phi$ determines exponentially how many resources to use. FLOPS is approximately linear in network depth $d$ and quadratic in width and resolution, $w$ and $r$, so the constraint keeps total FLOPS near $2^\phi$. Four parameters ultimately have to be selected: $(\alpha,\beta,\gamma,\phi)$.

Implementation

The baseline model uses the MBConv structure from MobileNet v2 and is called B0. Its architecture is obtained through NAS, which can be understood as searching for an efficient starting point.

Next, the authors fix $\phi=1$, assume twice as many resources are available, and run a grid search over $\alpha$, $\beta$, and $\gamma$. This is the $2^\phi$ constraint described above. For B0, the optimal values are $\alpha=1.2$, $\beta=1.1$, and $\gamma=1.15$. They then hold these three values constant and test different values of $\phi$. Varying $\phi$ in this way produces EfficientNet B1 through B7.

Because $\alpha$, $\beta$, and $\gamma$ were chosen carefully at the start, the resulting networks outperform earlier ones without dramatically increasing parameter count. So blindly throwing together a larger network was not the answer after all.

The figure uses CAM to visualize which regions the network attends to for each class. Compound scaling captures more detailed regions.

EfficientNet v2

We are not finished yet. If there is an EfficientNet v1, naturally there is a v2—like movie sequels, I suppose. Anyway, EfficientNet v2 appeared relatively recently, in 2021. When EfficientNet v1 enjoyed its heyday, newer computer-vision architectures such as ViT had not yet appeared. Times changed, and a better network was needed.

EfficientNet v2 searches for its network architecture in the same way as v1. Its first contribution is a changed baseline architecture; its second is the use of progressive learning, meaning that image size is adjusted over the course of training (see PGGAN). With these changes, it reports $11\times$ faster training and $6.8\times$ better parameter efficiency than EfficientNet v1.

Fused MBConv

To describe Fused MBConv briefly, the original MBConv combines a depthwise $3\times3$ convolution with an expansion $1\times1$ convolution. Fused MBConv replaces that pair with an ordinary $3\times3$ convolution.

Not every layer is converted to Fused MBConv. The table shows the effect of replacing only parts of the EfficientNet v1 architecture—the changed parts are marked as stages. Because the authors did not have a definitive answer to where Fused MBConv should be used, they left that decision to NAS.

The resulting baseline, named EfficientNetV2-S, leads to three conclusions:

1. Use Fused MBConv extensively in the early layers.
2. EfficientNet v2 favors smaller $3\times3$ kernels than v1's $5\times5$ kernels.
3. Remove the final stride-1 stage.

The details are complicated, but the short version is that NAS found the best arrangement.

Progressive Learning

The main idea of progressive learning is to begin with small images and increase their size gradually. For example, in a ten-epoch run, the first six epochs might use $224\times224$ images and the remaining four might fine-tune on larger $256\times256$ images. Simply growing from smaller to larger images, however, caused an accuracy drop.

The authors' explanation was that when image size changes, regularization strength should change with it. Applying the same regularization without adjustment created an imbalance.

Their solution was to increase both image size and the corresponding regularization strength progressively.

In the algorithm, $R_i$ denotes regularization strength and $S_i$ image size. As the iteration advances through $M$ stages, the image size increases linearly from its initial value $S_0$, and the regularization strength does the same. The natural question is: what kind of regularization? The paper argues that this training scheme applies to most existing regularizers and tests three.

• Dropout is network-level regularization that drops some channels during training—in other words, it temporarily disconnects some nodes. The dropout rate $\gamma$, the probability of dropping each node, is adjusted.
• RandAugment performs per-image data augmentation at random. Its magnitude $\epsilon$ is adjusted.
• Mixup augments across images. Given a dog image $(x_i,y_i)$ and a cat image $(x_j,y_j)$, for example, it creates a mixed image $\tilde{x_i}=\lambda x_j+(1-\lambda)x_i$ and label $\tilde{y_i}=\lambda y_j+(1-\lambda)y_i$, where $\lambda$ is between zero and one. Here, that mixup ratio is adjusted.

After coordinating these settings, the authors obtain the following results.