Training a network usually means finding good parameters from a large volume of data. If the class or domain encountered at inference has little or no training data, however, a network must learn from very few examples. This is the motivation for low-shot learning, where a “shot” means one training example provided to the network.

Few-Shot Learning

Conventional deep learning does not generalize reliably from only a few examples. Its many parameters are usually stabilized by large training sets, and regularizers such as extensive data augmentation are introduced to prevent overfitting. Producing an optimized network for a new task with almost no data can therefore seem impossible. Typical applications of few-shot learning (FSL) include:

• Character generation, where copyright or other restrictions may prevent collection of many examples of a character
• Advanced robotics
• Tasks whose samples are inherently difficult to obtain, including drug discovery, low-resource translation, and cold-start recommendation

In the classic definition of machine learning, a computer program learns from experience $E$ on a task $T$ and improves according to a performance measure $P$. For an image-classification task, a large set of labeled images supplies experience $E$, and classification accuracy is performance $P$. Few-shot learning asks what happens when $E$ is severely limited—or, in deep-learning terms, when supervision for $T$ is scarce.

FSL combines the small available supervision set $E$ with existing prior knowledge so that learning remains feasible. For character generation, supervision may be a handful of images and prior knowledge may describe how character parts and their relationships are constructed. For drug-toxicity discovery, a known molecule with a similar structure can provide prior knowledge for a new one. For image classification with few examples per class, a network trained on other classification tasks can play that role.

Special cases include one-shot learning, with one trainable example, and zero-shot learning, with no example at all for the target task. Zero-shot learning needs information from another modality—attributes or word embeddings, for instance—to transfer some supervision into inference.

People infer from background knowledge. Someone who has never seen a cat may initially call one a dog. If a friend immediately corrects them, they acquire the distinction from that single experience and can identify the next cat correctly. In deep-learning language, one error and loss update would generalize the task from one example. Humans can do this; optimizing a deep network on a single image is not normally an adequate solution. The familiar “learn from scratch” paradigm is unavailable.

Few-shot learning instead organizes one trainable episode around a support set. An $N$-way, $K$-shot problem has $N$ classes in the support set and $K$ examples per class. The query set contains the samples on which the episode's adapted model must make predictions.

In ordinary supervised classification, examples from every class are divided between training and testing. Few-shot evaluation instead pretrains on only some classes—five in the illustration—and constructs a support set from five entirely different classes. Only a few examples, such as three, are supplied for each new class. The examples not used for adaptation form the query set and measure few-shot performance.

Common approaches include:

• Transfer learning
• Data augmentation, transformation, and synthesis
• Meta-learning, or “learning to learn”
• Metric-based approaches and embedding learning
• Multi-task learning
• Generative learning

Transfer and multi-task learning were covered in my earlier article on deep-learning training paradigms. Here we focus on meta-learning as learning a training strategy, and embedding learning as metric-based inference, followed by a brief look at generative methods.

Using Prior Knowledge for FSL

Transfer learning alone often fails with only a few samples per class. Even fine-tuning only the last layer creates a high risk of overfitting. FSL therefore obtains experience from related problems and reuses it as prior knowledge. The prior can broadly come from:

1. Data
2. Similarity
3. The learning process itself

Prior Knowledge About Data

The simplest way to derive prior knowledge from data is to augment the support set.

Suppose the support set used for training is $D_{train}$. A learned transformation $t$ converts a sampled pair $(x_i,y_i)$ into an augmented pair $(t(x_i),y_i)$. Unlabeled or incompletely labeled samples $(\bar{x},-)$ can receive pseudo-labels predicted from $D_{train}$. Samples $(\hat{x}_j,\hat{y}_j)$ from related datasets can also be combined by an aggregator $t$; mixup is one example. The transformer $t$ acts as prior knowledge that expands each data category.

Prior Knowledge About Learning

Transfer learning belongs to this category, but meta-learning goes further by learning how to learn. To solve target task $T$, a meta-learning algorithm trains across batches of tasks $T_i$, with the crucial requirement

$$T_i \cap T = \emptyset.$$

It uses the experience of solving many distinct $T_i$ to adapt well to unseen $T$. A simple analogy is taking many mock exams to prepare for the real college entrance exam. Results across multiple training tasks optimize the learning algorithm itself. That algorithm then trains a model, and the model optimized by the optimized procedure is evaluated on the target task.

A base set contains many episodes, each split into a support set and a query set. Meta-learning the algorithm uses these base episodes. Afterward, an $N$-way, $K$-shot support set from the real target task adapts the network, and the target query set evaluates it. By updating across batches of episodes, the model learns a procedure that remains stable and effective when an unseen few-shot classification task appears.

Model-Agnostic Meta-Learning (MAML) is one such algorithm. It performs two levels of backpropagation. Its aim is to learn parameters from which a network can adapt to a new classification task quickly with only a few samples.

Conceptually, MAML uses two instances of the same neural architecture. One is the learner, whose initialization is meta-learned so it can support new tasks. The other is a task-specific adapter, copied from the learner and optimized rapidly on an episode. Inner and outer optimization use different learning rates. The procedure is:

• Initialize the learner randomly.
• Repeat over meta-training episodes until the desired epoch count or convergence:
  • Sample a batch of episodes.
  • Initialize each adapter from the learner's parameters.
  • For a fixed number of inner steps, train the adapter on the episode's support set.
  • Evaluate the adapted parameters on the episode's query set to obtain a meta-loss.
• Differentiate the query-set meta-loss through adaptation and update the learner's initialization with the meta-gradient.

For each task, adapter $f$ is copied from meta-learner $M$ and initialized with parameters $\Theta$. The support set rapidly fine-tunes $f$. The adapted model then evaluates the query set; its loss updates the meta-parameters, including $\Theta$. During meta-training, MAML learns an initialization from which each adapter can optimize a new few-shot task quickly. During meta-test inference, the unseen task still permits gradient-based adaptation on its support set, but the shared learner initialization itself remains fixed. Because the method can wrap many kinds of neural network, it is model-agnostic.

Prior Knowledge About Similarity

Similarity-based algorithms learn compact representations—embeddings—whose vectors are robust to within-class variation while preserving relationships among classes. This is called a metric-learning or embedding-learning approach. A similarity function pulls related samples closer and pushes different samples apart, much like contrastive learning.

A representative example is the Siamese network; other examples include matching, prototypical, and relation networks.

Suppose the support set in the figure is three-way, three-shot. Encoder $E$ maps its nine samples to embeddings $g$. In a well-learned embedding space, the three samples from each class cluster together, while different classes remain separated. A query image maps through $E$ to embedding $f$, and its predicted class is the nearest neighboring cluster.

A Siamese network contains two sibling branches with shared weights and optimizes the contrastive loss

$$(1-Y)\frac{1}{2}(D_W)^2 + Y\frac{1}{2} (\max (0, m-D_W))^2.$$

Here $Y\in\{0,1\}$ indicates whether the inputs come from different classes: $0$ for the same class and $1$ for different classes, like an exclusive OR over class identity. Margin $m>0$ means dissimilar pairs already farther apart than $m$ receive no additional penalty. Training passes similar pairs $((x_i,x_j),0)$ and dissimilar pairs $((x_i,x_k),1)$ through the branches. A balanced ratio tends to train most stably.

For $E$ examples per class and $C$ classes, the counts of possible pairs are

$$N_{same} = \binom{E}{2}C$$

and

$$N_{diff} = \binom{EC}{2}-\binom{E}{2}C.$$

A matching network follows a related idea. Let the support set contain $(x_i,y_i)$ with one-hot class vector $y_i$, and let $f$ and $g$ be embedding functions. For query $\hat{x}$, the classifier predicts a weighted combination of support labels:

$$\hat{y} = \sum_{i=1}^k a(\hat{x}, x_i)y_i = P(\hat{y} \vert \hat{x},~S).$$

The weighting factor is attention from the query to the support set:

$$a(\hat{x},~x_i) = \frac{e^{c(f(\hat{x}),g(x_i))}}{\sum_{j=1}^k e^{c(f(\hat{x}),g(x_j))}}.$$

Here $c$ is cosine similarity and can be interpreted as how likely two samples are to match.

This produces a similarity-weighted nearest-neighbor classifier. The Siamese network used Euclidean distance $D_W$; the matching network uses cosine similarity. It maximizes the log-likelihood of correct query classes conditioned on support set $S$:

$$\theta = \arg \max_\theta \mathbb{E}_{L \sim T}\left[\mathbb{E}_{S \sim L,~B \sim L} \left[\sum_{(x,~y) \in B} \log P_\theta (y \vert x,~S) \right]\right].$$

Zero-Shot Learning

The preceding methods require at least one example. Zero-shot learning uses none.

A person who has never seen a zebra may still know what a horse is. Hearing that a zebra resembles a horse but has stripes can be enough to recognize one. Zero-shot learning (ZSL) similarly uses known, or seen, classes to infer unseen ones. Pretrained knowledge or attributes bridge them. An attribute is represented as a kind of category vector.

A ZSL dataset contains:

• Seen classes: labeled images available during training
• Unseen classes: image classes absent from training
• Auxiliary information: descriptions, semantic attributes, or word embeddings that connect seen and unseen classes

Semantic Vectors

An attribute vector maps visual appearance into properties. Even without a training image labeled “cat,” properties such as “has a tail” and “has fur” can support the prediction.

Word vectors offer another route, using NLP to learn attribute information automatically.

A deep network maps image features into a common embedding space shared with semantic attributes—a meeting place for the two modalities. It may be the high-level visual space, a lower-level semantic space, or a newly learned intermediate space. Training uses semantic vector $v$ as supervision for a seen image $x$, learning a projection $v=f(x)$.

At inference, each new class has a specified semantic vector $v^\ast$. An unseen image $x^\ast$ is mapped to $f(x^\ast)$ and classified by its nearest semantic neighbor. If attributes are represented well, unseen classes can occupy sensible positions in the space and become distinguishable.

Standard zero-shot evaluation discriminates only among new classes at test time. Generalized zero-shot learning must discriminate both the seen training classes and unseen classes together.

Generative-Model-Based Methods

Embedding methods suffer from bias and domain shift. Projection $f(\cdot)$ trains only on seen classes, so its output is biased toward them. Domain shift also gives no guarantee that unseen image features will map correctly into semantic vectors. In plain terms, the method works only if it can extract good semantic vectors from test samples, yet zero-shot learning provides no unseen-class examples with which to ensure that.

We therefore want a zero-shot classifier to receive guidance for both seen and unseen classes during training.

One solution feeds an attribute vector—say, the attributes of a cat—into a conditional GAN. Its fake examples stand in for out-of-distribution unseen samples, allowing generated semantic features to supplement the missing unseen classes.

After training the conditional GAN, its generator is frozen. Attributes of unseen classes are supplied as conditions, and the generator produces image features for classifier training rather than requiring actual images. The visual quality of a synthetic image is not the central concern; generated features provide a class-agnostic bridge that allows the classifier to learn about otherwise unseen categories.