AnimateDiff

Background to the Paper

The motivation behind the paper can be summarized as follows.

1. Existing text-to-image (T2I) models have improved considerably through strong diffusion priors. Personalization methods such as DreamBooth and LoRA have also enabled increasingly precise and controllable image generation.
2. In particular, personalization has allowed many customized models to be shared in communities such as Hugging Face without requiring massive training resources.
3. Beyond personalizing objects or styles, however, research on adding specific motion to a T2I model—a video or motion prior—remains limited. Diffusion models require especially large amounts of training, so methods that reduce that burden are urgently needed.

What Is T2I Model Personalization?

In text-to-image models, personalization was originally defined as the goal of obtaining a model fine-tuned to a particular style or object.

DreamBooth

The goal is to preserve a particular object's appearance while composing a variety of scenes around that object.

Conventional T2I models learn prompt–image pairs from vast datasets and thereby acquire a prior for high-quality image generation.

They can generate many appearances sharing the meaning of the word “dog,” but cannot focus specifically on one particular dog.

No matter how precisely the text description is written, the appearance of that particular object keeps changing.

This problem persists even when the language-model and vision-model embeddings are mapped into a shared space, as illustrated by failures across the models below.

DreamBooth takes this as its target problem and fixes the visual characteristics to be generated for a particular word, such as a dog or a watch.

Anticipated Problems

We want to implant an object prior in a diffusion model—that is, add images of the desired object to the output domain.

But what happens if we fine-tune for one object using a few-shot dataset? Predictably, the risk of mode collapse and overfitting increases, potentially damaging the original model's performance.

Earlier methods therefore make a conservative choice: instead of allowing the model to learn the subject prior directly, they define the task only so that it follows the target distribution. As a result, they cannot fully personalize the object.

With a carefully designed diffusion-model loss—the expression below—we can learn information about the object while preserving the existing prior.

$$\mathbb{E}_{\mathbf{x},\,\mathbf{c},\,\boldsymbol{\epsilon},\,t} \!\Bigl[ w_t \bigl\lVert \hat{\mathbf{x}}_{\theta}\!\bigl(\alpha_t \mathbf{x} + \sigma_t \boldsymbol{\epsilon},\,\mathbf{c}\bigr) - \mathbf{x} \bigr\rVert_2^{2} \Bigr]$$

Defining an Identifier

Learning images of a particular object also requires a text description that makes it unique.

(e.g.) [that exact] [dog] / [my personalized] [cat]

In the example above, dog and cat are nouns referring to ordinary objects. By placing a special identifier before the noun and conditioning on the resulting text, we want to train the model to produce the exact object we want.

A problem arises if we randomly use existing English words such as unique or special. The token must disentangle itself from the prior it previously learned—forgetting the visual properties that the word originally generated—and attach itself to the object being personalized. This is difficult to train and should be avoided.

One proposed solution is to insert random text such as xy5syt00 and use it like a serial number.

Oddly enough, even a random string may be split by the tokenizer into tokens with strong priors. In such cases it is not very different from using an ordinary word, so this can also be a problem.

The eventual solution was to find and use rare tokens directly. This is a kind of reverse engineering: randomly sample token IDs, measure their entanglement with existing priors, and decode the token with the lowest entanglement back into a word.

For the T5-XXL tokenizer, words decoded from token IDs between roughly 5,000 and 10,000 reportedly worked best.

Training Considerations

Language drift: Carelessly training the language model can destroy representations it acquired from a large text corpus.

Reduced output diversity: Incorrectly training an image-generation model can impair the diversity of its images.

$$\mathbb{E}_{\mathbf{x},\,\mathbf{c},\,\boldsymbol{\epsilon},\,\boldsymbol{\epsilon}',\,t} \!\Bigl[ w_t\, \bigl\lVert \hat{\mathbf{x}}_{\theta}\!\bigl(\alpha_t \mathbf{x} + \sigma_t \boldsymbol{\epsilon},\,\mathbf{c}\bigr) - \mathbf{x} \bigr\rVert_2^{2} \;+\; \lambda\,w_{t'}\, \bigl\lVert \hat{\mathbf{x}}_{\theta}\!\bigl(\alpha_{t'} \mathbf{x}_{\mathrm{pr}} + \sigma_{t'} \boldsymbol{\epsilon}',\,\mathbf{c}_{\mathrm{pr}}\bigr) - \mathbf{x}_{\mathrm{pr}} \bigr\rVert_2^{2} \Bigr]$$

The second term is a regularization term. Samples are drawn in advance from the pretrained model and used as a source dataset during training, just as in a conventional domain-adaptation method.

LoRA (Low-Rank Adaptation)

LoRA is a plug-and-play training method that leaves the original model untouched and conditions it in a lower-dimensional space than the existing weights. It uses few parameters without altering the base model's performance. The paper is based on the idea that learning an implicit space in a low-rank region does not hurt performance.

$$h = W_0 x + \Delta W x = W_0 x + BAx$$

In the expression above, only $BA$ is trained efficiently. The goal is to obtain the effect of fine-tuning through residual learning.

Training Framework

AnimateDiff is fundamentally based on a text-to-image diffusion model. Many well-known T2I architectures exist, including GLIDE and DALL-E, but the most widely used high-quality example is Stable Diffusion, an LDM, whose lineage can be summarized as follows.

The original DDPM uses attention on a $16 \times 16$ feature map inside its ResNet to pass timestep information. ADM expanded attention across resolutions from $32 \times 32$ down to $8 \times 8$ and searched for the optimal architecture by adjusting the number and dimensions of attention heads. Beginning with ADM, diffusion models started to outperform GANs in earnest.

Subsequent work studied how to inject text-embedding information effectively into the attention layers of the ADM architecture. Representative examples include GLIDE, DALL-E, and Imagen.

LDM took a somewhat different path. Rather than operating directly on images, it encodes them into latents with a model such as VQ-GAN and predicts pre-decoding image embeddings—not images—from noise, reducing computation. For conditioning, earlier methods generally pooled text embeddings and inserted them into the architecture's attention along with positional embeddings from timestep $t$. LDM instead adopts a true Transformer structure that mixes self-attention and cross-attention. Latent-space computation reduces the overall cost and makes it possible to focus more heavily on conditioning from text embeddings.

Most T2I references in AnimateDiff likewise come from personalized models. Because there was a period when nearly every T2I model was created by LoRA-tuning an LDM, it is more accurate to view the method in this paper as a Transformer architecture rather than an ablated U-Net. Stable Diffusion extends and further develops this approach.

AnimateDiff is trained through a three-stage pipeline.

1. Close the gap between the image and video domains (additional module).
2. Learn a prior for motion (additional module).
3. Personalize the model for a desired motion (additional module).

Stages 2 and 3 use similar structures for learning motion information, but stage 2 learns generic video characteristics—temporal information—whereas stage 3 creates a model for a specific motion at generation time. Stages 1 and 3 therefore apply LoRA as fine-tuning, while stage 2 pretrains a Transformer structure.

Alleviate Negative Effects

Viewed frame by frame, images and videos are not fundamentally different. An image can be regarded as a video containing one static frame.

Unlike ordinary images, however, individual video frames are often imperfect. Object or camera motion introduces motion blur; encoding and decoding compression can smear pixels; and frames may contain watermarks or other artifacts.

This creates distribution misalignment between the image and video domains. Attempting to learn a motion prior without correcting it causes two problems.

1. It damages the pretrained diffusion model's text–image conditioning space.
2. It cannot train on images outside the pretrained diffusion model's distribution.

The model therefore attaches an adapter to the conditioning Transformer block—the attention modules—between ResNet blocks, as shown on the left. The expression makes clear that this is trained with LoRA.

$$Q = \mathcal{W}^Q z + \operatorname{AdapterLayer}(z) = \mathcal{W}^Q z + \alpha \cdot AB^\top z$$

Learn Motion Priors

Once the adapter has aligned the distribution of the diffusion-model block's output, the next step is to learn temporal information.

The preceding stage sampled individual frames from a video dataset and familiarized the domain adapter with images that look like video frames. It aligned the domain only at the image level, not at the video level, so the diffusion model still applies only to a single image.

This training stage is added specifically to learn motion characteristics.

Much of the concept of extending a diffusion model into the video domain was implemented in the Make-A-Video paper. The simplest method treats each video frame as an image. For a batch size $b$ and $f$ frames, this means processing a batch of $b \times f$ images.

Because that structure cannot learn a separate prior for video motion, the model adds a temporal Transformer architecture that learns only the motion component. Unlike a Transformer that processes ordinary video, the motion-prior module operates while ignoring the spatial dimensions, treating them as $b \times h \times w$. Since this is a conditioning component, zero initialization and residual connections ensure that it learns only residual motion information. The overall design appears to borrow from Make-A-Video. The excerpt below is based on the Versatile Attention implementation in the GitHub source.

def forward(self, hidden_states, encoder_hidden_states=None, attention_mask=None, video_length=None):
        batch_size, sequence_length, _ = hidden_states.shape

        if self.attention_mode == "Temporal":
            d = hidden_states.shape[1]
            hidden_states = rearrange(hidden_states, "(b f) d c -> (b d) f c", f=video_length)
            
            if self.pos_encoder is not None:
                hidden_states = self.pos_encoder(hidden_states)
            
            encoder_hidden_states = repeat(encoder_hidden_states, "b n c -> (b d) n c", d=d) if encoder_hidden_states is not None else encoder_hidden_states
        else:
            raise NotImplementedError

        # ... code for computing hidden states omitted

        # linear proj
        hidden_states = self.to_out[0](hidden_states)

        # dropout
        hidden_states = self.to_out[1](hidden_states)

        if self.attention_mode == "Temporal":
            hidden_states = rearrange(hidden_states, "(b d) f c -> (b f) d c", d=d)

        return hidden_states

Because this section does not appear to use cross-attention, the Temporal Attention component seems to exist solely to enforce temporal consistency among latents generated by the preceding layers.

In other words, it assumes that the individual frame images are already substantially complete and uses temporal attention to learn only the temporal information between frames.

Customize motion prior

The first two stages are enough to add a T2V prior to a T2I model. To specialize a model for a particular motion—such as a camera movement—the paper proposes tuning it with only 20–50 reference videos of that motion. The earlier stages turned a personalized T2I model into a personalized T2V model; stage 3 turns that into a personalized + motion-personalized T2V model. Nothing unusual is required: LoRA is attached to the motion module added in stage 2 and trained.

Experiments

Training

The Domain Adapter can be tuned with the standard Stable Diffusion training objective. Training the Motion Module and LoRA is similar, except that learning motion requires frame information within the training range, so the objective is

$$\mathcal{L} = \mathbb{E}_{\mathcal{E}\!\left(x_{0}^{1:f}\right),\,y,\,\epsilon^{1:f}\sim\mathcal{N}(0,I),\,t} \!\left[ \left\| \epsilon - \epsilon_{\theta}\!\left(z_{t}^{1:f},\,t,\,\tau_{\theta}(y)\right) \right\|_{2}^{2} \right].$$

The empirical distribution observed by the objective is also updated over frames 1 ~ f. All parameters except those in the component being trained are frozen.

Quantitative Evaluation

A CLIP metric measures alignment with text, while a user study evaluates practical perceived quality.

Inference Results

Because most modules are trained in LoRA form—with the exception of the temporal Transformer that learns the motion prior—ablations can be performed by changing their $\alpha$ values. Since image quality in the video domain is noticeably worse than in the image domain, quality can be seen to decline gradually as the video influence increases.