Related posts — Video Inpainting Starts with the Reference Frame · Same Mask, Different Problem

In the previous post, I ran OSOR and MaskFlow on the same frame. OSOR looked more conservative by the numbers. Its RGB MAE outside the mask was lower, and more pixels were bit-identical to the source. The crop told a different story. The generated region was heavily blurred. MaskFlow changed a wider area, but its sofa and floor were reconstructed much more naturally.

That result was more awkward than I expected. I could not call the model that changed fewer pixels the better Inpainting model. I could not drop the more plausible image straight into a composite either. Preservation and generation quality were not moving in the same direction.

This article goes back to the formulation rather than another list of model features. I want to understand why a strong generative prior can repair a scene and cause unwanted changes at the same time, and why adding a mask and a prompt does not by itself guarantee pixel locality.

This Is Not an Impossibility Theorem

I am not arguing that a general model must fail at precise editing. With a condition that fully identifies the task, enough capacity and data, and an explicit spatial constraint, one model could handle many downstream tasks well.

Our current conditions are not that complete. A prompt compresses intent. A mask gives an approximate location. Training pairs rarely remove every incidental change outside the requested edit. Latent diffusion converts the source into a lossy representation and sends it through a global sampling path. In this setup, locality is not a property that automatically appears when training succeeds. It is a separate constraint that has to be inserted and verified.

Inpainting Is Constrained Posterior Sampling, Not Ground-Truth Recovery

Let the original image be $x^\ast$, let $A$ read the known pixels, and let $y$ be the observed values. The Inpainting input can be written as

$$y=Ax^\ast.$$

For a binary mask $m$, we can view $A$ as $\operatorname{diag}(1-m)$. Every image that preserves the known pixels belongs to the constraint set

$$\mathcal C(y)=\{x\mid Ax=y\}.$$

The problem is that $A$ does not observe the hidden region. Any change $v\in\operatorname{Null}(A)$ leaves the known pixels untouched:

$$A(x+v)=Ax.$$

Infinitely many backgrounds therefore produce the same observation $y$. A single photograph cannot reveal the exact wood grain behind a sofa or the true wall texture behind a defocused crew member. A larger model does not create a missing measurement.

A diffusion model fills that absence with a prior. The posterior of an inverse problem is commonly written as

$$p_\theta(x\mid y)\propto p_\theta(x)\,p(y\mid x).$$

The prior $p_\theta(x)$ pulls the result toward images that looked natural in the training distribution. The likelihood $p(y\mid x)$ pushes away solutions that contradict the observation. Diffusion Posterior Sampling combines a diffusion prior and a measurement likelihood in this way for general inverse problems.

Nothing in this equation says that Inpainting recovers the actual hidden scene. It finds a high-prior sample among the candidates that do not contradict the visible pixels. That is where plausibility and recovery part ways on paintings, reflections, and defocus.

This also explains the advantage of Video Inpainting over a single image. When another frame reveals a previously hidden region, we gain another observation and shrink $\operatorname{Null}(A)$. The space that the generative prior has to guess becomes smaller.

“Leave Everything Else Untouched” Is Not in the Diffusion Objective

A standard denoising objective trains a network to predict noise $\epsilon$ from a noisy sample $x_t$:

$$\mathcal L_{\mathrm{diff}} = \mathbb E_{x,\epsilon,t} \left[ \left\|\epsilon-\epsilon_\theta(x_t,t,c)\right\|_2^2 \right].$$

An editing model adds a source image $x_s$, an instruction $c$, and a mask $m$ to the condition:

$$\mathcal L_{\mathrm{edit}} = \mathbb E \left[ \left\|\epsilon-\epsilon_\theta(x_t,t,x_s,c,m)\right\|_2^2 \right].$$

The optimum approximates the score of the conditional data distribution. The following equality, however, is not automatically part of that loss:

$$(1-m)\odot x_{\mathrm{out}} = (1-m)\odot x_s.$$

Define the pixel-space preservation constraint as

$$\mathcal C(x_s,m) = \{x\mid(1-m)\odot x=(1-m)\odot x_s\}.$$

Strict preservation requires the support of the output distribution itself to remain inside this set:

$$\operatorname{supp} p_\theta(x_{\mathrm{out}}\mid x_s,c,m) \subseteq \mathcal C(x_s,m).$$

If the training pairs contain tone shifts, small geometry changes, or regeneration outside the mask, a model can minimize its diffusion loss and correctly learn those changes as valid edits. Providing a mask as a condition is not the same as projecting every sample back onto the constraint set.

RePaint makes the distinction concrete. It uses a pretrained unconditional DDPM as the prior, then alters the reverse process by resampling the unmasked region from the known input information. This is closer to “put this observation back during sampling” than “look at the source and try to preserve it.”

The boundary does not solve itself. Generated and known regions can follow different trajectories and meet with a seam. A hard constraint can guarantee locality; it does not guarantee a natural transition.

A General Editor Learns a Mixture of Tasks

A general editing dataset contains more than removal. Addition, replacement, attribute changes, and global restyling share the same model. Let the task be $\tau$ and the full condition be $z=(x_s,c,m)$. The training distribution can be viewed as a mixture:

$$p_{\mathrm{mix}}(x_{\mathrm{out}}\mid z) = \sum_\tau p(\tau\mid z) p_\tau(x_{\mathrm{out}}\mid z).$$

The mixture itself is not a problem if $z$ fully identifies the desired task and scope. Real inputs do not. “Remove the cushions” does not say whether matching cushions elsewhere should also disappear. An object mask does not specify whether contact shadows belong to the target, or whether a foreground prop should be preserved while a person behind it is removed. The background hidden inside the mask is absent from the condition altogether.

That ambiguity remains as conditional entropy:

$$H(X_{\mathrm{out}}\mid X_s,C,M)>0.$$

Adding more tasks does not always increase this entropy. Detailed conditions and enough capacity can let tasks improve together. With finite data and a shared representation, however, multiple valid outputs can occupy the neighborhood of the same condition. The model then has no evidence that identifies the one intervention the user intended. Some of what we call output randomness is missing information in the condition, not sampling noise alone.

The training pairs reinforce the problem. A valid Object Removal counterfactual should keep everything except the target fixed. Pairs generated by a general editor are scalable, but incidental changes made by the upstream editor enter the supervision. Strict task-specific filtering gives cleaner support, then tends to discard rare defocus, reflection, and depth-overlap cases that fail the verifier.

Guidance Already Exposes the Two Objectives

SDEdit adds noise to an input, then denoises it with a pretrained SDE prior. A small starting noise level $t_0$ keeps the original strokes. As $t_0$ increases, the prior has more room to produce a natural image and more room to leave the input behind.

Faithfulness and realism are not one scalar quality here. Changing $t_0$ moves the result between them. A stronger model does not remove the axis; the sampling procedure still decides how far to trust the input and the prior.

InstructPix2Pix writes the split more directly with separate image and text guidance terms:

$$\begin{aligned} \tilde\epsilon_\theta =\;&\epsilon_\theta(z_t,\varnothing,\varnothing)\\ &+s_I\left[\epsilon_\theta(z_t,c_I,\varnothing)-\epsilon_\theta(z_t,\varnothing,\varnothing)\right]\\ &+s_T\left[\epsilon_\theta(z_t,c_I,c_T)-\epsilon_\theta(z_t,c_I,\varnothing)\right]. \end{aligned}$$

The scale $s_I$ pushes the result toward the source image. The scale $s_T$ applies the instruction more strongly.

These are different directions. Stronger source guidance does not automatically improve instruction following. A stronger edit does not preserve the source by default. That is why preservation and edit strength appear as separate controls in a general editor.

Latent Preservation Is Not Pixel Preservation

Most large image models now run diffusion in a VAE latent rather than directly in pixels. With an encoder $E$ and decoder $D$, the source follows

$$z_s=E(x_s),\qquad \hat{x}_s=D(z_s).$$

For a lossy autoencoder, $D(E(x_s))=x_s$ does not generally hold. Latent Diffusion itself begins by looking for a perceptual-compression point that reduces computation without discarding too much detail.

Even if a method keeps the unmasked latent identical to the source latent,

$$(1-m_z)\odot z_{\mathrm{out}} =(1-m_z)\odot E(x_s),$$

the following pixel-space equality does not follow:

$$(1-m)\odot D(z_{\mathrm{out}}) =(1-m)\odot x_s.$$

The decoder's receptive field crosses the mask boundary, and compression discards high-frequency detail. Latent blending is a useful way to reduce unwanted change. It is not a proof of bit identity. A pixel bypass that composites the source back into the final output is required for that kind of contract.

The bypass creates another cost. If an object's shadow or reflection extends beyond the mask, copying the original pixels back preserves the removal artifact. Expanding the effect mask repairs more of the scene and reduces the preserved area. Enforcing a constraint and finding its correct scope are separate problems.

Re-reading OSOR Blur and MaskFlow Drift

The earlier experiment is not a benchmark that proves this formulation. One sample cannot identify the cause of an architecture. It can still help classify the failures we observed.

OSOR combines a narrow removal task with latent composition and preserved more pixels outside the mask. Its one-step restoration, however, produced an over-smoothed fill. Sofa and floor texture collapsed. It stayed closer to the constraint and found a poor posterior sample for the region we needed.

MaskFlow used a broader editing prior and reconstructed the structure and texture more naturally. It also changed other cushions and produced a larger outside-mask difference. It found a better sample under the prior, but its intervention scope did not fully match our constraint.

Neither model won on every axis because they were not merely better and worse optimizers of the same objective. They combined the prior and constraint differently, and their training distributions had different support.

Why One Larger Model Does Not End the Problem

The trade-off between generality and downstream performance is not only a capacity problem. At least four issues remain in the current formulation:

1. The true hidden background is not identifiable from one image.
2. A denoising objective does not automatically enforce a preservation constraint.
3. Prompts and masks do not fully specify the task or intervention scope.
4. Latent compression and decoding disturb pixel identity again.

This does not mean the trade-off is permanent. More observations, explicit depth and object relations, task-separating conditions, constraint-aware samplers, and pixel bypasses can shrink the feasible set substantially. The point is that a stronger prior and a stronger constraint do not replace each other.

The next article will take that premise back into the production pipeline. Outside-mask MAE alone misses OSOR's blur. A plausible fill alone misses MaskFlow's semantic spillover. The practical question is how to separate those failures with metrics and quality gates, when to route a frame to another model or human review, and how each image error grows after it enters Video Inpainting propagation.