Related posts — Video Inpainting Starts with the Reference Frame · Same Mask, Different Problem · Why General Generative Models Struggle with Precise Editing

My first version of this workflow sounded straightforward: choose one good reference frame, inpaint it as an image, then propagate that result through the video. It is also an easy system to build. Pick a frame with little object motion, plenty of visible background, and no blur. Remove the object with OSOR or MaskFlow. Use that output as the starting point for temporal propagation.

A few dynamic shots are enough to break the assumption. A background that is plausible in one frame is not necessarily the right background for the entire shot. Camera motion changes parallax. Water and foliage carry their own motion phase. Occlusion order changes as foreground and background objects move. The more accurately we propagate a plausible but incorrect image fill, the more stably the wrong scene can persist through time.

That is why I want to redefine the reference frame in this article. A useful reference is not one representative image. It is closer to temporal evidence: for every target pixel and region, which observation at which time can actually be trusted?

When a Reference Is Actually Useful

Video Inpainting has an advantage over a single image because other frames provide additional observations. An object may cover a wall in the target frame, then move away and reveal the wall later. There is no reason to ask a generative model to imagine that wall if the camera has already captured its pixels.

References matter most in a few recurring cases:

1. A large object covers the background for a long interval. Adjacent frames contain no evidence, but a more distant frame finally reveals the surface.
2. Camera motion creates disocclusion. Pans, dollies, and handheld motion expose regions that were hidden before.
3. The scene contains unique texture that is difficult to synthesize. Paintings, signs, irregular brickwork, and set decoration are better recovered from captured pixels than from a generic prior.
4. An edit needs a stable identity. For object replacement or insertion, a keyframe can define appearance and geometry as a visual condition.
5. The mask is large. Local spatial context cannot determine the missing structure, while observations from other times can reduce the ambiguity.

If the background is never visible anywhere in the video, there is no reference to select. Picking the least bad nearby frame does not add an observation; it repeats the same occlusion from another time. The phrase “correct reference” is also awkward for water, fire, smoke, or crowds because the state itself keeps changing. In those cases, texture evidence and motion evidence need to be treated separately.

Reference Selection Is a Per-Pixel Evidence Problem

Let the video be $X=\{x_t\}_{t=1}^{T}$ and its masks be $M=\{m_t\}_{t=1}^{T}$. Write the correspondence that maps a target position $p$ in frame $t$ to reference frame $r$ as $W_{t\rightarrow r}(p)$.

A reference pixel is useful only if several conditions hold:

• its mapped location is visible rather than masked in the reference;
• forward and backward flow do not contradict one another;
• the path does not cross an occlusion;
• color, depth, and local appearance support the claim that both pixels belong to the same surface.

We can compress these checks into a confidence score:

$$q_{r\rightarrow t}(p) = v_r(p_r)\, c_{\mathrm{flow}}(p,r,t)\, c_{\mathrm{occ}}(p,r,t)\, c_{\mathrm{app}}(p,r,t)\, c_{\mathrm{depth}}(p,r,t),$$

where $p_r=W_{t\rightarrow r}(p)$ and $v_r(p_r)$ is one only when that reference pixel is observed. The best reference is therefore not global. It changes with the target pixel:

$$r^\ast(t,p)=\arg\max_r q_{r\rightarrow t}(p).$$

Inside the same mask, a distant past frame may reveal the left wall, while the next frame contains the sharpest evidence for the floor. Some regions may remain invisible in every frame. Only those residual regions should fall back to a generative prior.

Temporal distance is only one term in the score. A nearby frame is easier to align, but it is more likely to contain the same occlusion. A distant frame may reveal clean background while changing pose, illumination, and parallax. A single heuristic such as “use the nearest clean frame” cannot handle both cases.

What Goes Wrong with One Image-Inpainted Reference

Selecting one frame, inpainting it, and warping it through the shot can work well for a static scene. The problem is that a reference error does not remain local to that image.

1. Spatially Plausible, Temporally Wrong Texture

Foliage, water, screens, and defocused lights have temporal phase. Image Inpainting generates a still texture that fits the spatial context of one frame. Propagating it can make the fill slide with the camera or remain frozen while the surrounding scene moves.

2. Parallax and Depth Ordering

With camera translation, foreground and background move at different rates. A single homography or inaccurate optical flow creates doubled edges and tearing around depth boundaries. If another object sits in front of the removal target, the system must synthesize the background behind the target while retaining that foreground object. A 2D mask alone does not fully express this layer ordering.

3. One Hallucination Becomes the Source for the Entire Clip

Suppose an inpainted painting bends a line slightly or continues a brick pattern incorrectly. That may be a small defect in one frame. Successful reference propagation repeats the same defect everywhere. Temporal consistency goes up while scene fidelity goes down. The absence of flicker does not make the reconstruction correct.

4. Repeated Warping Accumulates Blur

Passing color through one frame after another repeatedly resamples sub-pixel values. Fine texture softens and boundaries spread. The longer the chain from a distant reference, the worse the accumulation becomes. Reference selection and the propagation algorithm have to be evaluated separately.

5. The Object's Footprint Extends Beyond Its Mask

Removing a person can leave a shadow, reflection, or translucent trace. A harder case is a person kicking a ball or colliding with another object. If the person had not existed, the later trajectory should also change. There is no clean reference-frame solution to this problem. The desired background has become a counterfactual scene.

I Put the References into VACE

The value of a reference can look larger on a paper diagram than it does in a working pipeline. To check that gap, I ran the same 49-frame clip under three conditions. The resolution was 432×240 at 24 fps. The first run gave VACE only the mask sequence. The other two added a single image-inpainted reference produced by OSOR or MaskFlow. I follow the pipeline's notation here: the middle image is zero-based frame index 24, or the 25th image if counted from one.

This sample also brought me back to the title of the article. It is the DAVIS bmx-trees sequence, and VACE clip index 24 matches source frame 40. I reloaded the original RGB frames and dense object masks, then aligned indices 18–30 to frame 24. The area hidden at frame 24 becomes genuinely visible at other times.

The figure below is not a strip of identical image coordinates. Each source frame is homography-warped into the reference coordinate system, and only pixels outside that frame's object mask remain bright. The bright regions in the second and third panels are therefore known pixels captured by the original video, not fills produced by an image model.

By mask coverage alone, this scene did not require image generation. That does not mean one distant frame could be copied wholesale. Nearby past and future frames occlude different places, and camera motion creates different parallax for the graffiti wall and the grass in front of it. The nearest-known mosaic in the lower middle still contains blur and boundary error. Evidence being available is not the same as placing that evidence correctly in the target frame.

“The reference frame is not one image” is therefore literal here. Even inside this small mask, the upper graffiti is revealed earlier by nearby past frames, while the bicycle-wheel and grass regions receive earlier evidence from future frames. A single image-inpainted patch averages or ignores those separate observations. A video pipeline instead needs a source frame, visibility state, and alignment confidence for each pixel.

At normal playback speed, the three VACE outputs did not look dramatically different. The mask is relatively small, and enough graffiti, grass, and pavement remains around it for the video model to infer a plausible fill. This is a case VACE can already handle from the mask alone. Adding a reference did not visibly improve the structure of the entire clip.

Frame 24 tells a different story when viewed slowly. Both reference-conditioned outputs suddenly move closer to their image-inpainted input at that frame, then return to the trajectory established by the video model. The GIF below shows indices 18–30 at roughly one quarter of the original speed, using the same crop around the removed cyclist.

I used a simple residual to check whether that small visual jolt was measurable. Optical flow first aligns frames $t-1$ and $t+1$ to $t$. I then measure the MAE between the current frame and the average of those two aligned neighbors. The region $\Omega$ is the object mask expanded by 15 pixels so that the seam is not excluded from the measurement.

$$E_t = \frac{1}{|\Omega|} \sum_{p\in\Omega} \left| I_t(p) - \frac{ \widetilde{I}_{t-1\rightarrow t}(p) + \widetilde{I}_{t+1\rightarrow t}(p) }{2} \right|.$$

This does not mean the reference controls the whole clip. Its effect is quite local. Away from index 24, the reference-conditioned runs differ from mask-only by roughly 3.2–4.2 MAE inside the expanded mask. At index 24, that value jumps to 11.85. Whatever VACE's internal conditioning path, the output behaves as if the supplied image were a strong anchor at that frame.

The OSOR–MaskFlow difference appears more clearly in texture than in the temporal metric. A simple Sobel gradient magnitude inside the same mask is 52.0 for the OSOR reference and 74.4 for the MaskFlow reference. At VACE frame 24, the corresponding values are 51.2 and 74.3. These are not universal image-quality scores, but they are useful as a sharpness proxy for the same scene and support. VACE did not repair OSOR's blur; it preserved it. It also preserved MaskFlow's relatively sharper lines.

That is still not enough to declare MaskFlow the better video reference. The temporal-residual peaks of the two reference runs are effectively identical. A sharper keyframe can still pop if its texture does not lie on the motion trajectory of adjacent frames. Conversely, a blurred reference may score as stable under some metrics simply because it contains less high-frequency detail. Image quality and temporal compatibility are different axes.

This is one clip without a clean-background ground truth. Optical-flow alignment and MAE do not replace perceptual evaluation either. The narrower conclusion is more useful: creating a good image reference and placing that reference naturally inside a video are separate problems. If success means reproducing the supplied keyframe, frame 24 worked. If success means continuous motion, that strong reproduction became the source of the discontinuity.

Research Direction 1: Do Not Generate What the Video Already Shows

Flow-guided Video Inpainting moves valid pixels from other frames into a target. Deep Flow-Guided Video Inpainting first completed flow inside the hole and then propagated pixels. E2FGVI jointly trained flow completion, feature propagation, and content hallucination. ProPainter combines global image-domain propagation with local feature-domain propagation and uses flow-consistency checks to restrict transfer to reliable regions.

The benefit is concrete: the model does not redraw pixels that were captured elsewhere in the video. The limitation is just as concrete. Inaccurate flow or a large mask with no correspondences produces blur and incomplete fills. Transformers can expand temporal context, but memory and compute increase with the number of frames.

RGVI separates the roles more explicitly. It first propagates known pixels within the video. A large image generator completes the still-missing area of a reference frame. The generated reference is then propagated across the sequence, and a final per-frame stage handles residual holes.

RGVI is the paper closest to the workflow I originally had in mind: create a high-quality inpainted image reference first. The important difference is the order. It aggregates video evidence and verifies propagation reliability before reference generation. That is the reverse of choosing an arbitrary frame and immediately asking an image model to fill it.

Research Direction 2: Let Diffusion Handle Only the Unsupported Residual

The center of a large mask may have no correspondence in any frame. That area must eventually be generated. VipDiff first uses flow to bring valid reference pixels into the target, then treats that partial result as a constraint during reverse diffusion. Observed content constrains the sample; an image diffusion prior fills what remains unknown.

The last point also needs caution. Once a generated frame is added back to the evidence pool, captured pixels and model samples no longer have the same provenance. A production pipeline should record whether every propagated pixel came from the camera or from an earlier generation stage. Treating both with the same confidence allows a small hallucination to harden across a long sequence.

Research Direction 3: Propagate a Keyframe with Motion Context, Not Image Warping Alone

VideoRepainter lets a user edit a keyframe with an established image model, then uses an image-to-video diffusion model to carry the modification through the remaining frames. It targets creative editing more than restoration, but the implication is relevant here: even when a keyframe defines appearance, a video prior still has to solve motion and temporal coherence. A static result is not enough.

Long videos add another problem. AVID uses Temporal MultiDiffusion to process a long sequence as overlapping clips. Averaging overlaps improves transitions but does not stop identity from drifting between the beginning and the end. AVID therefore lets every frame query keys and values retained from the middle frame.

MultiPaint, published in July 2026, combines text-, image-, and keyframe-guided modes through dynamic frame masking and extends the setup to multiple objects and long videos. The reference is becoming less of a separate input image and more of a scheduling question: which frames should remain visible as conditions, and when?

The 2026 Shift: A Background Reference Is Not Enough

The most noticeable change in work available through September 2, 2026 is the meaning of the mask. Classical Video Inpainting usually treats the object's pixels as a hole and fills what is behind them. Recent work asks the system to remove the object's footprint on the scene as well.

Object-WIPER combines a user mask with query tokens such as “duck and its reflection.” Cross-attention and self-attention localize tokens corresponding to reflections, shadows, mirror images, and translucent effects. It inverts the video into structured DiT noise, reinitializes foreground tokens, and copies saved background token values during denoising to preserve scene fidelity.

EasyOmnimatte frames the issue as layered decomposition. It assigns effect-sensitive DiT blocks and alpha-quality refinement to complementary experts, separating foreground and associated effects. SVOR directly addresses abrupt motion, defective masks, and the target pixels lost during temporal mask downsampling. Both point to the same requirement: before selecting a reference, the system has to define what is being removed consistently over time.

Appearance restoration is not the final boundary. VOID, posted to arXiv in April 2026, treats object removal with collisions and downstream interactions as counterfactual generation. If the removed object had never existed, how would the motion of other objects have changed? The affected region guides a video diffusion model toward that alternative outcome. VOID is a preprint, not a peer-reviewed conference result like the CVPR papers above, and should be read as such. It still exposes a hard boundary of reference-based inpainting: no captured frame contains evidence of a world in which the object was never present.

How I Would Rebuild the Workflow

After reading this line of work, I would not leave reference selection as one stage immediately before Image Inpainting.

I would begin with the mask sequence, then separate the object mask, possible shadows and reflections, and occlusion layers. References would be retrieved per target region rather than per target frame. The score would combine visibility, forward-backward flow error, sharpness, photometric difference, parallax, and depth boundaries instead of temporal distance alone.

High-confidence captured pixels would be propagated first. To reduce blur, I would accumulate source-to-target correspondences and sample from the original source once rather than repeatedly warping RGB through intermediate frames. Only residual holes that remain unsupported by every reference would reach an image or video generative model.

Generated pixels would keep different provenance from observed pixels. When reused as references, they would receive lower confidence and be checked against independent anchors from distant parts of the shot. For dynamic texture or large camera motion, I would split a shot into motion regimes and use multiple anchors rather than force one keyframe across the whole clip.

The final evaluation would not stop at frame quality. At minimum, it should separate:

• pixel preservation in known regions;
• spatial quality and blur inside the mask;
• optical-flow warp error and flicker;
• long-range texture and identity drift;
• residual shadows, reflections, and foreground effects;
• stability in the final Video Inpainting propagation stage.

In the first article, I wrote that Image Inpainting quality has to be judged through the final Video Inpainting result. I would revise that statement slightly now. A good reference image is not an answer on its own. We need to track which pixels came from observation, which were generated, and how long each piece of evidence remains valid along the temporal axis.

Conclusion

For a static scene, choosing one frame, inpainting it well, and propagating it is still a useful workflow. The failure begins when the same assumption is applied to a dynamic video. Spatial quality in the reference does not capture motion phase, parallax, occlusion, or causal change.

The field is not only replacing flow with Transformers, diffusion models, and DiTs. It is redistributing responsibility. Propagate pixels that were actually observed. Generate only unsupported residuals. Use a keyframe as temporal conditioning rather than a bitmap to copy. Expand an object mask to include effects and, eventually, interaction regions.

Video Inpainting therefore does not need one “best reference frame.” It needs routing: distinguish observation from generation, connect each region to evidence from a different time, and invoke the generative prior only when that connection fails. Image Inpainting remains important, but inside a dynamic video it becomes one piece of evidence in a larger temporal context.