What Does It Mean to Remove an Object? From Image Inpainting to Video Object Removal

A KOBA 2026 talk wrap-up — This post expands my KOBA 2026 Conference talk, “Removing Distractions from Video: Generative AI-Based Video Object Removal.” It revisits the research path behind the talk and adds work released through August 2026, including OSOR, TurboClear, and EffectLearner.

Related posts — AI Does Not Remember the World · Image Manipulation with GANs · AnimateDiff Paper Review

The question I kept returning to while preparing the KOBA talk was not how clean the output looked. It was what problem the model was actually solving. Removing a person from a photograph appears to be ordinary image inpainting: draw a mask and fill the hole. In practice, the model must reveal a background that was never observed, erase a shadow on the floor and a reflection in the window, and leave the rest of the frame untouched.

Taken seriously, the request becomes stranger. What would the scene have looked like if that object had never been there? In video, should the motion of another object change if it originally collided with the removed one? At that point, object removal is closer to counterfactual generation than pixel repair.

This survey follows the literature from classical inpainting to image and video object removal in 2026. The chronology matters, but the definition of the target matters more. I will track how the meaning of the mask and the ground truth changed, then ask what the reported evaluations still fail to measure.

Inpainting and Object Removal Have Different Targets

Standard image inpainting receives a corrupted image $I$ and a region $M$ to complete, then predicts a finished image $\hat{I}$.

$$\hat{I}=G(I\odot(1-M), M)$$

The usual training recipe applies a random mask to a complete image and treats the original as the target. The model is asked to recover what used to be inside the hole. This is a natural objective for scratches and missing regions.

Object removal crosses between two versions of a scene. The input $I^{\text{with}}$ contains the unwanted object, while the target $I^{\text{without}}$ does not.

$$G(I^{\text{with}}, M_{\text{object}}) \approx I^{\text{without}}$$

Under random-mask reconstruction, the object inside the mask was part of the correct answer. At inference time, we suddenly ask the model to forget it and invent what was behind it. A stronger generative prior can create a more convincing background, but it can also reconstruct the unwanted object or replace it with a semantically similar one.

The user-provided object mask is rarely the full edit region. Removing shadows, reflections, illumination, smoke, or ripples requires another region:

$$M_{\text{edit}}=M_{\text{object}}\cup M_{\text{effect}}.$$

$M_{\text{effect}}$ is usually not given at inference time. The object mask can no longer be interpreted as a hard boundary saying “edit these pixels.” It has become a query asking the model to find every trace associated with this object. That shift explains much of the recent literature.

2000–2022: Changing How the Hole Is Filled

Early inpainting had no generative model. Bertalmio et al. propagated lines and colors inward from the boundary. The method worked well for thin scratches but could not invent a large structure hidden behind an object.

Criminisi et al. copied similar patches from elsewhere in the image and filled strong structural boundaries first. PatchMatch made approximate patch correspondence fast enough for practical editing. These methods were effective on repeated grass, walls, and sky. They had nothing to copy when the required background was absent from the image.

Deep networks began predicting unseen content from context. Context Encoders combined a CNN with adversarial loss. Generative Image Inpainting introduced contextual attention to retrieve distant features. Partial Convolution and Gated Convolution distinguished valid pixels from holes and handled irregular masks.

LaMa used Fast Fourier Convolutions for a wide receptive field. MAT modeled long-range structure with a mask-aware transformer. RePaint repeatedly conditioned the sampling process of a pretrained diffusion model on observed pixels. The strong image prior and text conditioning of Latent Diffusion Models then became a foundation for many editing systems.

This progression dramatically improved hole completion. Yet the benchmark question remained: how plausible is the content inside the mask? Object removal must also ask whether the target disappeared and whether everything else stayed fixed. A better generator is not automatically a better eraser.

2023–2025: Teaching Removal as Its Own Task

Inst-Inpaint identified the removal target from a natural-language instruction. PowerPaint separated text-guided filling, object removal, and shape-guided generation using task prompts. The same backbone needed different conditions for filling and erasing.

Work in 2024 focused on suppressing the target's semantic influence. MagicEraser combined content initialization, prompt tuning, and attention refocusing. CLIPAway conditioned on a background-focused CLIP embedding rather than the foreground. Encoding the target too strongly can encourage the denoiser to bring it back.

One design choice in this period is particularly revealing: what should happen to the pixels inside the mask? A traditional pipeline discards them before inpainting. That makes sense when the pixels are corruption. In removal, they are the most direct evidence of what must disappear.

SmartEraser retained this evidence with Masked-Region Guidance, allowing the model to separate the true object from surrounding content covered by an oversized mask. Erase Diffusion changed the denoising path itself. If ordinary diffusion learns a path back to the original image containing the object, a removal model needs a path in which the object fades into background.

These approaches expose an awkward requirement. The model must recognize the object well enough to remove it, yet prevent that semantic representation from surviving into the output. The separation between recognition and generation remains incomplete.

Counterfactual Pairs Change the Data

Once the mismatch with random-mask reconstruction became clear, data construction moved to the center of the problem. ObjectDrop fixed the camera and scene, then photographed them before and after an object was removed. A real object-free target supervises changes in shadows, reflections, and occlusion. The paper argues that these physical effects are difficult to recover through self-supervision alone.

RORem built an iterative data engine. People labeled the outputs of an initial removal model, a discriminator learned from those labels, and the discriminator filtered another round of pairs. Three human-annotation rounds produced more than 200,000 training pairs. The initial model succeeded on fewer than half of its samples. Generating a large number of edits does not automatically produce a removal dataset.

More paired data does not close every gap. Real before-and-after captures contain small changes in lighting, foliage, and water. Synthetic pairs align perfectly but miss contact deformation and physically correct shadows. Recent datasets mix both sources because real pairs are physically richer while synthetic pairs are easier to control.

There is also no single correct completion. A paired capture may reveal the actual floor behind an object, but a different generated floor texture can still be plausible. PSNR and LPIPS penalize deviation from one captured target. Reference-free metrics can reward realism while missing a hallucination that changes the identity of the scene. Better data makes the disagreement over what counts as correct easier to see, not easier to avoid.

2026: Finding Effects Beyond the Object

Erasing only the silhouette leaves a floating shadow or a reflection of an object that no longer exists. The image may look clean at first glance while its internal relationships are wrong. Recent work expands the object mask into an effect region.

ObjectClear uses Adaptive Target-Aware Attention to localize the object and its effects, then Attention-Guided Fusion to combine the input and generated output. Its OBER dataset contains 2,878 captured real pairs and 10,000 synthetic pairs. The controlled capture idea in ObjectDrop became larger-scale supervision for effect-aware removal.

OmniPaint learned insertion and removal as disentangled object-oriented tasks rather than prompt variants of generic inpainting. The same idea later appears in EffectErase, which trains video insertion and removal as reciprocal tasks.

Removal is therefore a spatially asymmetric optimization problem:

$$\mathcal{L} = \lambda_{\text{edit}}\mathcal{L}_{\text{edit}}(M_{\text{effect}}) + \lambda_{\text{keep}}\mathcal{L}_{\text{keep}}(1-M_{\text{effect}}).$$

An effect region that is too small leaves shadows and reflections behind. A region that is too large regenerates valid background and changes the source. Complete removal and precise preservation collide at the mask boundary, which is why recent systems combine attention, alpha maps, and input-output fusion.

The Race to One-Step Removal

Dozens of diffusion steps are expensive in an interactive editor. In 2026, several projects compressed effect-aware removal into a few steps or a single forward pass.

FlashClear distills ObjectClear with Region-aware Adversarial Distillation. FPAC prioritizes foreground tokens and caches features. The paper reports up to an 8.26× speedup over ObjectClear and 122× over OmniPaint on OBER.

OSOR formulates the task as one-step latent restoration. A single pass cannot refine a boundary over many iterations, so its occupancy-guided discriminator receives the fractional mask occupancy of each patch. The first training phase uses accurate effect masks and hard latent blending. The second uses incomplete object masks and an alpha head that predicts the true soft edit region. The SAVP filtering pipeline produces CORNE, a collection of 280,000 verified pairs.

TurboClear combines a one-step SDXL model with Region-Calibrated Distribution Matching and Learnable Spatial Fusion. It distills the teacher's generative distribution inside the effect region while applying paired preservation losses to the background. A learned gate merges a removal stream with an identity stream. The paper reports up to 40.04× less computation than ObjectClear and up to 665× less than Flux-based OmniPaint.

The speed numbers need context. The papers use different backbones, resolutions, hardware, and baseline implementations, and often report a maximum speedup. Such measurements are valuable within an ablation, but they do not form a fair leaderboard when copied from separate tables. End-to-end latency should also include segmentation, mask refinement, and compositing.

One deeper question remains. A one-step model compresses a teacher's long sampling path, including the teacher's habit of missing an effect or modifying valid background. I would like to see not only average quality but where the failures move under rough user masks, rare effects, and out-of-domain scenes.

What Should an Object-Removal Metric Measure?

No single inpainting score covers the full task. At least three outcomes should be separated.

ReMOVE is a reference-free metric designed to consider both object erasure and background continuity. It attempts to distinguish removal from replacement. Paired benchmarks such as OBER allow direct comparison to an object-free target, but no such target exists for an arbitrary open-world photograph. Reference-free metrics and human evaluation are unavoidable there.

The definition of the effect mask is the part I find hardest to settle. A cast shadow can often be localized by pixel difference. Indirect illumination, fluid motion, and another person's reaction have no clean boundary. If a model edits a wider area and makes the scene more plausible, did it find the effect correctly or violate preservation? A benchmark should state the allowed scope of intervention before collapsing quality into one score.

The Two Sources of Video Inpainting

Temporal consistency is the first concern in video. Independent frame-wise inpainting flickers and changes structure. Reducing flicker does not solve video object removal by itself.

Pixels inside the removed region can be divided into known and unknown. A pixel occluded in the current frame is known if the corresponding background appears in another frame; flow or feature correspondence can propagate that observation. A region hidden throughout the entire clip is unknown and must be generated.

For aligned masks $M_1,\ldots,M_T$, the region occluded in every frame is their intersection:

$$M_{\text{unknown}}=\bigcap_{t=1}^{T}M_t, \qquad M_{\text{known}}=\left(\bigcup_{t=1}^{T}M_t\right)\setminus M_{\text{unknown}}.$$

ProPainter combines flow completion, feature propagation, and a mask-guided sparse video transformer. It represents the line of work that moves observed information first and uses a transformer for longer context.

Semantically Consistent Video Inpainting with Conditional Diffusion Models argues that propagation cannot recover novel content absent from every frame. DiffuEraser explicitly separates propagation of known pixels, generation of unknown pixels, and temporal consistency, combining an image prior from Stable Diffusion with a video diffusion prior.

The boundary is less crisp in real footage. A pixel visible elsewhere is not always reliably known. Illumination changes, non-rigid surfaces deform, and water moves, making correspondence uncertain. A system needs confidence to choose between a noisy observation carried by flow and a generated estimate. Propagation is better understood as another prior with a different error distribution than as a source of guaranteed truth.

Temporal consistency is not an independent certificate of quality either. A wrong brick pattern can remain perfectly stable across every frame. Conversely, correctly propagated observations can look wrong when their boundaries and illumination do not match. Video inpainting must coordinate propagation, generation, and temporal regularization.

2025–2026: Tracking Effects Through Time

Video effects do not necessarily move with their source object. Smoke persists after separating from it, ripples travel outward, and a reflection moves according to viewpoint. Perfectly tracking the object mask can still miss the effect.

ROSE constructs paired data and effect-difference masks for shadows, reflections, lighting, and translucency. Work in 2026 divides the problem across mask robustness, semantic understanding, and multimodal conditioning.

SVOR introduces MUSE, a union over masks within temporal windows, and Denoising-Aware Segmentation. The design aims to keep the target localized when a user mask flickers or misses part of a rapidly moving object. UnderEraser distills object-effect relationships from vision foundation models into video diffusion and grounds denoising in unmasked context through frame-wise cross-attention.

Object-WIPER locates targets and effects through attention in a pretrained text-to-video DiT. It replaces removal tokens in inverted structured noise with Gaussian noise while preserving background tokens. The method avoids task-specific training, although its effect understanding is bounded by the associations already present in the pretrained video model.

EffectErase builds VOR, a collection of 60,000 real and synthetic paired videos, and learns insertion as the inverse auxiliary task of removal. GenEraser balances text and mask conditions, separating a Locator that finds effects from a Preserver that maintains pixel alignment.

EffectLearner, released in August 2026, places a VLM-based Object-Effect Reasoner before a DiT Video Eraser. It examines a target-highlighted video and summarizes the target, motion, effects, expected post-removal state, and context to preserve. Rather than memorizing a fixed list of effect categories, it targets spatially detached, weakly correlated, compositional, and evolving effects.

The role of the mask has changed along this path. It was once an operational boundary: fill these pixels. It now behaves more like a query: find every trace associated with this target. The model chooses the final edit region. That makes complete removal easier, but it also transfers responsibility for over-editing to the model.

Beyond Pixels: Physical Causality

Removing every shadow and reflection still may not produce the video that would have happened without the object. If a falling ball bounced off a platform, deleting the platform while preserving the bounce creates impossible motion. A vehicle that turned to avoid a person may have followed another trajectory in a world without that person.

VOID: Video Object Insertion and Deletion expands appearance editing into a counterfactual over physical interactions. It identifies regions affected by the removed object and regenerates downstream dynamics with a video diffusion model. Counterfactual data built with Kubric and HUMOTO provides supervision.

$$p(V^{\text{without }o}\mid V^{\text{with }o},M_o) \neq p(\text{background pixels}\mid \text{masked video}).$$

The left side is a distribution over worlds in which the object was absent. The right side fills holes in an observed video. They are close when the object merely occludes the background. They diverge when the object changes light, fluids, collisions, or another agent's behavior.

Evaluation becomes harder again. The same real event cannot be filmed twice with and without the object. A simulator can generate exact counterfactual pairs but introduces a simulation-to-reality gap. We must also decide whether an alternative is physically plausible and how much of the original event's identity it should preserve. Causal removal is where video inpainting begins to look like a world model, and where benchmark construction is most difficult.

Questions the Survey Leaves Open

Is There One Correct Image?

Paired capture provides strong supervision, but an occluded background admits many plausible completions. Pixel distance to one ground truth may punish the generative model for producing a valid alternative. Reference-free realism, on the other hand, can ignore scene identity. Removal, plausible completion, and preservation should be reported separately.

How Far Does an Effect Extend?

Shadows and reflections are relatively easy to agree on. Illumination, fluids, and another person's behavior have no clean boundary. Higher effect recall increases the risk of modifying valid content. An editing interface may need to expose the user's allowed intervention region separately from the larger effect region inferred by the model.

Where Did One-Step Failures Move?

Fewer sampling steps improve average latency but remove iterative correction. I am more interested in the failure distribution under incomplete masks, difficult boundaries, and long-tail effects than in one average score. Mobile and interactive evaluations should include segmentation and compositing in end-to-end latency.

What Should a Video Model Preserve?

Pixel consistency, scene identity, and physical plausibility do not always point in the same direction. Changing downstream dynamics intentionally breaks some temporal correspondences. Future evaluations must ask not only whether the video is stable, but whether the motion that should have changed actually changed.

Closing Thoughts

Image inpainting began by propagating boundary structure into a hole. Patch copying gave way to CNNs, transformers, and a generative diffusion prior. Object-removal research then added task-specific conditioning, counterfactual pairs, effect supervision, and explicit preservation paths. OSOR and TurboClear now attempt to compress this compound objective into one-step inference.

Video contains both observed pixels that can be propagated and unknown pixels that must be generated. Coordinating them over time is already difficult. Recent work searches for shadows, reflections, illumination, and fluid effects, then goes further by regenerating downstream motion caused by the removed object.

In the KOBA talk, I described video object removal as a combination of propagation and generation. After adding the newest papers, that description no longer feels complete. Removal also requires inferring the reach of an effect and, sometimes, simulating an event that never happened.

Calling it “erasing without a trace” misses the hardest part. The model has to answer a more demanding question: if the object had never existed, what should change, and what must remain the same?

References

• Image Inpainting
• Region Filling and Object Removal by Exemplar-Based Image Inpainting
• PatchMatch
• Context Encoders
• Generative Image Inpainting with Contextual Attention
• Partial Convolutions
• Gated Convolution
• LaMa
• MAT
• RePaint
• High-Resolution Image Synthesis with Latent Diffusion Models
• Inst-Inpaint
• PowerPaint
• ObjectDrop
• MagicEraser
• CLIPAway
• ReMOVE
• RORem
• SmartEraser
• Erase Diffusion
• ObjectClear
• OmniPaint
• FlashClear
• OSOR
• TurboClear
• ProPainter
• Semantically Consistent Video Inpainting with Conditional Diffusion Models
• DiffuEraser
• ROSE
• SVOR
• UnderEraser
• Object-WIPER
• EffectErase
• VOID
• GenEraser
• EffectLearner