Related posts — Video Inpainting Starts with the Reference Frame · What It Means to Remove an Object

In the previous post, I described how I chose an image model for producing reference frames in a broadcast Video Inpainting pipeline. General-purpose editors had stronger generative priors, but they did not guarantee pixel alignment with the source. Purpose-built Inpainting models were more conservative, yet failed on complex backgrounds and ambiguous targets. I ended up using OSOR as the default path and keeping MaskFlow for cases the removal model could not resolve.

That practical decision left a more interesting question open. OSOR, SmartEraser, ObjectClear, and MaskFlow all accept an image and a mask. Their interfaces look similar, but their papers assign very different meanings to that mask. It can be a hole to fill, a query that identifies a removal target, or a constraint on the sampling trajectory itself.

This article compares the papers rather than running the models again. I read them along four axes:

• whether the model can see the original pixels inside the mask;
• whether a user-provided object mask is treated as the final edit region;
• where in the pipeline the source background is preserved;
• how the source-target training pairs are obtained.

The Same Mask Can Optimize a Different Task

A purpose-built Inpainting model begins with locality. The mask limits where new content is generated, while the rest of the image is preserved. The weakness is semantic: once the pixels inside the hole have been removed from the input, the model may not know what it is meant to erase. Recent Object Removal work adds that information back by retaining target pixels or learning a separate object-effect region.

A general editor starts at the other end. It is good at interpreting an instruction and reconstructing the scene around the requested change. Its noise or vector-field prediction, however, usually spans the entire image. A mask can be present as conditioning without making the outside region immutable. MaskFlow adds locality by putting the mask inside the flow formulation itself.

Seen this way, Inpainting and Editing are not two fixed categories. Each is borrowing the property it originally lacked. Convergence at the interface, however, does not mean that the training signals have converged as well.

SmartEraser: Hiding the Pixels Also Hides the Target

The conventional mask-and-inpaint condition can be written as

$$p(\hat{x}\mid m, x\odot(1-m)).$$

Pixels inside the mask are replaced by a placeholder or noise. The model reconstructs the hole from the surrounding context. That is reasonable for damage restoration. In Object Removal, it also discards the most direct evidence of what should disappear. If the only visible context is a road, the model may fill the hole with another car rather than reconstructing an empty road.

SmartEraser changes the condition to $[m,x]$:

$$p(\hat{x}\mid m,x).$$

The central idea behind Masked-Region Guidance is almost disarmingly simple. The model receives the complete image, not an image with the masked content removed. The mask now means something closer to “remove the target inside this region” than “redraw every pixel inside this region.” If a user draws a loose mask, the model can still inspect the original texture and any overlapping content to decide what should remain.

This formulation cannot be trained with ordinary random-mask reconstruction. Because every target pixel is still visible in $x$, the model could learn a shortcut and simply copy the masked content. SmartEraser therefore builds Syn4Removal by pasting foreground instances onto other backgrounds and using the pre-composite background as ground truth. The paper reports roughly one million triplets. Erosion, dilation, convex hulls, ellipses, and bounding shapes are mixed during training to cover realistic user masks.

The construction solves target identity and loose-mask ambiguity more directly than conventional Inpainting. It also exposes the prior of synthetic pasting. The clean background is perfectly aligned and easy to obtain, but defocus, contact shadows, reflections, and foreground occlusion are difficult to reproduce faithfully. Keeping target pixels tells the model what the object is; it does not automatically reveal every effect that object had on the scene.

ObjectClear and OSOR: The Object Mask Is Not the Final Boundary

Treating $M_{object}$ as a hard edit boundary leaves shadows and reflections behind. Asking the user to paint all of those effects creates the opposite problem: the expanded mask may include foreground content that must remain untouched. Recent removal models separate the input mask from the region eventually modified.

$$M_{edit}=M_{object}\cup M_{effect}$$

ObjectClear feeds the complete source image $I_{in}$ to SDXL-Inpainting rather than a masked image. This resembles SmartEraser, but the treatment of effects is more explicit. A CLIP visual embedding of the object crop and a fixed text prompt are inserted into cross-attention. The resulting attention map is supervised against $M_{fg}$, an annotation that covers both the object and its visual effects. The paper calls the mechanism Adaptive Target-Aware Attention, or ATA.

The final-step attention map becomes a compositing mask at inference. Instead of accepting the whole generated frame, Attention-Guided Fusion alpha-blends it with the source using a soft object-effect estimate. The same attention that recognizes the target becomes the spatial gate that preserves the background.

There is a useful asymmetry here. ObjectClear can predict an edit region wider than the input mask while changing less of the background. Generation and acceptance are separated: the model may render broadly, but the attention map decides which generated pixels survive. In the paper's ablation, adding AGF raises background PSNR from 28.80 to 35.50. The dependency also creates a clear failure mode. If attention misses an effect, fusion will preserve it. Target recognition and boundary estimation share the same map.

OSOR handles the problem with a separate alpha head and a two-phase curriculum. Phase I uses an effect-aware mask for both conditioning and supervision, followed by hard blending in latent space. Because one-step restoration cannot repair boundary artifacts over a long denoising trajectory, OSOR also introduces an occupancy-guided discriminator. Each discriminator patch is supervised by the fraction of the mask it contains rather than by a binary inside-outside label.

Phase II deliberately corrupts the conditioning mask. It may retain only the object core, erode or shift the mask, or drop holes from it. The alpha head must still recover the full effect-aware extent $\hat{\alpha}$:

$$z_{out}=\hat{\alpha}\odot\hat{z}_0+(1-\hat{\alpha})\odot \bar{z}.$$

The user mask is a condition; $\hat{\alpha}$ is the region used for the final latent composition. This is why the predicted alpha can expand toward a shadow or reflection even when the input contains only an object mask.

Both ObjectClear and OSOR read an object mask as a query from which an effect region should be inferred. ObjectClear reuses cross-attention for effect estimation and fusion. OSOR emits a dedicated alpha output and trains it with incomplete-mask supervision. The one-step design is attractive for latency, but it makes the quality of boundary supervision and the compressed restoration step correspondingly more important.

MaskFlow: Put the Mask Inside the Editor's Generative Path

The first three methods start from a dedicated removal model and add target semantics or effect awareness. MaskFlow moves in the opposite direction. It keeps the semantic editing capability of Qwen-Image-Edit-2511 while constraining the tendency of a general editor to change content outside the requested region.

An instruction-based editor normally predicts a vector field over the whole spatial representation. Supplying a mask as another condition does not, by itself, split the probability path. For a source latent $x_S$, a target latent $x_1$, and an editable mask $m$, MaskFlow defines

$$x(t)=m\odot(\alpha(t)x_1+\beta(t)\epsilon) +(1-m)\odot(\alpha(t)x_S+\beta(t)\epsilon).$$

The masked region travels toward the target, while the unmasked region follows a source trajectory on the same time axis. Instead of merely asking attention to look at a mask image, the formulation decides which spatial locations follow the generated distribution and which follow the source. The loss is also normalized by mask area so that small edits are not overwhelmed by large regions.

When a generated foreground and a preserved background follow different trajectories, color and gradient seams can appear at the boundary. Soft-Poisson de-seaming is not a final pixel-space cleanup pass. At every sampling step, it refines the estimated vector field with a gradient-domain objective. The center of the region stays close to the generated edit, while the transition zone is progressively anchored to source features.

The ablation separates the contributions. On MEData, the paper reports an FID of 29.85 for base QwenImage-Edit-2511. Adding the mask-aware path reduces it to 20.51; Soft-Poisson reduces it further to 19.90. LPIPS moves from 0.2070 to 0.1074 and then 0.1047. Within this experiment, the largest gain comes from changing the probability path, not from boundary refinement alone.

This still should not be read as a proof of bit-identical pixels outside the mask. The paper reports background MSE and LPIPS rounded to 0.0000 for MaskFlow on its MEData evaluation. That is a metric under a particular dataset and precision, not an API contract that every output pixel is copied from the source. A VAE and a finite-step solver remain in the path. MaskFlow directly addresses the locality problem from the previous post, but it does not make downstream pixel-difference validation obsolete.

Running OSOR and MaskFlow on the Same Frame

Paper tables alone did not tell me which model should go first in a compositing pipeline. I therefore kept the base.jpg and alpha mask from the first post fixed, then ran OSOR and MaskFlow separately. The input and both outputs are 1920×1306, so I computed RGB absolute differences without resizing or registration. I defined alpha < 128 as the edit region. The delta below is amplified 4× for visibility; the cyan line marks that boundary. The mask covers about 5.25% of the frame.

The measurements were as follows. An identical pixel outside the mask has the same value as the source in all three RGB channels. Difference > 5 means that the pixel's mean absolute RGB difference exceeds 5.

Neither model regenerated the whole frame in the way the general editing APIs did in the first post. More than 96% of the pixels outside the mask are bit-identical, and OSOR is slightly more conservative on every locality measure in this sample. The deltas add an important qualification, however. The outside-mask differences do not spread diffusely across the wall or lighting. They cluster on other orange cushions beyond the cyan boundary. Both models followed target similarity beyond the spatial mask. The error here is better described as semantic spillover than as global background drift.

The crop makes it clear that a lower pixel difference is not the same as better generation quality. OSOR preserves more of the area outside the mask, but the sofa and floor it generates are heavily blurred. High-frequency detail in the wood grain and fabric is smeared, and the exposed surface becomes unnaturally flat. The boundary does not jump out, but that smooth transition appears to have been obtained by over-smoothing the structure inside it.

MaskFlow redraws a broader silhouette and has a higher outside-mask difference. It nevertheless maintains the sofa volume and floor structure more naturally, with noticeably better sharpness inside the generated region. On qualitative image quality, MaskFlow is the better result in this sample. If this frame is used as a reference for video propagation, OSOR's blur can be propagated just as consistently as any other reconstruction error. Its lower outside-mask MAE is therefore not enough to make it the preferred output.

This single example is not a model ranking. There is no true background after removal, so inside-mask MAE would measure the amount of change rather than its quality, and the sample size is one. It does show that locality and fill quality have to be evaluated on separate axes. OSOR scores better on preservation but fails on the generated detail; MaskFlow changes more pixels and produces the better reconstruction. OSOR may still serve as a fast default, but it needs a quality gate for blur and texture continuity that can route the frame to MaskFlow.

The Dataset Defines the World the Model Believes In

The mask and fusion modules are the most visible differences among the papers. The deeper distinction is how each paper manufactures a valid before-and-after pair.

SmartEraser's data is clean and abundant, but the physical effects that cannot be obtained by simple insertion remain underrepresented. ObjectClear captures those effects with real before-and-after photography, at the physical cost of collecting 2,878 pairs. OSOR scales to 280K by validating existing editing triplets. That is efficient, but complex cases that fail localization or semantic verification can vanish from the training distribution along with the bad pairs.

MEData serves a different task. It is a regional editing dataset, not a removal-only counterfactual dataset. A full prompt with an explicit position is used to generate the target. The training edit prompt omits the position so that the model must read location from the mask. This cleanly separates the roles of instruction and mask. It also mixes addition, replacement, attribute changes, and removal, so the prior “erase this and invent nothing else” is necessarily weaker than it is in a dedicated removal model.

This lines up with what I observed on production images. A removal model is stable for the object and effects represented by its pairs, then reaches a ceiling on defocused backgrounds, complex paintings, and depth overlap. An editing model can disambiguate the target through language and draw on a stronger generative prior. The same freedom returns as background drift and output variance.

Where the Papers Meet—and Where They Do Not

The four methods can be reduced to four moves:

• SmartEraser returns target evidence to Inpainting.
• ObjectClear turns target attention into an effect region and a fusion mask.
• OSOR predicts adaptive alpha from an incomplete object mask.
• MaskFlow splits an editor's probability path into editable and preserved regions.

Purpose-built Inpainting is becoming more semantic. General Editing is becoming more spatial. Both are moving toward the same ideal: understand what should change semantically, then restrict the change to the pixels that actually require it.

Three objectives still resist complete unification.

First, stronger target recognition can leave a stronger visual feature in generation and cause the object to reappear. Second, expanding the edit region to catch every effect makes source preservation harder. Third, strictly freezing the outside region makes boundary-crossing effects such as light and shadow more difficult to repair coherently.

Depth awareness is also mostly implicit. ObjectClear uses a depth-gradient filter when selecting flat areas for synthetic placement, but none of these models reconstructs explicit 3D layers at inference. They infer “keep the foreground prop, remove the person behind it” from 2D appearance and learned priors. Defocus, transparent surfaces, and foreground overlap in broadcast sets remain difficult for that reason.

Preservation guarantees must also be distinguished by stage. ObjectClear performs a final pixel-space fusion. OSOR composites in latent space. MaskFlow maintains a source trajectory outside the mask. All three reduce unwanted changes, but they do not provide the same guarantee. A latent blend still passes through VAE decoding. If a compositing pipeline requires bit identity, the source must be restored outside the accepted effect mask and the boundary must be validated separately.

My Takeaway from the Comparison

Classifying a model as either Inpainting or Editing is not enough to choose it for Object Removal. Four questions are more useful:

1. Can the model see and identify the target inside the mask?
2. Who predicts effects outside the object mask, and what supervision teaches that prediction?
3. Is the preserved background restored during sampling, in latent space, or in pixel space?
4. Do the before-and-after pairs actually contain the scenes in which the model will be used?

Under this framing, OSOR and MaskFlow are not interchangeable competitors. OSOR is a fast, conservative default for a narrow removal task. MaskFlow adds locality to a strong editor and uses language plus a broader generative prior when mask-only removal cannot understand the scene. Neither contains the other.

The reference frame for Video Inpainting therefore cannot be judged only by how plausible one still image looks. The reconstructed background has to be credible, the surrounding pixels and geometry have to remain aligned, and the hypothesis must remain consistent when it meets known pixels from other frames. Choosing an image prior is not a problem that ends at the front of the video pipeline.