Introduction
Object removal—particularly in video—has become a critical capability in modern AI systems, enabling applications from content editing to privacy protection. However, as the technology advances, a growing disconnect has emerged between the performance of object removal models and the metrics used to evaluate them. This mismatch highlights a fundamental challenge in AI evaluation: the inadequacy of traditional metrics for assessing tasks that are inherently ambiguous or ill-posed.
Xiaomi's MiLM Plus has introduced a new benchmark and evaluation metrics—PROVE (Perception-Aligned Object Removal Metrics), specifically RC-S and RC-T—that aim to address this issue. These metrics are designed to align more closely with human perception and the real-world complexity of object removal tasks. This article explores the concept of ill-posed problems, why traditional metrics fail, and how PROVE addresses these shortcomings.
What is an Ill-Posed Problem?
In mathematics and optimization, an ill-posed problem is one where a solution either does not exist, is not unique, or is unstable with respect to small changes in the input. In the context of object removal, the problem is ill-posed because removing an object from an image or video does not have a single, correct answer.
Consider a photo of a person standing in front of a building. If we remove the person, the background must be reconstructed. But there are countless ways to do this correctly—each depending on the lighting, shadows, reflections, and occlusions. For example, a shadow cast by the person might be missing, or a reflection in a window might need to be regenerated. Each of these elements introduces ambiguity, meaning the problem is not just hard to solve but fundamentally under-constrained.
How Do Traditional Metrics Fall Short?
Traditional metrics like PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity Index), and LPIPS (Learned Perceptual Image Patch Similarity) are designed to measure pixel-level differences between a generated image and a ground truth. These metrics work well for tasks with a single, correct output—like image classification or super-resolution where the goal is to enhance a clear, defined image.
However, in object removal, there is no single ground truth. The reconstructed scene can be correct in multiple ways, and traditional metrics often fail to capture this. For example, a model might remove a person from a scene and reconstruct the background in a way that is visually plausible but not pixel-perfect compared to a reference image. Traditional metrics might rate this poorly, even though the output is perceptually correct.
Additionally, metrics like ReMOVE and CFD (Contextual Feature Distance) are more advanced but still rely on reference images or fixed reconstruction rules, which do not reflect the true variability of real-world scenarios. These metrics are not perception-aligned, meaning they do not account for how humans actually perceive object removal success.
Why Does This Matter?
As AI systems become more sophisticated, the gap between technical performance and perceptual quality is increasingly critical. In practical applications, such as video editing or surveillance, the goal is not just to remove an object but to do so in a way that is visually convincing and contextually appropriate.
The PROVE benchmark and its metrics—RC-S (Reconstruction Consistency Score) and RC-T (Reconstruction Temporal Consistency Score)—are designed to close this gap. RC-S evaluates how well the reconstructed scene aligns with the surrounding visual context, while RC-T evaluates consistency across video frames. These metrics are built to reflect human perception, ensuring that models are not just technically sound but also visually believable.
For example, in a video where a person is removed from a scene, a model must not only reconstruct the background but also ensure that the lighting, shadows, and motion are consistent across frames. Traditional metrics might miss this, but RC-T ensures that temporal coherence is a key part of the evaluation.
Key Takeaways
- Object removal is an ill-posed problem—there is no single correct solution due to the ambiguity of visual reconstruction.
- Traditional metrics are inadequate for evaluating object removal because they assume a fixed ground truth and do not account for perceptual quality.
- PROVE benchmark introduces perception-aligned metrics (RC-S and RC-T) that better reflect human judgment and real-world usability.
- Temporal consistency is crucial in video object removal and must be evaluated using metrics that consider motion and context over time.
- Aligning metrics with human perception is essential for advancing AI systems in complex, real-world applications.
