This is a cog implementation of https://github.com/andreas128/RePaint

RePaint

Inpainting using Denoising Diffusion Probabilistic Models

CVPR 2022 [Paper]

RePaint fills a missing image part using diffusion models

What are the blue parts?
Those parts are missing and therefore have to be filled by RePaint.
RePaint generates the missing parts inspired by the known parts.

How does it work?
RePaint starts from pure noise. Then the image is denoised step-by-step.
It uses the known part to fill the unknown part in each step.

Why does the noise level fluctuate during generation?
Our noise schedule improves the harmony between the generated and
the known part [4.2 Resampling].

RePaint on diverse content and shapes of missing regions

The blue region is unknown and filled by RePaint:

Note: RePaint creates many meaningful fillings.
1) Face: Expressions and features like an earring or a mole.
2) Computer: The computer screen shows different images, text, and even a logo.
3) Greens: RePaint makes sense of the tiny known part and incorporates it in a beetle, spaghetti, and plants.
4) Garden: From simple filling like a curtain to complex filling like a human.

Extreme Case 1: Generate every second line

• Every Second line of the input image is unknown.
• Most inpainting methods fail on such masks.

Extreme Case 2: Upscale an image

• The inpainting only knows pixels with a stridden access of 2.
• A ratio of 3/4 of the image has to be filled.
• This is equivalent to Super-Resolution with the Nearest Neighbor kernel.

RePaint conditions the diffusion model on the known part

• RePaint uses unconditionally trained Denoising Diffusion Probabilistic Models.
• We condition during inference on the given image content.

Intuition of one conditioned denoising step: 1) Sample the known part: Add gaussian noise to the known regions of the image.
We obtain a noisy image that follows the denoising process exactly. 2) Denoise one step: Denoise the previous image for one step. This generates
content for the unknown region conditioned on the known region. 3) Join: Merge the images from both steps.

Details are in Algorithm 1 on Page 5. [Paper]

How to harmonize the generated with the known part?

• Fail: When using only the algorithm above, the filling is not well harmonized with the known part (n=1).
• Fix: When applying the [4.2 Resampling] technique, the images are better harmonized (n>1).

RePaint Fails

• The ImageNet model is biased towards inpainting dogs.
• This is due to the high ratio of dog images in ImageNet.

User Study State-of-the-Art Comparison

• Outperforms autoregression-based and GAN-based SOTA methods,
  with 95% significance for all masks except for two inconclusive cases.
• The user study was done for six different masks on three datasets.
• RePaint outperformed SOTA methods in 42 of 44 cases. [Paper]

Explore the Visual Examples

• Datasets: CelebA-HQ, ImageNet, Places2
• Masks: Random strokes, half image, huge, sparse
• Explore more examples like this in the [Appendix].

Acknowledgement

This work was supported by the ETH Zürich Fund (OK), a Huawei Technologies Oy (Finland) project, and an Nvidia GPU grant.

This repository is based on guided-diffuion from OpenAI.