cjwbw / repaint

Inpainting using Denoising Diffusion Probabilistic Models

  • Public
  • 3.9K runs
  • GitHub
  • Paper
Run with an API
This model uses a mask. Visit the new playground to try this model with our mask editing tool.
Playground API Examples README Versions

This is a cog implementation of https://github.com/andreas128/RePaint

RePaint

Inpainting using Denoising Diffusion Probabilistic Models

CVPR 2022 [Paper]

Denoising_Diffusion_Inpainting_Animation

RePaint fills a missing image part using diffusion models

RePaint Inpainting using Denoising Diffusion Probabilistic Models Demo 1 RePaint Inpainting using Denoising Diffusion Probabilistic Models Demo 2

What are the blue parts?
Those parts are missing and therefore have to be filled by RePaint.
RePaint generates the missing parts inspired by the known parts.

How does it work?
RePaint starts from pure noise. Then the image is denoised step-by-step.
It uses the known part to fill the unknown part in each step.

Why does the noise level fluctuate during generation?
Our noise schedule improves the harmony between the generated and
the known part [4.2 Resampling].


RePaint on diverse content and shapes of missing regions

The blue region is unknown and filled by RePaint:

Denoising Diffusion Probabilistic Models Inpainting

Note: RePaint creates many meaningful fillings.
1) Face: Expressions and features like an earring or a mole.
2) Computer: The computer screen shows different images, text, and even a logo.
3) Greens: RePaint makes sense of the tiny known part and incorporates it in a beetle, spaghetti, and plants.
4) Garden: From simple filling like a curtain to complex filling like a human.


Extreme Case 1: Generate every second line

Denoising_Diffusion_Probabilistic_Models_Inpainting_Every_Second_Line

  • Every Second line of the input image is unknown.
  • Most inpainting methods fail on such masks.


Extreme Case 2: Upscale an image

Denoising_Diffusion_Probabilistic_Models_Inpainting_Super_Resolution

  • The inpainting only knows pixels with a stridden access of 2.
  • A ratio of 3/4 of the image has to be filled.
  • This is equivalent to Super-Resolution with the Nearest Neighbor kernel.


RePaint conditions the diffusion model on the known part

  • RePaint uses unconditionally trained Denoising Diffusion Probabilistic Models.
  • We condition during inference on the given image content.

Denoising Diffusion Probabilistic Models Inpainting Method

Intuition of one conditioned denoising step: 1) Sample the known part: Add gaussian noise to the known regions of the image.
We obtain a noisy image that follows the denoising process exactly. 2) Denoise one step: Denoise the previous image for one step. This generates
content for the unknown region conditioned on the known region. 3) Join: Merge the images from both steps.

Details are in Algorithm 1 on Page 5. [Paper]


How to harmonize the generated with the known part?

  • Fail: When using only the algorithm above, the filling is not well harmonized with the known part (n=1).
  • Fix: When applying the [4.2 Resampling] technique, the images are better harmonized (n>1).

Diffusion Model Resampling


RePaint Fails

  • The ImageNet model is biased towards inpainting dogs.
  • This is due to the high ratio of dog images in ImageNet.

RePaint Fails


User Study State-of-the-Art Comparison

  • Outperforms autoregression-based and GAN-based SOTA methods,
    with 95% significance for all masks except for two inconclusive cases.
  • The user study was done for six different masks on three datasets.
  • RePaint outperformed SOTA methods in 42 of 44 cases. [Paper]


Explore the Visual Examples

  • Datasets: CelebA-HQ, ImageNet, Places2
  • Masks: Random strokes, half image, huge, sparse
  • Explore more examples like this in the [Appendix].

Denosing Diffusion Inpainting Examples


Acknowledgement

This work was supported by the ETH Zürich Fund (OK), a Huawei Technologies Oy (Finland) project, and an Nvidia GPU grant.

This repository is based on guided-diffuion from OpenAI.