stability-ai/stable-diffusion-inpainting – Run with an API on Replicate

stability-ai / stable-diffusion-inpainting

Fill in masked parts of images with Stable Diffusion

Warm

Public
19.5M runs
A100 (80GB)
GitHub
License

Run with an API

Playground API Examples README Versions

Input

prompt

string

Shift + Return to add a new line

Face of a yellow cat, high resolution, sitting on a park benchFace of a yellow cat, high resolution, sitting on a park bench

Input prompt

Default: "a vision of paradise. unreal engine"

image

*file

Preview

Initial image to generate variations of. Will be resized to height x width

mask

*file

Preview

Black and white image to use as mask for inpainting over the image provided. White pixels are inpainted and black pixels are preserved.

height

integer

Height of generated image in pixels. Needs to be a multiple of 64

Default: 512

width

integer

Width of generated image in pixels. Needs to be a multiple of 64

Default: 512

negative_prompt

string

Shift + Return to add a new line

Specify things to not see in the output

num_outputs

integer

(minimum: 1, maximum: 4)

Number of images to generate.

Default: 1

num_inference_steps

integer

(minimum: 1, maximum: 500)

Number of denoising steps

Default: 50

guidance_scale

number

(minimum: 1, maximum: 20)

Scale for classifier-free guidance

Default: 7.5

scheduler

string

Choose a scheduler.

Default: "DPMSolverMultistep"

seed

integer

Random seed. Leave blank to randomize the seed

disable_safety_checker

boolean

Disable safety checker for generated images. This feature is only available through the API. See [https://replicate.com/docs/how-does-replicate-work#safety](https://replicate.com/docs/how-does-replicate-work#safety)

Default: false

Run this model in Node.js with one line of code:

npx create-replicate --model=stability-ai/stable-diffusion-inpainting

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run stability-ai/stable-diffusion-inpainting using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "stability-ai/stable-diffusion-inpainting:95b7223104132402a9ae91cc677285bc5eb997834bd2349fa486f53910fd68b3",
  {
    input: {
      mask: "https://replicate.delivery/pbxt/HtGQBqO9MtVbPm0G0K43nsvvjBB0E0PaWOhuNRrRBBT4ttbf/mask.png",
      image: "https://replicate.delivery/pbxt/HtGQBfA5TrqFYZBf0UL18NTqHrzt8UiSIsAkUuMHtjvFDO6p/overture-creations-5sI6fQgYIuo.png",
      width: 512,
      height: 512,
      prompt: "Face of a yellow cat, high resolution, sitting on a park bench",
      scheduler: "DPMSolverMultistep",
      num_outputs: 1,
      guidance_scale: 7.5,
      num_inference_steps: 25
    }
  }
);
console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run stability-ai/stable-diffusion-inpainting using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "stability-ai/stable-diffusion-inpainting:95b7223104132402a9ae91cc677285bc5eb997834bd2349fa486f53910fd68b3",
    input={
        "mask": "https://replicate.delivery/pbxt/HtGQBqO9MtVbPm0G0K43nsvvjBB0E0PaWOhuNRrRBBT4ttbf/mask.png",
        "image": "https://replicate.delivery/pbxt/HtGQBfA5TrqFYZBf0UL18NTqHrzt8UiSIsAkUuMHtjvFDO6p/overture-creations-5sI6fQgYIuo.png",
        "width": 512,
        "height": 512,
        "prompt": "Face of a yellow cat, high resolution, sitting on a park bench",
        "scheduler": "DPMSolverMultistep",
        "num_outputs": 1,
        "guidance_scale": 7.5,
        "num_inference_steps": 25
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run stability-ai/stable-diffusion-inpainting using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "95b7223104132402a9ae91cc677285bc5eb997834bd2349fa486f53910fd68b3",
    "input": {
      "mask": "https://replicate.delivery/pbxt/HtGQBqO9MtVbPm0G0K43nsvvjBB0E0PaWOhuNRrRBBT4ttbf/mask.png",
      "image": "https://replicate.delivery/pbxt/HtGQBfA5TrqFYZBf0UL18NTqHrzt8UiSIsAkUuMHtjvFDO6p/overture-creations-5sI6fQgYIuo.png",
      "width": 512,
      "height": 512,
      "prompt": "Face of a yellow cat, high resolution, sitting on a park bench",
      "scheduler": "DPMSolverMultistep",
      "num_outputs": 1,
      "guidance_scale": 7.5,
      "num_inference_steps": 25
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

You can run this model locally using Cog. First, install Cog:

brew install cog

If you don’t have Homebrew, there are other installation options available.

Run this to download the model and run it in your local environment:

cog predict r8.im/stability-ai/stable-diffusion-inpainting@sha256:95b7223104132402a9ae91cc677285bc5eb997834bd2349fa486f53910fd68b3 \
  -i 'mask="https://replicate.delivery/pbxt/HtGQBqO9MtVbPm0G0K43nsvvjBB0E0PaWOhuNRrRBBT4ttbf/mask.png"' \
  -i 'image="https://replicate.delivery/pbxt/HtGQBfA5TrqFYZBf0UL18NTqHrzt8UiSIsAkUuMHtjvFDO6p/overture-creations-5sI6fQgYIuo.png"' \
  -i 'width=512' \
  -i 'height=512' \
  -i 'prompt="Face of a yellow cat, high resolution, sitting on a park bench"' \
  -i 'scheduler="DPMSolverMultistep"' \
  -i 'num_outputs=1' \
  -i 'guidance_scale=7.5' \
  -i 'num_inference_steps=25'

To learn more, take a look at the Cog documentation.

Run this to download the model and run it in your local environment:

docker run -d -p 5000:5000 --gpus=all r8.im/stability-ai/stable-diffusion-inpainting@sha256:95b7223104132402a9ae91cc677285bc5eb997834bd2349fa486f53910fd68b3
curl -s -X POST \
  -H "Content-Type: application/json" \
  -d $'{
    "input": {
      "mask": "https://replicate.delivery/pbxt/HtGQBqO9MtVbPm0G0K43nsvvjBB0E0PaWOhuNRrRBBT4ttbf/mask.png",
      "image": "https://replicate.delivery/pbxt/HtGQBfA5TrqFYZBf0UL18NTqHrzt8UiSIsAkUuMHtjvFDO6p/overture-creations-5sI6fQgYIuo.png",
      "width": 512,
      "height": 512,
      "prompt": "Face of a yellow cat, high resolution, sitting on a park bench",
      "scheduler": "DPMSolverMultistep",
      "num_outputs": 1,
      "guidance_scale": 7.5,
      "num_inference_steps": 25
    }
  }' \
  http://localhost:5000/predictions

To learn more, take a look at the Cog documentation.

Output

{
  "completed_at": "2022-12-02T18:40:24.805380Z",
  "created_at": "2022-12-02T18:40:22.058556Z",
  "data_removed": false,
  "error": null,
  "id": "xkyxrm6j3jb3llitusuf4hycoa",
  "input": {
    "mask": "https://replicate.delivery/pbxt/HtGQBqO9MtVbPm0G0K43nsvvjBB0E0PaWOhuNRrRBBT4ttbf/mask.png",
    "image": "https://replicate.delivery/pbxt/HtGQBfA5TrqFYZBf0UL18NTqHrzt8UiSIsAkUuMHtjvFDO6p/overture-creations-5sI6fQgYIuo.png",
    "prompt": "Face of a yellow cat, high resolution, sitting on a park bench",
    "num_outputs": 1,
    "guidance_scale": 7.5,
    "prompt_strength": 0.8,
    "num_inference_steps": 25
  },
  "logs": "Using seed: 62372\n  0%|          | 0/26 [00:00<?, ?it/s]\n  8%|▊         | 2/26 [00:00<00:01, 14.22it/s]\n 15%|█▌        | 4/26 [00:00<00:01, 15.62it/s]\n 23%|██▎       | 6/26 [00:00<00:01, 15.98it/s]\n 31%|███       | 8/26 [00:00<00:01, 16.07it/s]\n 38%|███▊      | 10/26 [00:00<00:00, 16.14it/s]\n 46%|████▌     | 12/26 [00:00<00:00, 16.29it/s]\n 54%|█████▍    | 14/26 [00:00<00:00, 16.39it/s]\n 62%|██████▏   | 16/26 [00:00<00:00, 16.40it/s]\n 69%|██████▉   | 18/26 [00:01<00:00, 16.42it/s]\n 77%|███████▋  | 20/26 [00:01<00:00, 16.49it/s]\n 85%|████████▍ | 22/26 [00:01<00:00, 16.31it/s]\n 92%|█████████▏| 24/26 [00:01<00:00, 16.37it/s]\n100%|██████████| 26/26 [00:01<00:00, 16.47it/s]\n100%|██████████| 26/26 [00:01<00:00, 16.26it/s]",
  "metrics": {
    "predict_time": 2.711119,
    "total_time": 2.746824
  },
  "output": [
    "https://replicate.delivery/pbxt/xs0pPOUM6HKmPlJJBXqKfE1YsiMzgNsCuGedlX0VqvPYifLgA/out-0.png"
  ],
  "started_at": "2022-12-02T18:40:22.094261Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/xkyxrm6j3jb3llitusuf4hycoa",
    "cancel": "https://api.replicate.com/v1/predictions/xkyxrm6j3jb3llitusuf4hycoa/cancel"
  },
  "version": "e5a34f913de0adc560d20e002c45ad43a80031b62caacc3d84010c6b6a64870c"
}

Generated in

2.7 seconds

Tweak itReport

This example was created by a different version, stability-ai/stable-diffusion-inpainting:e5a34f91.

Examples

View more examples

Run time and cost

This model costs approximately $0.0023 to run on Replicate, or 434 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia A100 (80GB) GPU hardware. Predictions typically complete within 2 seconds.

Readme

This stable-diffusion-2-inpainting model is resumed from stable-diffusion-2-base (512-base-ema.ckpt) and trained for another 200k steps. Follows the mask-generation strategy presented in LAMA which, in combination with the latent VAE representations of the masked image, are used as an additional conditioning.

Developed by: Robin Rombach, Patrick Esser
Model type: Diffusion-based text-to-image generation model
Language(s): English
License: CreativeML Open RAIL++-M License
Model Description: This is a model that can be used to generate and modify images based on text prompts. It is a Latent Diffusion Model that uses a fixed, pretrained text encoder (OpenCLIP-ViT/H).
Resources for more information: GitHub Repository.

Intended use

See stability-ai/stable-diffusion for direct use, misuse, malicious use, out-of-scope use, limitations, and bias.

Training

Training Data

The model developers used the following dataset for training the model:

LAION-5B and subsets (details below). The training data is further filtered using LAION’s NSFW detector, with a “p_unsafe” score of 0.1 (conservative). For more details, please refer to LAION-5B’s NeurIPS 2022 paper and reviewer discussions on the topic.

Training Procedure

Stable Diffusion v2 is a latent diffusion model which combines an autoencoder with a diffusion model that is trained in the latent space of the autoencoder. During training,

Images are encoded through an encoder, which turns images into latent representations. The autoencoder uses a relative downsampling factor of 8 and maps images of shape H x W x 3 to latents of shape H/f x W/f x 4
Text prompts are encoded through the OpenCLIP-ViT/H text-encoder.
The output of the text encoder is fed into the UNet backbone of the latent diffusion model via cross-attention.
The loss is a reconstruction objective between the noise that was added to the latent and the prediction made by the UNet. We also use the so-called v-objective, see https://arxiv.org/abs/2202.00512.