omerbt / multidiffusion

Fusing Diffusion Paths for Controlled Image Generation

Cold

Public
2.2K runs
A100 (80GB)
GitHub
Paper

Run with an API

Playground API Examples README Versions

Input

prompt

string

Shift + Return to add a new line

Input prompt

Default: "a photo of the dolomites"

negative_prompt

string

Shift + Return to add a new line

The prompt or prompts not to guide the image generation (what you do not want to see in the generation). Ignored when not using guidance.

width

integer

Width of output image. Lower the setting if out of memory.

Default: 4096

height

integer

Height of output image. Lower the setting if out of memory.

Default: 512

num_outputs

integer

Number of images to output

Default: 1

num_inference_steps

integer

(minimum: 1, maximum: 500)

Number of denoising steps

Default: 50

guidance_scale

number

(minimum: 1, maximum: 20)

Scale for classifier-free guidance

Default: 7.5

scheduler

string

Choose a scheduler.

Default: "DDIM"

seed

integer

Random seed. Leave blank to randomize the seed

Run this model in Node.js with one line of code:

npx create-replicate --model=omerbt/multidiffusion

or set up a project from scratch

Install Replicate’s Node.js client library

npm install replicate

Set the REPLICATE_API_TOKEN environment variable

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run omerbt/multidiffusion using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "omerbt/multidiffusion:fd8e12cbfb7cf355d9d80afef487a34bf58b916bedc5bff5f7a8273fe72c1387",
  {
    input: {
      width: 4096,
      height: 512,
      prompt: "a photo of the dolomites",
      scheduler: "K_EULER",
      num_outputs: 1,
      guidance_scale: 7.5,
      num_inference_steps: 20
    }
  }
);
console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library

pip install replicate

Set the REPLICATE_API_TOKEN environment variable

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client

import replicate

Run omerbt/multidiffusion using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "omerbt/multidiffusion:fd8e12cbfb7cf355d9d80afef487a34bf58b916bedc5bff5f7a8273fe72c1387",
    input={
        "width": 4096,
        "height": 512,
        "prompt": "a photo of the dolomites",
        "scheduler": "K_EULER",
        "num_outputs": 1,
        "guidance_scale": 7.5,
        "num_inference_steps": 20
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run omerbt/multidiffusion using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "fd8e12cbfb7cf355d9d80afef487a34bf58b916bedc5bff5f7a8273fe72c1387",
    "input": {
      "width": 4096,
      "height": 512,
      "prompt": "a photo of the dolomites",
      "scheduler": "K_EULER",
      "num_outputs": 1,
      "guidance_scale": 7.5,
      "num_inference_steps": 20
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Install Cog

brew install cog

If you don’t have Homebrew, there are other installation options available.

Pull and run omerbt/multidiffusion using Cog (this will download the full model and run it in your local environment):

cog predict r8.im/omerbt/multidiffusion@sha256:fd8e12cbfb7cf355d9d80afef487a34bf58b916bedc5bff5f7a8273fe72c1387 \
  -i 'width=4096' \
  -i 'height=512' \
  -i 'prompt="a photo of the dolomites"' \
  -i 'scheduler="K_EULER"' \
  -i 'num_outputs=1' \
  -i 'guidance_scale=7.5' \
  -i 'num_inference_steps=20'

To learn more, take a look at the Cog documentation.

Pull and run omerbt/multidiffusion using Docker (this will download the full model and run it in your local environment):

docker run -d -p 5000:5000 --gpus=all r8.im/omerbt/multidiffusion@sha256:fd8e12cbfb7cf355d9d80afef487a34bf58b916bedc5bff5f7a8273fe72c1387
curl -s -X POST \
  -H "Content-Type: application/json" \
  -d $'{
    "input": {
      "width": 4096,
      "height": 512,
      "prompt": "a photo of the dolomites",
      "scheduler": "K_EULER",
      "num_outputs": 1,
      "guidance_scale": 7.5,
      "num_inference_steps": 20
    }
  }' \
  http://localhost:5000/predictions

Output

{
  "completed_at": "2023-02-24T22:54:34.259451Z",
  "created_at": "2023-02-24T22:49:16.619509Z",
  "data_removed": false,
  "error": null,
  "id": "iwwaat27dnam3jag2kq4ofao6i",
  "input": {
    "width": 4096,
    "height": 512,
    "prompt": "a photo of the dolomites",
    "scheduler": "K_EULER",
    "num_outputs": 1,
    "guidance_scale": 7.5,
    "num_inference_steps": 20
  },
  "logs": "Using seed: 11915\n  0%|          | 0/20 [00:00<?, ?it/s]\n  5%|▌         | 1/20 [00:10<03:21, 10.59s/it]\n 10%|█         | 2/20 [00:19<02:51,  9.54s/it]\n 15%|█▌        | 3/20 [00:28<02:37,  9.24s/it]\n 20%|██        | 4/20 [00:37<02:26,  9.13s/it]\n 25%|██▌       | 5/20 [00:46<02:16,  9.09s/it]\n 30%|███       | 6/20 [00:55<02:07,  9.09s/it]\n 35%|███▌      | 7/20 [01:04<01:58,  9.12s/it]\n 40%|████      | 8/20 [01:13<01:49,  9.16s/it]\n 45%|████▌     | 9/20 [01:23<01:41,  9.19s/it]\n 50%|█████     | 10/20 [01:32<01:32,  9.23s/it]\n 55%|█████▌    | 11/20 [01:41<01:23,  9.29s/it]\n 60%|██████    | 12/20 [01:51<01:14,  9.36s/it]\n 65%|██████▌   | 13/20 [02:00<01:05,  9.39s/it]\n 70%|███████   | 14/20 [02:10<00:56,  9.39s/it]\n 75%|███████▌  | 15/20 [02:19<00:46,  9.38s/it]\n 80%|████████  | 16/20 [02:28<00:37,  9.36s/it]\n 85%|████████▌ | 17/20 [02:38<00:28,  9.34s/it]\n 90%|█████████ | 18/20 [02:47<00:18,  9.33s/it]\n 95%|█████████▌| 19/20 [02:56<00:09,  9.34s/it]\n100%|██████████| 20/20 [03:06<00:00,  9.35s/it]\n100%|██████████| 20/20 [03:06<00:00,  9.31s/it]",
  "metrics": {
    "predict_time": 191.736824,
    "total_time": 317.639942
  },
  "output": [
    "https://replicate.delivery/pbxt/5yWjQbNu3paaFZEUdR9eBvJCg3gtWgDheTjSNvM9HYmpIvhQA/out-0.png"
  ],
  "started_at": "2023-02-24T22:51:22.522627Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/iwwaat27dnam3jag2kq4ofao6i",
    "cancel": "https://api.replicate.com/v1/predictions/iwwaat27dnam3jag2kq4ofao6i/cancel"
  },
  "version": "963ffeaa7d992479bbbf6ad5b07f2ebbd51453f5214de98c87b32b97b44f5863"
}

Generated in

3 minutes 12 seconds

Tweak itReport

This example was created by a different version, omerbt/multidiffusion:963ffeaa.

Examples

View more examples

Run time and cost

This model costs approximately $0.068 to run on Replicate, or 14 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia A100 (80GB) GPU hardware. Predictions typically complete within 49 seconds. The predict time for this model varies significantly based on the inputs.

Readme

MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation

Cog Implementation: https://github.com/chenxwh/cog-themed-diffusion/tree/MultiDiffusion

This page demonstrates MultiDiffusion Text2Panorama using Stable Diffusion model

teaser

MultiDiffusion is a unified framework that enables versatile and controllable image generation, using a pre-trained text-to-image diffusion model, without any further training or finetuning, as described in (link to paper).

Recent advances in text-to-image generation with diffusion models present transformative capabilities in image quality. However, user controllability of the generated image, and fast adaptation to new tasks still remains an open challenge, currently mostly addressed by costly and long re-training and fine-tuning or ad-hoc adaptations to specific image generation tasks. In this work, we present MultiDiffusion, a unified framework that enables versatile and controllable image generation, using a pre-trained text-to-image diffusion model, without any further training or finetuning. At the center of our approach is a new generation process, based on an optimization task that binds together multiple diffusion generation processes with a shared set of parameters or constraints. We show that MultiDiffusion can be readily applied to generate high quality and diverse images that adhere to user-provided controls, such as desired aspect ratio (e.g., panorama), and spatial guiding signals, ranging from tight segmentation masks to bounding boxes.

Citation

@article{bar2023multidiffusion,
  title={MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation},
  author={Bar-Tal, Omer and Yariv, Lior and Lipman, Yaron and Dekel, Tali},
  journal={arXiv preprint arXiv:2302.08113},
  year={2023}
}