chenxwh / ominicontrol-spatial

Minimal and Universal Control for Diffusion Transformer - demo for Spatially aligned control

Cold

Public
101 runs
L40S
GitHub
Weights
Paper
License

Run with an API

Playground API Examples README Versions

Input

Run this model in Node.js with one line of code:

npx create-replicate --model=chenxwh/ominicontrol-spatial

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run chenxwh/ominicontrol-spatial using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "chenxwh/ominicontrol-spatial:1b9423cc867733ca0fcf0bc9e677a731fd8654206fc6f3fbddb7279cd91ce917",
  {
    input: {
      image: "https://replicate.delivery/pbxt/MF6fbaFCPPXLDUYYRQZhP0S93K3HQzzZDduHwxXQNAWiOgdb/masked.png",
      model: "fill",
      prompt: "The Mona Lisa is wearing a white VR headset with \"OMINI\" written on it.",
      guidance_scale: 7.5,
      num_inference_steps: 50
    }
  }
);

// To access the file URL:
console.log(output.url()); //=> "http://example.com"

// To write the file to disk:
fs.writeFile("my-image.png", output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run chenxwh/ominicontrol-spatial using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "chenxwh/ominicontrol-spatial:1b9423cc867733ca0fcf0bc9e677a731fd8654206fc6f3fbddb7279cd91ce917",
    input={
        "image": "https://replicate.delivery/pbxt/MF6fbaFCPPXLDUYYRQZhP0S93K3HQzzZDduHwxXQNAWiOgdb/masked.png",
        "model": "fill",
        "prompt": "The Mona Lisa is wearing a white VR headset with \"OMINI\" written on it.",
        "guidance_scale": 7.5,
        "num_inference_steps": 50
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run chenxwh/ominicontrol-spatial using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "1b9423cc867733ca0fcf0bc9e677a731fd8654206fc6f3fbddb7279cd91ce917",
    "input": {
      "image": "https://replicate.delivery/pbxt/MF6fbaFCPPXLDUYYRQZhP0S93K3HQzzZDduHwxXQNAWiOgdb/masked.png",
      "model": "fill",
      "prompt": "The Mona Lisa is wearing a white VR headset with \\"OMINI\\" written on it.",
      "guidance_scale": 7.5,
      "num_inference_steps": 50
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

{
  "completed_at": "2024-12-31T23:48:35.195807Z",
  "created_at": "2024-12-31T23:48:18.777000Z",
  "data_removed": false,
  "error": null,
  "id": "8akda4e8v5rma0cm3wfbn9fd3c",
  "input": {
    "image": "https://replicate.delivery/pbxt/MF6fbaFCPPXLDUYYRQZhP0S93K3HQzzZDduHwxXQNAWiOgdb/masked.png",
    "model": "fill",
    "prompt": "The Mona Lisa is wearing a white VR headset with \"OMINI\" written on it.",
    "guidance_scale": 7.5,
    "num_inference_steps": 50
  },
  "logs": "Using seed: 32895\n  0%|          | 0/50 [00:00<?, ?it/s]\n  2%|▏         | 1/50 [00:00<00:15,  3.19it/s]\n  4%|▍         | 2/50 [00:00<00:14,  3.39it/s]\n  6%|▌         | 3/50 [00:00<00:14,  3.27it/s]\n  8%|▊         | 4/50 [00:01<00:14,  3.22it/s]\n 10%|█         | 5/50 [00:01<00:14,  3.19it/s]\n 12%|█▏        | 6/50 [00:01<00:13,  3.18it/s]\n 14%|█▍        | 7/50 [00:02<00:13,  3.16it/s]\n 16%|█▌        | 8/50 [00:02<00:13,  3.16it/s]\n 18%|█▊        | 9/50 [00:02<00:13,  3.15it/s]\n 20%|██        | 10/50 [00:03<00:12,  3.14it/s]\n 22%|██▏       | 11/50 [00:03<00:12,  3.14it/s]\n 24%|██▍       | 12/50 [00:03<00:12,  3.14it/s]\n 26%|██▌       | 13/50 [00:04<00:11,  3.13it/s]\n 28%|██▊       | 14/50 [00:04<00:11,  3.13it/s]\n 30%|███       | 15/50 [00:04<00:11,  3.12it/s]\n 32%|███▏      | 16/50 [00:05<00:10,  3.12it/s]\n 34%|███▍      | 17/50 [00:05<00:10,  3.13it/s]\n 36%|███▌      | 18/50 [00:05<00:10,  3.12it/s]\n 38%|███▊      | 19/50 [00:06<00:09,  3.11it/s]\n 40%|████      | 20/50 [00:06<00:09,  3.11it/s]\n 42%|████▏     | 21/50 [00:06<00:09,  3.11it/s]\n 44%|████▍     | 22/50 [00:06<00:09,  3.10it/s]\n 46%|████▌     | 23/50 [00:07<00:08,  3.10it/s]\n 48%|████▊     | 24/50 [00:07<00:08,  3.11it/s]\n 50%|█████     | 25/50 [00:07<00:08,  3.10it/s]\n 52%|█████▏    | 26/50 [00:08<00:07,  3.11it/s]\n 54%|█████▍    | 27/50 [00:08<00:07,  3.10it/s]\n 56%|█████▌    | 28/50 [00:08<00:07,  3.10it/s]\n 58%|█████▊    | 29/50 [00:09<00:06,  3.10it/s]\n 60%|██████    | 30/50 [00:09<00:06,  3.10it/s]\n 62%|██████▏   | 31/50 [00:09<00:06,  3.10it/s]\n 64%|██████▍   | 32/50 [00:10<00:05,  3.10it/s]\n 66%|██████▌   | 33/50 [00:10<00:05,  3.10it/s]\n 68%|██████▊   | 34/50 [00:10<00:05,  3.11it/s]\n 70%|███████   | 35/50 [00:11<00:04,  3.11it/s]\n 72%|███████▏  | 36/50 [00:11<00:04,  3.11it/s]\n 74%|███████▍  | 37/50 [00:11<00:04,  3.11it/s]\n 76%|███████▌  | 38/50 [00:12<00:03,  3.10it/s]\n 78%|███████▊  | 39/50 [00:12<00:03,  3.10it/s]\n 80%|████████  | 40/50 [00:12<00:03,  3.10it/s]\n 82%|████████▏ | 41/50 [00:13<00:02,  3.10it/s]\n 84%|████████▍ | 42/50 [00:13<00:02,  3.09it/s]\n 86%|████████▌ | 43/50 [00:13<00:02,  3.10it/s]\n 88%|████████▊ | 44/50 [00:14<00:01,  3.10it/s]\n 90%|█████████ | 45/50 [00:14<00:01,  3.10it/s]\n 92%|█████████▏| 46/50 [00:14<00:01,  3.10it/s]\n 94%|█████████▍| 47/50 [00:15<00:00,  3.10it/s]\n 96%|█████████▌| 48/50 [00:15<00:00,  3.09it/s]\n 98%|█████████▊| 49/50 [00:15<00:00,  3.09it/s]\n100%|██████████| 50/50 [00:16<00:00,  3.09it/s]\n100%|██████████| 50/50 [00:16<00:00,  3.12it/s]",
  "metrics": {
    "predict_time": 16.411346081,
    "total_time": 16.418807
  },
  "output": "https://replicate.delivery/xezq/N8TXdUfGht2WNqNAufhyuZMpep0PhRqxHgCk04AnISGmmGBoA/out.png",
  "started_at": "2024-12-31T23:48:18.784461Z",
  "status": "succeeded",
  "urls": {
    "stream": "https://stream.replicate.com/v1/files/bcwr-mzad2jq7z3am3jbqlndv3l4clchmcqmy724qqtoelndmobl7u2cq",
    "get": "https://api.replicate.com/v1/predictions/8akda4e8v5rma0cm3wfbn9fd3c",
    "cancel": "https://api.replicate.com/v1/predictions/8akda4e8v5rma0cm3wfbn9fd3c/cancel"
  },
  "version": "1b9423cc867733ca0fcf0bc9e677a731fd8654206fc6f3fbddb7279cd91ce917"
}

Generated in

16.4 seconds

Tweak it ShareReport

Using seed: 32895
  0%|          | 0/50 [00:00<?, ?it/s]
  2%|▏         | 1/50 [00:00<00:15,  3.19it/s]
  4%|▍         | 2/50 [00:00<00:14,  3.39it/s]
  6%|▌         | 3/50 [00:00<00:14,  3.27it/s]
  8%|▊         | 4/50 [00:01<00:14,  3.22it/s]
 10%|█         | 5/50 [00:01<00:14,  3.19it/s]
 12%|█▏        | 6/50 [00:01<00:13,  3.18it/s]
 14%|█▍        | 7/50 [00:02<00:13,  3.16it/s]
 16%|█▌        | 8/50 [00:02<00:13,  3.16it/s]
 18%|█▊        | 9/50 [00:02<00:13,  3.15it/s]
 20%|██        | 10/50 [00:03<00:12,  3.14it/s]
 22%|██▏       | 11/50 [00:03<00:12,  3.14it/s]
 24%|██▍       | 12/50 [00:03<00:12,  3.14it/s]
 26%|██▌       | 13/50 [00:04<00:11,  3.13it/s]
 28%|██▊       | 14/50 [00:04<00:11,  3.13it/s]
 30%|███       | 15/50 [00:04<00:11,  3.12it/s]
 32%|███▏      | 16/50 [00:05<00:10,  3.12it/s]
 34%|███▍      | 17/50 [00:05<00:10,  3.13it/s]
 36%|███▌      | 18/50 [00:05<00:10,  3.12it/s]
 38%|███▊      | 19/50 [00:06<00:09,  3.11it/s]
 40%|████      | 20/50 [00:06<00:09,  3.11it/s]
 42%|████▏     | 21/50 [00:06<00:09,  3.11it/s]
 44%|████▍     | 22/50 [00:06<00:09,  3.10it/s]
 46%|████▌     | 23/50 [00:07<00:08,  3.10it/s]
 48%|████▊     | 24/50 [00:07<00:08,  3.11it/s]
 50%|█████     | 25/50 [00:07<00:08,  3.10it/s]
 52%|█████▏    | 26/50 [00:08<00:07,  3.11it/s]
 54%|█████▍    | 27/50 [00:08<00:07,  3.10it/s]
 56%|█████▌    | 28/50 [00:08<00:07,  3.10it/s]
 58%|█████▊    | 29/50 [00:09<00:06,  3.10it/s]
 60%|██████    | 30/50 [00:09<00:06,  3.10it/s]
 62%|██████▏   | 31/50 [00:09<00:06,  3.10it/s]
 64%|██████▍   | 32/50 [00:10<00:05,  3.10it/s]
 66%|██████▌   | 33/50 [00:10<00:05,  3.10it/s]
 68%|██████▊   | 34/50 [00:10<00:05,  3.11it/s]
 70%|███████   | 35/50 [00:11<00:04,  3.11it/s]
 72%|███████▏  | 36/50 [00:11<00:04,  3.11it/s]
 74%|███████▍  | 37/50 [00:11<00:04,  3.11it/s]
 76%|███████▌  | 38/50 [00:12<00:03,  3.10it/s]
 78%|███████▊  | 39/50 [00:12<00:03,  3.10it/s]
 80%|████████  | 40/50 [00:12<00:03,  3.10it/s]
 82%|████████▏ | 41/50 [00:13<00:02,  3.10it/s]
 84%|████████▍ | 42/50 [00:13<00:02,  3.09it/s]
 86%|████████▌ | 43/50 [00:13<00:02,  3.10it/s]
 88%|████████▊ | 44/50 [00:14<00:01,  3.10it/s]
 90%|█████████ | 45/50 [00:14<00:01,  3.10it/s]
 92%|█████████▏| 46/50 [00:14<00:01,  3.10it/s]
 94%|█████████▍| 47/50 [00:15<00:00,  3.10it/s]
 96%|█████████▌| 48/50 [00:15<00:00,  3.09it/s]
 98%|█████████▊| 49/50 [00:15<00:00,  3.09it/s]
100%|██████████| 50/50 [00:16<00:00,  3.09it/s]
100%|██████████| 50/50 [00:16<00:00,  3.12it/s]

Examples

View more examples

Run time and cost

This model runs on Nvidia L40S GPU hardware. We don't yet have enough runs of this model to provide performance information.

Readme

OminiControl: Minimal and Universal Control for Diffusion Transformer

This is the demo for Spatially aligned control. See https://replicate.com/chenxwh/ominicontrol-subject for Subject-driven generation.

Features

OminiControl is a minimal yet powerful universal control framework for Diffusion Transformer models like FLUX.

Universal Control 🌐: A unified control framework that supports both subject-driven control and spatial control (such as edge-guided and in-painting generation).
Minimal Design 🚀: Injects control signals while preserving original model structure. Only introduces 0.1% additional parameters to the base model.

Citation

@article{
  tan2024omini,
  title={OminiControl: Minimal and Universal Control for Diffusion Transformer},
  author={Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang},
  journal={arXiv preprint arXiv:2411.15098},
  year={2024}
}