alexgenovese / ominicontrol-lora

WIP | Minimal and Universal Control for Diffusion Transformer

Cold

Public
30 runs
L40S
GitHub
Weights
Paper
License

Run with an API

Playground API Examples README Versions

Input

prompt

*string

Shift + Return to add a new line

Prompt for generated image

aspect_ratio

string

Aspect ratio for the generated image

Default: "1:1"

image

file

Image for omni

num_outputs

integer

(minimum: 1, maximum: 4)

Number of images to output.

Default: 1

num_inference_steps

integer

(minimum: 1, maximum: 50)

Number of inference steps

Default: 28

guidance_scale

number

(minimum: 0, maximum: 10)

Guidance scale for the diffusion process

Default: 3.5

seed

integer

Random seed. Set for reproducible generation

output_format

string

Format of the output images

Default: "webp"

output_quality

integer

(minimum: 0, maximum: 100)

Quality when saving the output images, from 0 to 100. 100 is best quality, 0 is lowest quality. Not relevant for .png outputs

Default: 80

hf_lora

string

Shift + Return to add a new line

HF, Replicate, CivitAI, or URL to a LoRA. Ex: alvdansen/frosting_lane_flux

lora_scale

number

(minimum: 0, maximum: 1)

Scale for the LoRA weights

Default: 0.8

disable_safety_checker

boolean

Disable safety checker for generated images. This feature is only available through the API. See [https://replicate.com/docs/how-does-replicate-work#safety](https://replicate.com/docs/how-does-replicate-work#safety)

Default: false

Run this model in Node.js with one line of code:

npx create-replicate --model=alexgenovese/ominicontrol-lora

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run alexgenovese/ominicontrol-lora using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "alexgenovese/ominicontrol-lora:e5722dc6ff1a32a4f3b47ec4906bb1c5f06ebedab08dfd04d1dbd268d304b34d",
  {
    input: {
      lora_scale: 0.8,
      num_outputs: 1,
      aspect_ratio: "1:1",
      output_format: "webp",
      guidance_scale: 3.5,
      output_quality: 80,
      num_inference_steps: 28
    }
  }
);

// To access the file URL:
console.log(output[0].url()); //=> "http://example.com"

// To write the file to disk:
fs.writeFile("my-image.png", output[0]);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run alexgenovese/ominicontrol-lora using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "alexgenovese/ominicontrol-lora:e5722dc6ff1a32a4f3b47ec4906bb1c5f06ebedab08dfd04d1dbd268d304b34d",
    input={
        "lora_scale": 0.8,
        "num_outputs": 1,
        "aspect_ratio": "1:1",
        "output_format": "webp",
        "guidance_scale": 3.5,
        "output_quality": 80,
        "num_inference_steps": 28
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run alexgenovese/ominicontrol-lora using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "alexgenovese/ominicontrol-lora:e5722dc6ff1a32a4f3b47ec4906bb1c5f06ebedab08dfd04d1dbd268d304b34d",
    "input": {
      "lora_scale": 0.8,
      "num_outputs": 1,
      "aspect_ratio": "1:1",
      "output_format": "webp",
      "guidance_scale": 3.5,
      "output_quality": 80,
      "num_inference_steps": 28
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

No output yet! Press "Submit" to start a prediction.

Run time and cost

This model runs on Nvidia L40S GPU hardware. We don't yet have enough runs of this model to provide performance information.

Readme

About

This endpoint is a Cog implementation of OminiControl with Flux Dev 1.

⚠ This version is not working due to a huge model folder of 56GB (flux + omini).

The working copy is on Github; please follow the Github URL below the title

OminiControl is a minimal yet powerful universal control framework for Diffusion Transformer models like FLUX.

Universal Control 🌐: A unified control framework that supports both subject-driven control and spatial control (such as edge-guided and in-painting generation).
Minimal Design 🚀: Injects control signals while preserving original model structure. Only introduces 0.1% additional parameters to the base model.

Cover Image

Contact me if you need customizations.

Follow me on Twitter/X | Website

Licensing and commercial use

You can use the images commercially if you generate images on Replicate with FLUX.1 models and their fine-tunes.

If you download the weights off Replicate and generate images on your computer, you can’t use the images commercially.

Credits

OminiControl: Minimal and Universal Control for Diffusion Transformer Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang Learning and Vision Lab, National University of Singapore