zsxkib / sd3-controlnet

✨Stable Diffusion 3 w/ ⚡InstantX's Canny, Pose, and Tile ControlNets🖼️

Cold

Public
1.2K runs
L40S
GitHub
Paper
License

Run with an API

Playground API Examples README Versions

Input

input_image

*file

Input image

prompt

*string

Shift + Return to add a new line

A surreal interpretation of an ice cream cone transforming into a fantastical cloud city, with miniature buildings and flying vehicles emerging from the swirls of ice creamA surreal interpretation of an ice cream cone transforming into a fantastical cloud city, with miniature buildings and flying vehicles emerging from the swirls of ice cream

Prompt

negative_prompt

string

Shift + Return to add a new line

NSFW, nude, naked, porn, ugly, melting, dripping, realistic ice creamNSFW, nude, naked, porn, ugly, melting, dripping, realistic ice cream

Negative prompt

Default: "NSFW, nude, naked, porn, ugly"

structure

string

Structure type

Default: "canny"

aspect_ratio

string

Aspect ratio for the generated image. Note that the model performs best at 1024x1024 resolution. Other sizes may yield suboptimal results.

Default: "1:1"

num_outputs

integer

(minimum: 1, maximum: 4)

Number of images to output.

Default: 1

inference_steps

integer

(minimum: 1, maximum: 50)

Inference steps

Default: 25

guidance_scale

number

(minimum: 0, maximum: 50)

Guidance scale

Default: 7

control_weight

number

(minimum: 0, maximum: 1)

Control weight

Default: 0.7

low_threshold

integer

(minimum: 1, maximum: 255)

[Canny only] Line detection low threshold

Default: 100

high_threshold

integer

(minimum: 1, maximum: 255)

[Canny only] Line detection high threshold

Default: 200

seed

integer

Random seed. Leave blank to randomize the seed

output_format

string

Format of the output images

Default: "webp"

output_quality

integer

(minimum: 0, maximum: 100)

Quality of the output images, from 0 to 100. 100 is best quality, 0 is lowest quality.

Default: 80

disable_safety_checker

boolean

Disable safety checker for generated images. This feature is only available through the API. See [https://replicate.com/docs/how-does-replicate-work#safety](https://replicate.com/docs/how-does-replicate-work#safety)

Default: false

Run this model in Node.js with one line of code:

npx create-replicate --model=zsxkib/sd3-controlnet

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run zsxkib/sd3-controlnet using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "zsxkib/sd3-controlnet:1512680a528ba127ad49e92e36780974e5aa12c7d717af83c4a98a9253adf2c6",
  {
    input: {
      prompt: "A surreal interpretation of an ice cream cone transforming into a fantastical cloud city, with miniature buildings and flying vehicles emerging from the swirls of ice cream",
      structure: "canny",
      input_image: "https://replicate.delivery/pbxt/L8XLfd3JY13a51BlD1xA1APD4lPCqFgVJD5pG00Xpr4klsbQ/canny.png",
      num_outputs: 1,
      aspect_ratio: "1:1",
      low_threshold: 1,
      output_format: "webp",
      control_weight: 0.7,
      guidance_scale: 7,
      high_threshold: 255,
      output_quality: 90,
      inference_steps: 20,
      negative_prompt: "NSFW, nude, naked, porn, ugly, melting, dripping, realistic ice cream"
    }
  }
);
console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run zsxkib/sd3-controlnet using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "zsxkib/sd3-controlnet:1512680a528ba127ad49e92e36780974e5aa12c7d717af83c4a98a9253adf2c6",
    input={
        "prompt": "A surreal interpretation of an ice cream cone transforming into a fantastical cloud city, with miniature buildings and flying vehicles emerging from the swirls of ice cream",
        "structure": "canny",
        "input_image": "https://replicate.delivery/pbxt/L8XLfd3JY13a51BlD1xA1APD4lPCqFgVJD5pG00Xpr4klsbQ/canny.png",
        "num_outputs": 1,
        "aspect_ratio": "1:1",
        "low_threshold": 1,
        "output_format": "webp",
        "control_weight": 0.7,
        "guidance_scale": 7,
        "high_threshold": 255,
        "output_quality": 90,
        "inference_steps": 20,
        "negative_prompt": "NSFW, nude, naked, porn, ugly, melting, dripping, realistic ice cream"
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run zsxkib/sd3-controlnet using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "1512680a528ba127ad49e92e36780974e5aa12c7d717af83c4a98a9253adf2c6",
    "input": {
      "prompt": "A surreal interpretation of an ice cream cone transforming into a fantastical cloud city, with miniature buildings and flying vehicles emerging from the swirls of ice cream",
      "structure": "canny",
      "input_image": "https://replicate.delivery/pbxt/L8XLfd3JY13a51BlD1xA1APD4lPCqFgVJD5pG00Xpr4klsbQ/canny.png",
      "num_outputs": 1,
      "aspect_ratio": "1:1",
      "low_threshold": 1,
      "output_format": "webp",
      "control_weight": 0.7,
      "guidance_scale": 7,
      "high_threshold": 255,
      "output_quality": 90,
      "inference_steps": 20,
      "negative_prompt": "NSFW, nude, naked, porn, ugly, melting, dripping, realistic ice cream"
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

{
  "completed_at": "2024-06-21T19:32:19.903292Z",
  "created_at": "2024-06-21T19:32:11.347000Z",
  "data_removed": false,
  "error": null,
  "id": "9g6b0dqftdrgg0cg7h49p5tx44",
  "input": {
    "prompt": "A surreal interpretation of an ice cream cone transforming into a fantastical cloud city, with miniature buildings and flying vehicles emerging from the swirls of ice cream",
    "structure": "canny",
    "input_image": "https://replicate.delivery/pbxt/L8XLfd3JY13a51BlD1xA1APD4lPCqFgVJD5pG00Xpr4klsbQ/canny.png",
    "num_outputs": 1,
    "aspect_ratio": "1:1",
    "low_threshold": 1,
    "output_format": "webp",
    "control_weight": 0.7,
    "guidance_scale": 7,
    "high_threshold": 255,
    "output_quality": 90,
    "inference_steps": 20,
    "negative_prompt": "NSFW, nude, naked, porn, ugly, melting, dripping, realistic ice cream"
  },
  "logs": "Using seed: 41082\nReusing existing pipeline for structure: canny\n  0%|          | 0/20 [00:00<?, ?it/s]\n  5%|▌         | 1/20 [00:00<00:05,  3.32it/s]\n 10%|█         | 2/20 [00:00<00:05,  3.30it/s]\n 15%|█▌        | 3/20 [00:00<00:05,  3.28it/s]\n 20%|██        | 4/20 [00:01<00:04,  3.29it/s]\n 25%|██▌       | 5/20 [00:01<00:04,  3.29it/s]\n 30%|███       | 6/20 [00:01<00:04,  3.29it/s]\n 35%|███▌      | 7/20 [00:02<00:03,  3.28it/s]\n 40%|████      | 8/20 [00:02<00:03,  3.28it/s]\n 45%|████▌     | 9/20 [00:02<00:03,  3.28it/s]\n 50%|█████     | 10/20 [00:03<00:03,  3.28it/s]\n 55%|█████▌    | 11/20 [00:03<00:02,  3.29it/s]\n 60%|██████    | 12/20 [00:03<00:02,  3.30it/s]\n 65%|██████▌   | 13/20 [00:03<00:02,  3.30it/s]\n 70%|███████   | 14/20 [00:04<00:01,  3.30it/s]\n 75%|███████▌  | 15/20 [00:04<00:01,  3.30it/s]\n 80%|████████  | 16/20 [00:04<00:01,  3.31it/s]\n 85%|████████▌ | 17/20 [00:05<00:00,  3.31it/s]\n 90%|█████████ | 18/20 [00:05<00:00,  3.30it/s]\n 95%|█████████▌| 19/20 [00:05<00:00,  3.30it/s]\n100%|██████████| 20/20 [00:06<00:00,  3.30it/s]\n100%|██████████| 20/20 [00:06<00:00,  3.29it/s]\n[~] Saving to output_0.webp...\n[~] Output format: WEBP\n[~] Output quality: 90",
  "metrics": {
    "predict_time": 8.522457477,
    "total_time": 8.556292
  },
  "output": [
    "https://replicate.delivery/pbxt/nba7X1PHtfXhTCoZ7ZbmnMkjFE4AUeVeQVVenAXZyXjL0hDMB/output_0.webp"
  ],
  "started_at": "2024-06-21T19:32:11.380834Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/9g6b0dqftdrgg0cg7h49p5tx44",
    "cancel": "https://api.replicate.com/v1/predictions/9g6b0dqftdrgg0cg7h49p5tx44/cancel"
  },
  "version": "1512680a528ba127ad49e92e36780974e5aa12c7d717af83c4a98a9253adf2c6"
}

Generated in

8.5 seconds

Tweak it Share Report

Examples

View more examples

Run time and cost

This model costs approximately $0.035 to run on Replicate, or 28 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia L40S GPU hardware. Predictions typically complete within 36 seconds. The predict time for this model varies significantly based on the inputs.

Readme

✨Stable Diffusion 3 w/ ⚡InstantX’s Canny, Pose, and Tile ControlNets🖼️

About

Implementation of InstantX/SD3-Controlnet-Canny, InstantX/SD3-Controlnet-Pose, and InstantX/SD3-Controlnet-Tile

Changelog

Added InstantX/SD3-Controlnet-Canny.
PNGs with alpha (transparency) channels are now converted to RGB.
Added InstantX/SD3-Controlnet-Pose.
Added InstantX/SD3-Controlnet-Tile. Implemented lazy loading for controlnets to manage GPU memory limitations, as loading all three controlnets simultaneously is not feasible.

Examples

Tile

Here are examples of outputs with different weights applied to the control image:

Control Image	Weight=0.0	Weight=0.3	Weight=0.5	Weight=0.7	Weight=0.9

Pose

Control Image	Weight=0.0	Weight=0.3	Weight=0.5	Weight=0.7	Weight=0.9

Canny

Control Image	Weight=0.0	Weight=0.3	Weight=0.5	Weight=0.7	Weight=0.9

Limitations

Due to the fact that only 1024x1024 pixel resolution was used during the training phase, the inference performs best at this size, with other sizes yielding suboptimal results. We will initiate multi-resolution training in the future, and at that time, we will open-source the new weights.

Stable Diffusion 3 Medium is a 2 billion parameter text-to-image model developed by Stability AI. It excels at photorealism, typography, and prompt following.

Stable Diffusion 3 on Replicate can be used for commercial work.

Core Model

Architecture diagram

Stable Diffusion 3 Medium is a Multimodal Diffusion Transformer (MMDiT) text-to-image model that features greatly improved performance in image quality, typography, complex prompt understanding, and resource-efficiency.

For more technical details, please refer to the Research paper.

Safety

As part of our safety-by-design and responsible AI deployment approach, Stability AI implement safety measures throughout the development of our models, from the time we begin pre-training a model to the ongoing development, fine-tuning, and deployment of each model. We have implemented a number of safety mitigations that are intended to reduce the risk of severe harms, however we recommend that developers conduct their own testing and apply additional mitigations based on their specific use cases.

For more about our approach to Safety, please visit our Safety page.

Support

All credit goes to the InstantX team Give me a follow on Twitter if you like my work! @zsakib_