mikelyndon/controlnet-multi | Run with an API on Replicate

mikelyndon / controlnet-multi

A multi-input ControlNet model. Pass in control images and set the weights. (Updated 2 years ago)

Cold

Public
253 runs
T4
GitHub

Iterate in playground

Run with an API

Playground API Examples README Versions

Input

prompt

string

Shift + Return to add a new line

a giant standing in a fantasy landscape, best qualitya giant standing in a fantasy landscape, best quality

Input prompt

Default: "a magpie wearing a tophat"

negative_prompt

string

Shift + Return to add a new line

monochrome, lowres, bad anatomy, worst quality, low qualitymonochrome, lowres, bad anatomy, worst quality, low quality

Specify things to not see in the output

Default: "monochrome, lowres, bad anatomy, worst quality, low quality"

seed

integer

Random seed. Leave blank to randomize the seed

canny

*file

Preview

Canny input image

pose

*file

Preview

Pose input image

depth

*file

Preview

Depth input image

mlsd

*file

Preview

MLSD input image

seg

*file

Preview

SEG input image

softedge

*file

Preview

Soft Edge (hed) input image

canny_weight

number

(minimum: 0, maximum: 1)

A value between 0 and 1 to emphasize the conditioning by the canny edges.

Default: 1

pose_weight

number

(minimum: 0, maximum: 1)

A value between 0 and 1 to emphasize the conditioning by the open pose.

Default: 0

depth_weight

number

(minimum: 0, maximum: 1)

A value between 0 and 1 to emphasize the conditioning by the depth.

Default: 0

mlsd_weight

number

(minimum: 0, maximum: 1)

A value between 0 and 1 to emphasize the conditioning by the mlsd.

Default: 0

seg_weight

number

(minimum: 0, maximum: 1)

A value between 0 and 1 to emphasize the conditioning by the seg.

Default: 0

softedge_weight

number

(minimum: 0, maximum: 1)

A value between 0 and 1 to emphasize the conditioning by the soft edge (hed).

Default: 0

Run this model in Node.js with one line of code:

npx create-replicate --model=mikelyndon/controlnet-multi

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";
import fs from "node:fs";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run mikelyndon/controlnet-multi using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "mikelyndon/controlnet-multi:69d16083e38779cdb340f7343a6594611b6b65ffd7715f783eef53b9640268e6",
  {
    input: {
      seg: "https://replicate.delivery/pbxt/IrarVUc6AdoRj3e5ypGIPSVHsThTOsAGZoD4xnsSJOzsVXt9/empty.png",
      mlsd: "https://replicate.delivery/pbxt/IrarWDb4d9n1pWwEV3na6aZogpYW0qpwZ4FNIFxq3B21nrPU/empty.png",
      pose: "https://replicate.delivery/pbxt/IrarVkMTOYSZ08KZWAMcA3ed87zelT55XdfoYaq4drAWI4G1/pose_man.png",
      canny: "https://replicate.delivery/pbxt/IrarVflkkV3TsUxP0sggxmthggyLFpUxrfOgY1CINeDbMY2J/canny_man.png",
      depth: "https://replicate.delivery/pbxt/IrarW1cJkljr1M6wZgoOqf5H1ERAobcp25R3VVpTe38hukqb/empty.png",
      prompt: "a giant standing in a fantasy landscape, best quality",
      softedge: "https://replicate.delivery/pbxt/IrarVcg0YvuRzjA6G0yelEeGq8IbQFIktkjga3X9zbxEJTYs/empty.png",
      seg_weight: 0,
      mlsd_weight: 0,
      pose_weight: 0.8,
      canny_weight: 1,
      depth_weight: 0,
      negative_prompt: "monochrome, lowres, bad anatomy, worst quality, low quality",
      softedge_weight: 0
    }
  }
);

// To access the file URL:
console.log(output.url()); //=> "http://example.com"

// To write the file to disk:
fs.writeFile("my-image.png", output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run mikelyndon/controlnet-multi using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "mikelyndon/controlnet-multi:69d16083e38779cdb340f7343a6594611b6b65ffd7715f783eef53b9640268e6",
    input={
        "seg": "https://replicate.delivery/pbxt/IrarVUc6AdoRj3e5ypGIPSVHsThTOsAGZoD4xnsSJOzsVXt9/empty.png",
        "mlsd": "https://replicate.delivery/pbxt/IrarWDb4d9n1pWwEV3na6aZogpYW0qpwZ4FNIFxq3B21nrPU/empty.png",
        "pose": "https://replicate.delivery/pbxt/IrarVkMTOYSZ08KZWAMcA3ed87zelT55XdfoYaq4drAWI4G1/pose_man.png",
        "canny": "https://replicate.delivery/pbxt/IrarVflkkV3TsUxP0sggxmthggyLFpUxrfOgY1CINeDbMY2J/canny_man.png",
        "depth": "https://replicate.delivery/pbxt/IrarW1cJkljr1M6wZgoOqf5H1ERAobcp25R3VVpTe38hukqb/empty.png",
        "prompt": "a giant standing in a fantasy landscape, best quality",
        "softedge": "https://replicate.delivery/pbxt/IrarVcg0YvuRzjA6G0yelEeGq8IbQFIktkjga3X9zbxEJTYs/empty.png",
        "seg_weight": 0,
        "mlsd_weight": 0,
        "pose_weight": 0.8,
        "canny_weight": 1,
        "depth_weight": 0,
        "negative_prompt": "monochrome, lowres, bad anatomy, worst quality, low quality",
        "softedge_weight": 0
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run mikelyndon/controlnet-multi using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "mikelyndon/controlnet-multi:69d16083e38779cdb340f7343a6594611b6b65ffd7715f783eef53b9640268e6",
    "input": {
      "seg": "https://replicate.delivery/pbxt/IrarVUc6AdoRj3e5ypGIPSVHsThTOsAGZoD4xnsSJOzsVXt9/empty.png",
      "mlsd": "https://replicate.delivery/pbxt/IrarWDb4d9n1pWwEV3na6aZogpYW0qpwZ4FNIFxq3B21nrPU/empty.png",
      "pose": "https://replicate.delivery/pbxt/IrarVkMTOYSZ08KZWAMcA3ed87zelT55XdfoYaq4drAWI4G1/pose_man.png",
      "canny": "https://replicate.delivery/pbxt/IrarVflkkV3TsUxP0sggxmthggyLFpUxrfOgY1CINeDbMY2J/canny_man.png",
      "depth": "https://replicate.delivery/pbxt/IrarW1cJkljr1M6wZgoOqf5H1ERAobcp25R3VVpTe38hukqb/empty.png",
      "prompt": "a giant standing in a fantasy landscape, best quality",
      "softedge": "https://replicate.delivery/pbxt/IrarVcg0YvuRzjA6G0yelEeGq8IbQFIktkjga3X9zbxEJTYs/empty.png",
      "seg_weight": 0,
      "mlsd_weight": 0,
      "pose_weight": 0.8,
      "canny_weight": 1,
      "depth_weight": 0,
      "negative_prompt": "monochrome, lowres, bad anatomy, worst quality, low quality",
      "softedge_weight": 0
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

{
  "completed_at": "2023-05-21T17:53:14.803147Z",
  "created_at": "2023-05-21T17:48:03.302041Z",
  "data_removed": false,
  "error": null,
  "id": "km2pxk6pbrbn5jkwbl2qg5jhkq",
  "input": {
    "seg": "https://replicate.delivery/pbxt/IrarVUc6AdoRj3e5ypGIPSVHsThTOsAGZoD4xnsSJOzsVXt9/empty.png",
    "mlsd": "https://replicate.delivery/pbxt/IrarWDb4d9n1pWwEV3na6aZogpYW0qpwZ4FNIFxq3B21nrPU/empty.png",
    "pose": "https://replicate.delivery/pbxt/IrarVkMTOYSZ08KZWAMcA3ed87zelT55XdfoYaq4drAWI4G1/pose_man.png",
    "canny": "https://replicate.delivery/pbxt/IrarVflkkV3TsUxP0sggxmthggyLFpUxrfOgY1CINeDbMY2J/canny_man.png",
    "depth": "https://replicate.delivery/pbxt/IrarW1cJkljr1M6wZgoOqf5H1ERAobcp25R3VVpTe38hukqb/empty.png",
    "prompt": "a giant standing in a fantasy landscape, best quality",
    "softedge": "https://replicate.delivery/pbxt/IrarVcg0YvuRzjA6G0yelEeGq8IbQFIktkjga3X9zbxEJTYs/empty.png",
    "pose_weight": 0.8,
    "canny_weight": 1,
    "negative_prompt": "monochrome, lowres, bad anatomy, worst quality, low quality"
  },
  "logs": "0%|          | 0/20 [00:00<?, ?it/s]\n  5%|▌         | 1/20 [00:05<01:42,  5.39s/it]\n 10%|█         | 2/20 [00:06<00:48,  2.67s/it]\n 15%|█▌        | 3/20 [00:08<00:46,  2.72s/it]\n 20%|██        | 4/20 [00:09<00:31,  1.98s/it]\n 25%|██▌       | 5/20 [00:10<00:23,  1.54s/it]\n 30%|███       | 6/20 [00:11<00:17,  1.27s/it]\n 35%|███▌      | 7/20 [00:12<00:14,  1.11s/it]\n 40%|████      | 8/20 [00:12<00:11,  1.00it/s]\n 45%|████▌     | 9/20 [00:13<00:10,  1.08it/s]\n 50%|█████     | 10/20 [00:14<00:08,  1.15it/s]\n 55%|█████▌    | 11/20 [00:15<00:07,  1.19it/s]\n 60%|██████    | 12/20 [00:15<00:06,  1.23it/s]\n 65%|██████▌   | 13/20 [00:16<00:05,  1.25it/s]\n 70%|███████   | 14/20 [00:17<00:04,  1.27it/s]\n 75%|███████▌  | 15/20 [00:18<00:03,  1.28it/s]\n 80%|████████  | 16/20 [00:18<00:03,  1.29it/s]\n 85%|████████▌ | 17/20 [00:19<00:02,  1.29it/s]\n 90%|█████████ | 18/20 [00:20<00:01,  1.29it/s]\n 95%|█████████▌| 19/20 [00:21<00:00,  1.30it/s]\n100%|██████████| 20/20 [00:21<00:00,  1.30it/s]\n100%|██████████| 20/20 [00:21<00:00,  1.10s/it]\nout_path /tmp/tmptvzifqnz/out.png",
  "metrics": {
    "predict_time": 34.753042,
    "total_time": 311.501106
  },
  "output": "https://replicate.delivery/pbxt/5Ns7TZxLyuZ6BRgrqpB2o2RuPEwvD45P6ozuJc2f2amEZAfQA/out.png",
  "started_at": "2023-05-21T17:52:40.050105Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/km2pxk6pbrbn5jkwbl2qg5jhkq",
    "cancel": "https://api.replicate.com/v1/predictions/km2pxk6pbrbn5jkwbl2qg5jhkq/cancel"
  },
  "version": "69d16083e38779cdb340f7343a6594611b6b65ffd7715f783eef53b9640268e6"
}

Generated in

34.8 seconds

Tweak itReport View full prediction

Examples

View more examples

Run time and cost

This model costs approximately $0.19 to run on Replicate, or 5 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia T4 GPU hardware. Predictions typically complete within 15 minutes. The predict time for this model varies significantly based on the inputs.

Readme

This is basically an expanded version of the HuggingFace - Running ControlNet with multiple conditions.

Check out the Model Cards.

Currently all path inputs are required. I suggest you pass a black image into the conditioning inputs you’re not using and set the scales to 0.