adirik/t2i-adapter-sdxl-openpose – Run with an API on Replicate

adirik / t2i-adapter-sdxl-openpose

Modify images using human pose

Cold

Public
61.7K runs
L40S
GitHub
Paper
License

Run with an API

Playground API Examples README Versions

Input

image

*file

Input image

prompt

string

Shift + Return to add a new line

Input prompt

Default: "A couple, 4k photo, highly detailed"

negative_prompt

string

Shift + Return to add a new line

anime, cartoon, graphic, text, painting, crayon, graphite, abstract, glitch, deformed, mutated, ugly, disfiguredanime, cartoon, graphic, text, painting, crayon, graphite, abstract, glitch, deformed, mutated, ugly, disfigured

Specify things to not see in the output

Default: "anime, cartoon, graphic, text, painting, crayon, graphite, abstract, glitch, deformed, mutated, ugly, disfigured"

num_inference_steps

integer

(minimum: 0, maximum: 100)

Number of diffusion steps

Default: 30

adapter_conditioning_scale

number

(minimum: 0, maximum: 5)

Conditioning scale

Default: 1

adapter_conditioning_factor

number

(minimum: 0, maximum: 1)

Factor to scale image by

Default: 1

guidance_scale

number

(minimum: 0, maximum: 10)

Guidance scale to match the prompt

Default: 7.5

num_samples

integer

(minimum: 1, maximum: 4)

Number of outputs to generate

Default: 1

scheduler

string

Which scheduler to use

Default: "K_EULER_ANCESTRAL"

random_seed

integer

Random seed for reproducibility, leave blank to randomize output

Run this model in Node.js with one line of code:

npx create-replicate --model=adirik/t2i-adapter-sdxl-openpose

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run adirik/t2i-adapter-sdxl-openpose using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "adirik/t2i-adapter-sdxl-openpose:ff250494f7328552c64ee50ae3ed9b61e09ca18c7aa51f77ed187a3fb9ec9093",
  {
    input: {
      image: "https://replicate.delivery/pbxt/JbnAELoOIkMhteHqHJnRfB0ATKgRdZqLjdIgcZB34WlRNCNF/people.jpg",
      prompt: "A couple, 4k photo, highly detailed",
      scheduler: "K_EULER_ANCESTRAL",
      num_samples: 1,
      guidance_scale: 7.5,
      negative_prompt: "anime, cartoon, graphic, text, painting, crayon, graphite, abstract, glitch, deformed, mutated, ugly, disfigured",
      num_inference_steps: 30,
      adapter_conditioning_scale: 0.9,
      adapter_conditioning_factor: 0.9
    }
  }
);

// To access the file URL:
console.log(output[0].url()); //=> "http://example.com"

// To write the file to disk:
fs.writeFile("my-image.png", output[0]);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run adirik/t2i-adapter-sdxl-openpose using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "adirik/t2i-adapter-sdxl-openpose:ff250494f7328552c64ee50ae3ed9b61e09ca18c7aa51f77ed187a3fb9ec9093",
    input={
        "image": "https://replicate.delivery/pbxt/JbnAELoOIkMhteHqHJnRfB0ATKgRdZqLjdIgcZB34WlRNCNF/people.jpg",
        "prompt": "A couple, 4k photo, highly detailed",
        "scheduler": "K_EULER_ANCESTRAL",
        "num_samples": 1,
        "guidance_scale": 7.5,
        "negative_prompt": "anime, cartoon, graphic, text, painting, crayon, graphite, abstract, glitch, deformed, mutated, ugly, disfigured",
        "num_inference_steps": 30,
        "adapter_conditioning_scale": 0.9,
        "adapter_conditioning_factor": 0.9
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run adirik/t2i-adapter-sdxl-openpose using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "ff250494f7328552c64ee50ae3ed9b61e09ca18c7aa51f77ed187a3fb9ec9093",
    "input": {
      "image": "https://replicate.delivery/pbxt/JbnAELoOIkMhteHqHJnRfB0ATKgRdZqLjdIgcZB34WlRNCNF/people.jpg",
      "prompt": "A couple, 4k photo, highly detailed",
      "scheduler": "K_EULER_ANCESTRAL",
      "num_samples": 1,
      "guidance_scale": 7.5,
      "negative_prompt": "anime, cartoon, graphic, text, painting, crayon, graphite, abstract, glitch, deformed, mutated, ugly, disfigured",
      "num_inference_steps": 30,
      "adapter_conditioning_scale": 0.9,
      "adapter_conditioning_factor": 0.9
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

{
  "completed_at": "2023-10-03T18:51:19.116216Z",
  "created_at": "2023-10-03T18:49:23.572065Z",
  "data_removed": false,
  "error": null,
  "id": "gk2lem3bosrbmdz5itfokdtvhe",
  "input": {
    "image": "https://replicate.delivery/pbxt/JbnAELoOIkMhteHqHJnRfB0ATKgRdZqLjdIgcZB34WlRNCNF/people.jpg",
    "prompt": "A couple, 4k photo, highly detailed",
    "scheduler": "K_EULER_ANCESTRAL",
    "num_samples": 1,
    "guidance_scale": 7.5,
    "negative_prompt": "anime, cartoon, graphic, text, painting, crayon, graphite, abstract, glitch, deformed, mutated, ugly, disfigured",
    "num_inference_steps": 30,
    "adapter_conditioning_scale": 0.9,
    "adapter_conditioning_factor": 0.9
  },
  "logs": "0%|          | 0/30 [00:00<?, ?it/s]\n  3%|▎         | 1/30 [00:00<00:04,  6.05it/s]\n  7%|▋         | 2/30 [00:00<00:06,  4.50it/s]\n 10%|█         | 3/30 [00:00<00:06,  4.12it/s]\n 13%|█▎        | 4/30 [00:00<00:06,  3.97it/s]\n 17%|█▋        | 5/30 [00:01<00:06,  3.89it/s]\n 20%|██        | 6/30 [00:01<00:06,  3.83it/s]\n 23%|██▎       | 7/30 [00:01<00:06,  3.80it/s]\n 27%|██▋       | 8/30 [00:02<00:05,  3.78it/s]\n 30%|███       | 9/30 [00:02<00:05,  3.77it/s]\n 33%|███▎      | 10/30 [00:02<00:05,  3.76it/s]\n 37%|███▋      | 11/30 [00:02<00:05,  3.74it/s]\n 40%|████      | 12/30 [00:03<00:04,  3.74it/s]\n 43%|████▎     | 13/30 [00:03<00:04,  3.74it/s]\n 47%|████▋     | 14/30 [00:03<00:04,  3.73it/s]\n 50%|█████     | 15/30 [00:03<00:04,  3.73it/s]\n 53%|█████▎    | 16/30 [00:04<00:03,  3.73it/s]\n 57%|█████▋    | 17/30 [00:04<00:03,  3.73it/s]\n 60%|██████    | 18/30 [00:04<00:03,  3.72it/s]\n 63%|██████▎   | 19/30 [00:04<00:02,  3.73it/s]\n 67%|██████▋   | 20/30 [00:05<00:02,  3.72it/s]\n 70%|███████   | 21/30 [00:05<00:02,  3.72it/s]\n 73%|███████▎  | 22/30 [00:05<00:02,  3.72it/s]\n 77%|███████▋  | 23/30 [00:06<00:01,  3.73it/s]\n 80%|████████  | 24/30 [00:06<00:01,  3.72it/s]\n 83%|████████▎ | 25/30 [00:06<00:01,  3.72it/s]\n 87%|████████▋ | 26/30 [00:06<00:01,  3.72it/s]\n 90%|█████████ | 27/30 [00:07<00:00,  3.71it/s]\n 93%|█████████▎| 28/30 [00:07<00:00,  3.72it/s]\n 97%|█████████▋| 29/30 [00:07<00:00,  3.72it/s]\n100%|██████████| 30/30 [00:07<00:00,  3.72it/s]\n100%|██████████| 30/30 [00:07<00:00,  3.78it/s]",
  "metrics": {
    "predict_time": 12.428266,
    "total_time": 115.544151
  },
  "output": [
    "https://pbxt.replicate.delivery/zLV4rptJWbKOKpMauzP4ASa0ZwMhs29NYD4Ckdy3uhSpUoaE/out-preprocessed.png",
    "https://pbxt.replicate.delivery/nuE2A9YWRkLaN1ptkgad1JeHtYJwUWAt6p7Bie0fYDcNlCVjA/out-0.png"
  ],
  "started_at": "2023-10-03T18:51:06.687950Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/gk2lem3bosrbmdz5itfokdtvhe",
    "cancel": "https://api.replicate.com/v1/predictions/gk2lem3bosrbmdz5itfokdtvhe/cancel"
  },
  "version": "6e87d127eb184d446b8cec0c2b034b3c7b24d08e211ec28247cdeb2e0a460670"
}

Generated in

12.4 seconds

Tweak itReport

This output was created using a different version of the model, adirik/t2i-adapter-sdxl-openpose:6e87d127.

Examples

View more examples

Run time and cost

This model costs approximately $0.039 to run on Replicate, or 25 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia L40S GPU hardware. Predictions typically complete within 40 seconds. The predict time for this model varies significantly based on the inputs.

Readme

Model Description

T2I-Adapter based on Stable Diffusion-XL by Tencent ARC Lab and Peking University VILLA. Cog-wrapper is adapted from the [official repository.]( (https://github.com/TencentARC/T2I-Adapter). T2I-Adapter performs image editing using text prompts combined with depth map, human body pose, line art, canny edge and sketch conditions.

Abstract: We propose T2I-Adapter, a simple and small (~70M parameters, ~300M storage space) network that can provide extra guidance to pre-trained text-to-image models while freezing the original large text-to-image models.

T2I-Adapter aligns internal knowledge in T2I models with external control signals. We can train various adapters according to different conditions, and achieve rich control and editing effects.

See the paper, official repository and Hugging Face model page and demo for more information.

Usage

To start, upload an image you would like to modify and prompt the model to generate an image as you would for Stable Diffusion. The model generating the image will use your input image as a template and internally detect the body pose to guide image generation.

Other T2I-Adapter Models

There are many different ways to use a T2I-Adapter to modify the output of Stable Diffusion XL and Stable Diffusion. Here are a few different options, all of which use an input image in addition to a prompt to generate an output. The methods process the input in different ways; try them out to see which works best for a given application.

T2I-Adapter for generating images from sketches
https://replicate.com/alaradirik/t2i-adapter-sdxl-sketch

T2I-Adapter for preserving general qualities about an input image
https://replicate.com/alaradirik/t2i-adapter-sdxl-canny https://replicate.com/alaradirik/t2i-adapter-sdxl-lineart https://replicate.com/alaradirik/t2i-adapter-sdxl-depth-midas

T2I-Adapter SD
https://replicate.com/cjwbw/t2i-adapter

Citation

@article{mou2023t2i,
  title={T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models},
  author={Mou, Chong and Wang, Xintao and Xie, Liangbin and Wu, Yanze and Zhang, Jian and Qi, Zhongang and Shan, Ying and Qie, Xiaohu},
  journal={arXiv preprint arXiv:2302.08453},
  year={2023}
}