cloneofsimo/hotshot-xl-lora-controlnet:75e26ffd

Version

You're looking at a specific version of this model. Jump to the model overview.

cloneofsimo /hotshot-xl-lora-controlnet:75e26ffd

Playground API

Input

prompt

string

Shift + Return to add a new line

SHRMI dog dancing in space, colorful galaxies on backgroundSHRMI dog dancing in space, colorful galaxies on background

The main prompt that guides the image generation.

Default: "Hi there doggo!"

negative_prompt

string

Shift + Return to add a new line

A negative prompt to avoid certain features in the generated images.

Default: ""

width

integer

The width of the generated images.

Default: 672

height

integer

The height of the generated images.

Default: 384

original_width

integer

The width of the `original_size` of images. If `original_size` is not the same as `target_size` the image will appear to be down- or upsampled. `original_size` defaults to `(width, height)` if not specified. Part of SDXL's micro-conditioning as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952).

Default: 1920

original_height

integer

The `original_size height` of the images.

Default: 1080

target_width

integer

The `target_size width` of the images.

Default: 512

target_height

integer

The `target_size height` of the images.

Default: 512

steps

integer

The number of steps for the prediction.

Default: 30

video_length

integer

The length of the video in frames.

Default: 8

video_duration

integer

The duration of the video in milliseconds.

Default: 1000

hf_lora_url

string

Shift + Return to add a new line

The URL for LoRA weights.

control_type

string

The type of control net to use for conditional generation.

gif

file

Input GIF for controlnet condition.

control_guidance_start

number

The start of the control guidance.

Default: 0

control_guidance_end

number

The end of the control guidance.

Default: 1

controlnet_conditioning_scale

number

The scale of the controlnet conditioning.

Default: 0.7

replicate_weights_url

string

Shift + Return to add a new line

https://pbxt.replicate.delivery/i2HfjZWkJC1yIqHPDaPHOx9eJsqFqZ2tnWMgn5FFd9evSGWjA/trained_model.tar?https://pbxt.replicate.delivery/i2HfjZWkJC1yIqHPDaPHOx9eJsqFqZ2tnWMgn5FFd9evSGWjA/trained_model.tar?

Replicate LoRA weights to use. Leave blank to use the default weights.

Run this model in Node.js with one line of code:

npx create-replicate --model=cloneofsimo/hotshot-xl-lora-controlnet

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";
import fs from "node:fs";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run cloneofsimo/hotshot-xl-lora-controlnet using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "cloneofsimo/hotshot-xl-lora-controlnet:75e26ffd033a59a78954a3d675632f47f7f8470402aec51c255b9f9b7b62568b",
  {
    input: {
      gif: "https://replicate.delivery/pbxt/JeGg0LkufHuNpTwb8fVpXPAW7M6B6GYB1WfFIuy8LuJWwBNp/tmp2.gif",
      steps: 30,
      width: 672,
      height: 384,
      prompt: "SHRMI dog dancing in space, colorful galaxies on background",
      control_type: "depth",
      target_width: 512,
      video_length: 8,
      target_height: 512,
      original_width: 1920,
      video_duration: 1000,
      negative_prompt: "",
      original_height: 1080,
      control_guidance_end: 1,
      replicate_weights_url: "https://pbxt.replicate.delivery/i2HfjZWkJC1yIqHPDaPHOx9eJsqFqZ2tnWMgn5FFd9evSGWjA/trained_model.tar?",
      control_guidance_start: 0,
      controlnet_conditioning_scale: 1
    }
  }
);

// To access the file URL:
console.log(output.url()); //=> "http://example.com"

// To write the file to disk:
fs.writeFile("my-image.png", output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run cloneofsimo/hotshot-xl-lora-controlnet using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "cloneofsimo/hotshot-xl-lora-controlnet:75e26ffd033a59a78954a3d675632f47f7f8470402aec51c255b9f9b7b62568b",
    input={
        "gif": "https://replicate.delivery/pbxt/JeGg0LkufHuNpTwb8fVpXPAW7M6B6GYB1WfFIuy8LuJWwBNp/tmp2.gif",
        "steps": 30,
        "width": 672,
        "height": 384,
        "prompt": "SHRMI dog dancing in space, colorful galaxies on background",
        "control_type": "depth",
        "target_width": 512,
        "video_length": 8,
        "target_height": 512,
        "original_width": 1920,
        "video_duration": 1000,
        "negative_prompt": "",
        "original_height": 1080,
        "control_guidance_end": 1,
        "replicate_weights_url": "https://pbxt.replicate.delivery/i2HfjZWkJC1yIqHPDaPHOx9eJsqFqZ2tnWMgn5FFd9evSGWjA/trained_model.tar?",
        "control_guidance_start": 0,
        "controlnet_conditioning_scale": 1
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run cloneofsimo/hotshot-xl-lora-controlnet using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "cloneofsimo/hotshot-xl-lora-controlnet:75e26ffd033a59a78954a3d675632f47f7f8470402aec51c255b9f9b7b62568b",
    "input": {
      "gif": "https://replicate.delivery/pbxt/JeGg0LkufHuNpTwb8fVpXPAW7M6B6GYB1WfFIuy8LuJWwBNp/tmp2.gif",
      "steps": 30,
      "width": 672,
      "height": 384,
      "prompt": "SHRMI dog dancing in space, colorful galaxies on background",
      "control_type": "depth",
      "target_width": 512,
      "video_length": 8,
      "target_height": 512,
      "original_width": 1920,
      "video_duration": 1000,
      "negative_prompt": "",
      "original_height": 1080,
      "control_guidance_end": 1,
      "replicate_weights_url": "https://pbxt.replicate.delivery/i2HfjZWkJC1yIqHPDaPHOx9eJsqFqZ2tnWMgn5FFd9evSGWjA/trained_model.tar?",
      "control_guidance_start": 0,
      "controlnet_conditioning_scale": 1
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

{
  "completed_at": "2023-10-06T02:32:59.788561Z",
  "created_at": "2023-10-06T02:32:22.723245Z",
  "data_removed": false,
  "error": null,
  "id": "vhiozkdbpzryxt3fkq63oakgia",
  "input": {
    "gif": "https://replicate.delivery/pbxt/JeGg0LkufHuNpTwb8fVpXPAW7M6B6GYB1WfFIuy8LuJWwBNp/tmp2.gif",
    "steps": 30,
    "width": 672,
    "height": 384,
    "prompt": "SHRMI dog dancing in space, colorful galaxies on background",
    "control_type": "depth",
    "target_width": 512,
    "video_length": 8,
    "target_height": 512,
    "original_width": 1920,
    "video_duration": 1000,
    "negative_prompt": "",
    "original_height": 1080,
    "control_guidance_end": 1,
    "replicate_weights_url": "https://pbxt.replicate.delivery/i2HfjZWkJC1yIqHPDaPHOx9eJsqFqZ2tnWMgn5FFd9evSGWjA/trained_model.tar?",
    "control_guidance_start": 0,
    "controlnet_conditioning_scale": 1
  },
  "logs": "Loading pipeline components...:   0%|          | 0/7 [00:00<?, ?it/s]\nLoading pipeline components...:  14%|█▍        | 1/7 [00:02<00:15,  2.63s/it]\nLoading pipeline components...:  71%|███████▏  | 5/7 [00:03<00:01,  1.87it/s]\nLoading pipeline components...:  86%|████████▌ | 6/7 [00:03<00:00,  2.27it/s]\nLoading pipeline components...: 100%|██████████| 7/7 [00:03<00:00,  2.59it/s]\nLoading pipeline components...: 100%|██████████| 7/7 [00:03<00:00,  1.91it/s]\nLoading Unet LoRA\nYou have saved the LoRA weights using the old format. To convert the old LoRA weights to the new format, you can first load them in a dictionary and then create a new dictionary like the following: `new_state_dict = {f'unet.{module_name}': params for module_name, params in old_state_dict.items()}`.\n  0%|          | 0/30 [00:00<?, ?it/s]\n  3%|▎         | 1/30 [00:00<00:22,  1.31it/s]\n  7%|▋         | 2/30 [00:01<00:24,  1.17it/s]\n 10%|█         | 3/30 [00:02<00:24,  1.12it/s]\n 13%|█▎        | 4/30 [00:03<00:23,  1.11it/s]\n 17%|█▋        | 5/30 [00:04<00:22,  1.10it/s]\n 20%|██        | 6/30 [00:05<00:22,  1.09it/s]\n 23%|██▎       | 7/30 [00:06<00:21,  1.09it/s]\n 27%|██▋       | 8/30 [00:07<00:20,  1.09it/s]\n 30%|███       | 9/30 [00:08<00:19,  1.08it/s]\n 33%|███▎      | 10/30 [00:09<00:18,  1.08it/s]\n 37%|███▋      | 11/30 [00:10<00:17,  1.08it/s]\n 40%|████      | 12/30 [00:10<00:16,  1.08it/s]\n 43%|████▎     | 13/30 [00:11<00:15,  1.08it/s]\n 47%|████▋     | 14/30 [00:12<00:14,  1.08it/s]\n 50%|█████     | 15/30 [00:13<00:13,  1.08it/s]\n 53%|█████▎    | 16/30 [00:14<00:12,  1.08it/s]\n 57%|█████▋    | 17/30 [00:15<00:12,  1.08it/s]\n 60%|██████    | 18/30 [00:16<00:11,  1.08it/s]\n 63%|██████▎   | 19/30 [00:17<00:10,  1.08it/s]\n 67%|██████▋   | 20/30 [00:18<00:09,  1.08it/s]\n 70%|███████   | 21/30 [00:19<00:08,  1.08it/s]\n 73%|███████▎  | 22/30 [00:20<00:07,  1.08it/s]\n 77%|███████▋  | 23/30 [00:21<00:06,  1.08it/s]\n 80%|████████  | 24/30 [00:22<00:05,  1.08it/s]\n 83%|████████▎ | 25/30 [00:22<00:04,  1.08it/s]\n 87%|████████▋ | 26/30 [00:23<00:03,  1.08it/s]\n 90%|█████████ | 27/30 [00:24<00:02,  1.08it/s]\n 93%|█████████▎| 28/30 [00:25<00:01,  1.08it/s]\n 97%|█████████▋| 29/30 [00:26<00:00,  1.08it/s]\n100%|██████████| 30/30 [00:27<00:00,  1.08it/s]\n100%|██████████| 30/30 [00:27<00:00,  1.09it/s]\n  0%|          | 0/8 [00:00<?, ?it/s]\n 25%|██▌       | 2/8 [00:00<00:00, 13.40it/s]\n 50%|█████     | 4/8 [00:00<00:00, 14.77it/s]\n 75%|███████▌  | 6/8 [00:00<00:00, 14.81it/s]\n100%|██████████| 8/8 [00:00<00:00, 14.84it/s]\n100%|██████████| 8/8 [00:00<00:00, 14.70it/s]",
  "metrics": {
    "predict_time": 37.093352,
    "total_time": 37.065316
  },
  "output": "https://pbxt.replicate.delivery/sGBhVb69oZYaA1qG5D5R2tOZUNwhgH7FdbeQjJn8n0GtHp1IA/tmp.gif",
  "started_at": "2023-10-06T02:32:22.695209Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/vhiozkdbpzryxt3fkq63oakgia",
    "cancel": "https://api.replicate.com/v1/predictions/vhiozkdbpzryxt3fkq63oakgia/cancel"
  },
  "version": "75e26ffd033a59a78954a3d675632f47f7f8470402aec51c255b9f9b7b62568b"
}

Generated in

37.1 seconds

Tweak it Report

Loading pipeline components...:   0%|          | 0/7 [00:00<?, ?it/s]
Loading pipeline components...:  14%|█▍        | 1/7 [00:02<00:15,  2.63s/it]
Loading pipeline components...:  71%|███████▏  | 5/7 [00:03<00:01,  1.87it/s]
Loading pipeline components...:  86%|████████▌ | 6/7 [00:03<00:00,  2.27it/s]
Loading pipeline components...: 100%|██████████| 7/7 [00:03<00:00,  2.59it/s]
Loading pipeline components...: 100%|██████████| 7/7 [00:03<00:00,  1.91it/s]
Loading Unet LoRA
You have saved the LoRA weights using the old format. To convert the old LoRA weights to the new format, you can first load them in a dictionary and then create a new dictionary like the following: `new_state_dict = {f'unet.{module_name}': params for module_name, params in old_state_dict.items()}`.
  0%|          | 0/30 [00:00<?, ?it/s]
  3%|▎         | 1/30 [00:00<00:22,  1.31it/s]
  7%|▋         | 2/30 [00:01<00:24,  1.17it/s]
 10%|█         | 3/30 [00:02<00:24,  1.12it/s]
 13%|█▎        | 4/30 [00:03<00:23,  1.11it/s]
 17%|█▋        | 5/30 [00:04<00:22,  1.10it/s]
 20%|██        | 6/30 [00:05<00:22,  1.09it/s]
 23%|██▎       | 7/30 [00:06<00:21,  1.09it/s]
 27%|██▋       | 8/30 [00:07<00:20,  1.09it/s]
 30%|███       | 9/30 [00:08<00:19,  1.08it/s]
 33%|███▎      | 10/30 [00:09<00:18,  1.08it/s]
 37%|███▋      | 11/30 [00:10<00:17,  1.08it/s]
 40%|████      | 12/30 [00:10<00:16,  1.08it/s]
 43%|████▎     | 13/30 [00:11<00:15,  1.08it/s]
 47%|████▋     | 14/30 [00:12<00:14,  1.08it/s]
 50%|█████     | 15/30 [00:13<00:13,  1.08it/s]
 53%|█████▎    | 16/30 [00:14<00:12,  1.08it/s]
 57%|█████▋    | 17/30 [00:15<00:12,  1.08it/s]
 60%|██████    | 18/30 [00:16<00:11,  1.08it/s]
 63%|██████▎   | 19/30 [00:17<00:10,  1.08it/s]
 67%|██████▋   | 20/30 [00:18<00:09,  1.08it/s]
 70%|███████   | 21/30 [00:19<00:08,  1.08it/s]
 73%|███████▎  | 22/30 [00:20<00:07,  1.08it/s]
 77%|███████▋  | 23/30 [00:21<00:06,  1.08it/s]
 80%|████████  | 24/30 [00:22<00:05,  1.08it/s]
 83%|████████▎ | 25/30 [00:22<00:04,  1.08it/s]
 87%|████████▋ | 26/30 [00:23<00:03,  1.08it/s]
 90%|█████████ | 27/30 [00:24<00:02,  1.08it/s]
 93%|█████████▎| 28/30 [00:25<00:01,  1.08it/s]
 97%|█████████▋| 29/30 [00:26<00:00,  1.08it/s]
100%|██████████| 30/30 [00:27<00:00,  1.08it/s]
100%|██████████| 30/30 [00:27<00:00,  1.09it/s]
  0%|          | 0/8 [00:00<?, ?it/s]
 25%|██▌       | 2/8 [00:00<00:00, 13.40it/s]
 50%|█████     | 4/8 [00:00<00:00, 14.77it/s]
 75%|███████▌  | 6/8 [00:00<00:00, 14.81it/s]
100%|██████████| 8/8 [00:00<00:00, 14.84it/s]
100%|██████████| 8/8 [00:00<00:00, 14.70it/s]