shimmercam/animatediff-v3 | Run with an API on Replicate

shimmercam / animatediff-v3

AnimateDiff v3 + SparseCtrl: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning. Created with Shimmer.

Cold

Public
699 runs
A100 (80GB)
GitHub
Paper
License

Iterate in playground

Run with an API

Playground API Examples README Versions

Input

prompt

string

Shift + Return to add a new line

Input prompt

Default: "closeup face photo of man in black clothes, night city street, fireworks in background"

negative_prompt

string

Shift + Return to add a new line

Negative prompt

Default: "worst quality, low quality, letterboxed"

image

file

Upload a controlnet image

Default: "https://raw.githubusercontent.com/guoyww/AnimateDiff/main/__assets__/demos/image/RealisticVision_firework.png"

dreambooth_model

string

Select a DreamBooth model

Default: "None"

steps

integer

Number of inference steps

Default: 25

guidance

number

Guidance scale

Default: 8.5

length

integer

Video length

Default: 16

width

integer

Width of the output video

height

integer

Height of the output video

seed

integer

Random seed

Default: -1

Run this model in Node.js with one line of code:

npx create-replicate --model=shimmercam/animatediff-v3

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";
import fs from "node:fs";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run shimmercam/animatediff-v3 using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "shimmercam/animatediff-v3:fd64261bc2a95198a2c7592ec501c2d75d8fded18b499399fd65e010adb49f2f",
  {
    input: {
      seed: -1,
      image: "https://raw.githubusercontent.com/guoyww/AnimateDiff/main/__assets__/demos/image/RealisticVision_firework.png",
      steps: 25,
      length: 16,
      prompt: "husky running in the snow",
      guidance: 8.5,
      negative_prompt: "worst quality, low quality, letterboxed",
      dreambooth_model: "None"
    }
  }
);

// To access the file URL:
console.log(output.url()); //=> "http://example.com"

// To write the file to disk:
fs.writeFile("my-image.png", output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run shimmercam/animatediff-v3 using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "shimmercam/animatediff-v3:fd64261bc2a95198a2c7592ec501c2d75d8fded18b499399fd65e010adb49f2f",
    input={
        "seed": -1,
        "image": "https://raw.githubusercontent.com/guoyww/AnimateDiff/main/__assets__/demos/image/RealisticVision_firework.png",
        "steps": 25,
        "length": 16,
        "prompt": "husky running in the snow",
        "guidance": 8.5,
        "negative_prompt": "worst quality, low quality, letterboxed",
        "dreambooth_model": "None"
    }
)

# To access the file URL:
print(output.url())
#=> "http://example.com"

# To write the file to disk:
with open("my-image.png", "wb") as file:
    file.write(output.read())

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run shimmercam/animatediff-v3 using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "shimmercam/animatediff-v3:fd64261bc2a95198a2c7592ec501c2d75d8fded18b499399fd65e010adb49f2f",
    "input": {
      "seed": -1,
      "image": "https://raw.githubusercontent.com/guoyww/AnimateDiff/main/__assets__/demos/image/RealisticVision_firework.png",
      "steps": 25,
      "length": 16,
      "prompt": "husky running in the snow",
      "guidance": 8.5,
      "negative_prompt": "worst quality, low quality, letterboxed",
      "dreambooth_model": "None"
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

{
  "completed_at": "2023-12-16T20:58:17.238586Z",
  "created_at": "2023-12-16T20:54:44.468540Z",
  "data_removed": false,
  "error": null,
  "id": "pf4uuybbagzq4z7w5p6tkec2ge",
  "input": {
    "seed": -1,
    "steps": 25,
    "length": 16,
    "prompt": "husky running in the snow",
    "guidance": 8.5,
    "negative_prompt": "worst quality, low quality, letterboxed",
    "dreambooth_model": "None"
  },
  "logs": "loading controlnet checkpoint from /AnimateDiff/models/SparseCtrl/v3_sd15_sparsectrl_rgb.ckpt ...\nload motion module from /AnimateDiff/models/Motion_Module/v3_sd15_mm.ckpt\nload domain lora from /AnimateDiff/models/Motion_Module/v3_sd15_adapter.ckpt\n  0%|          | 0/25 [00:00<?, ?it/s]\n  4%|▍         | 1/25 [00:02<00:49,  2.07s/it]\n  8%|▊         | 2/25 [00:03<00:40,  1.78s/it]\n 12%|█▏        | 3/25 [00:05<00:37,  1.68s/it]\n 16%|█▌        | 4/25 [00:06<00:34,  1.64s/it]\n 20%|██        | 5/25 [00:08<00:32,  1.61s/it]\n 24%|██▍       | 6/25 [00:09<00:30,  1.60s/it]\n 28%|██▊       | 7/25 [00:11<00:28,  1.59s/it]\n 32%|███▏      | 8/25 [00:13<00:26,  1.58s/it]\n 36%|███▌      | 9/25 [00:14<00:25,  1.58s/it]\n 40%|████      | 10/25 [00:16<00:23,  1.58s/it]\n 44%|████▍     | 11/25 [00:17<00:22,  1.58s/it]\n 48%|████▊     | 12/25 [00:19<00:20,  1.57s/it]\n 52%|█████▏    | 13/25 [00:20<00:18,  1.57s/it]\n 56%|█████▌    | 14/25 [00:22<00:17,  1.57s/it]\n 60%|██████    | 15/25 [00:24<00:15,  1.57s/it]\n 64%|██████▍   | 16/25 [00:25<00:14,  1.57s/it]\n 68%|██████▊   | 17/25 [00:27<00:12,  1.57s/it]\n 72%|███████▏  | 18/25 [00:28<00:11,  1.57s/it]\n 76%|███████▌  | 19/25 [00:30<00:09,  1.57s/it]\n 80%|████████  | 20/25 [00:31<00:07,  1.57s/it]\n 84%|████████▍ | 21/25 [00:33<00:06,  1.57s/it]\n 88%|████████▊ | 22/25 [00:35<00:04,  1.57s/it]\n 92%|█████████▏| 23/25 [00:36<00:03,  1.57s/it]\n 96%|█████████▌| 24/25 [00:38<00:01,  1.57s/it]\n100%|██████████| 25/25 [00:39<00:00,  1.57s/it]\n100%|██████████| 25/25 [00:39<00:00,  1.59s/it]\n  0%|          | 0/16 [00:00<?, ?it/s]\n 31%|███▏      | 5/16 [00:00<00:00, 44.00it/s]\n 62%|██████▎   | 10/16 [00:00<00:00, 23.75it/s]\n 81%|████████▏ | 13/16 [00:00<00:00, 21.43it/s]\n100%|██████████| 16/16 [00:00<00:00, 20.17it/s]\n100%|██████████| 16/16 [00:00<00:00, 22.03it/s]",
  "metrics": {
    "predict_time": 51.429076,
    "total_time": 212.770046
  },
  "output": "https://replicate.delivery/pbxt/NPJe0gaetTmb1Ee6rebJ6fFjnFfgAh1dCjp9UySp3UNOaBvgE/output.gif",
  "started_at": "2023-12-16T20:57:25.809510Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/pf4uuybbagzq4z7w5p6tkec2ge",
    "cancel": "https://api.replicate.com/v1/predictions/pf4uuybbagzq4z7w5p6tkec2ge/cancel"
  },
  "version": "dd87e0a6eed441ad5944d9b10a2020245889b6b85fe97bd9abeb76334d2c95ec"
}

Generated in

51.4 seconds

Tweak itReport View full prediction

loading controlnet checkpoint from /AnimateDiff/models/SparseCtrl/v3_sd15_sparsectrl_rgb.ckpt ...
load motion module from /AnimateDiff/models/Motion_Module/v3_sd15_mm.ckpt
load domain lora from /AnimateDiff/models/Motion_Module/v3_sd15_adapter.ckpt
  0%|          | 0/25 [00:00<?, ?it/s]
  4%|▍         | 1/25 [00:02<00:49,  2.07s/it]
  8%|▊         | 2/25 [00:03<00:40,  1.78s/it]
 12%|█▏        | 3/25 [00:05<00:37,  1.68s/it]
 16%|█▌        | 4/25 [00:06<00:34,  1.64s/it]
 20%|██        | 5/25 [00:08<00:32,  1.61s/it]
 24%|██▍       | 6/25 [00:09<00:30,  1.60s/it]
 28%|██▊       | 7/25 [00:11<00:28,  1.59s/it]
 32%|███▏      | 8/25 [00:13<00:26,  1.58s/it]
 36%|███▌      | 9/25 [00:14<00:25,  1.58s/it]
 40%|████      | 10/25 [00:16<00:23,  1.58s/it]
 44%|████▍     | 11/25 [00:17<00:22,  1.58s/it]
 48%|████▊     | 12/25 [00:19<00:20,  1.57s/it]
 52%|█████▏    | 13/25 [00:20<00:18,  1.57s/it]
 56%|█████▌    | 14/25 [00:22<00:17,  1.57s/it]
 60%|██████    | 15/25 [00:24<00:15,  1.57s/it]
 64%|██████▍   | 16/25 [00:25<00:14,  1.57s/it]
 68%|██████▊   | 17/25 [00:27<00:12,  1.57s/it]
 72%|███████▏  | 18/25 [00:28<00:11,  1.57s/it]
 76%|███████▌  | 19/25 [00:30<00:09,  1.57s/it]
 80%|████████  | 20/25 [00:31<00:07,  1.57s/it]
 84%|████████▍ | 21/25 [00:33<00:06,  1.57s/it]
 88%|████████▊ | 22/25 [00:35<00:04,  1.57s/it]
 92%|█████████▏| 23/25 [00:36<00:03,  1.57s/it]
 96%|█████████▌| 24/25 [00:38<00:01,  1.57s/it]
100%|██████████| 25/25 [00:39<00:00,  1.57s/it]
100%|██████████| 25/25 [00:39<00:00,  1.59s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|███▏      | 5/16 [00:00<00:00, 44.00it/s]
 62%|██████▎   | 10/16 [00:00<00:00, 23.75it/s]
 81%|████████▏ | 13/16 [00:00<00:00, 21.43it/s]
100%|██████████| 16/16 [00:00<00:00, 20.17it/s]
100%|██████████| 16/16 [00:00<00:00, 22.03it/s]

This output was created using a different version of the model, shimmercam/animatediff-v3:dd87e0a6.

Examples

View more examples

Run time and cost

This model costs approximately $0.068 to run on Replicate, or 14 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia A100 (80GB) GPU hardware. Predictions typically complete within 49 seconds. The predict time for this model varies significantly based on the inputs.

Readme

Compare, edit, and share this model at https://shimmer.cam

@article{guo2023animatediff, title={AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning}, author={Guo, Yuwei and Yang, Ceyuan and Rao, Anyi and Wang, Yaohui and Qiao, Yu and Lin, Dahua and Dai, Bo}, journal={arXiv preprint arXiv:2307.04725}, year={2023} }

@article{guo2023sparsectrl, title={SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models}, author={Guo, Yuwei and Yang, Ceyuan and Rao, Anyi and Agrawala, Maneesh and Lin, Dahua and Dai, Bo}, journal={arXiv preprint arXiv:2311.16933}, year={2023} }