lucataco/magic-animate – Run with an API on Replicate

lucataco / magic-animate

MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model

Cold

Public
55.9K runs
A100 (80GB)
GitHub
Paper
License

Run with an API

Playground API Examples README Versions

Input

Run this model in Node.js with one line of code:

npx create-replicate --model=lucataco/magic-animate

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run lucataco/magic-animate using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "lucataco/magic-animate:e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714",
  {
    input: {
      image: "https://replicate.delivery/pbxt/JzgM3WOPIopVwRj4Pe7KPxC5DYYC3GUZx4GhjRtCOXwOoBoT/dalle8.jpeg",
      video: "https://replicate.delivery/pbxt/JzgM380MyPyDjbgkCYGGPg1g2NWiV50W4PmD52JPIsjh9R9G/running.mp4",
      guidance_scale: 7.5,
      num_inference_steps: 25
    }
  }
);

// To access the file URL:
console.log(output.url()); //=> "http://example.com"

// To write the file to disk:
fs.writeFile("my-image.png", output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run lucataco/magic-animate using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "lucataco/magic-animate:e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714",
    input={
        "image": "https://replicate.delivery/pbxt/JzgM3WOPIopVwRj4Pe7KPxC5DYYC3GUZx4GhjRtCOXwOoBoT/dalle8.jpeg",
        "video": "https://replicate.delivery/pbxt/JzgM380MyPyDjbgkCYGGPg1g2NWiV50W4PmD52JPIsjh9R9G/running.mp4",
        "guidance_scale": 7.5,
        "num_inference_steps": 25
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run lucataco/magic-animate using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714",
    "input": {
      "image": "https://replicate.delivery/pbxt/JzgM3WOPIopVwRj4Pe7KPxC5DYYC3GUZx4GhjRtCOXwOoBoT/dalle8.jpeg",
      "video": "https://replicate.delivery/pbxt/JzgM380MyPyDjbgkCYGGPg1g2NWiV50W4PmD52JPIsjh9R9G/running.mp4",
      "guidance_scale": 7.5,
      "num_inference_steps": 25
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

{
  "completed_at": "2023-12-05T04:47:30.068057Z",
  "created_at": "2023-12-05T04:46:02.772954Z",
  "data_removed": false,
  "error": null,
  "id": "ja5opnrbi3pcoa5fzxxxxercje",
  "input": {
    "image": "https://replicate.delivery/pbxt/JzgM3WOPIopVwRj4Pe7KPxC5DYYC3GUZx4GhjRtCOXwOoBoT/dalle8.jpeg",
    "video": "https://replicate.delivery/pbxt/JzgM380MyPyDjbgkCYGGPg1g2NWiV50W4PmD52JPIsjh9R9G/running.mp4",
    "guidance_scale": 7.5,
    "num_inference_steps": 25
  },
  "logs": "Using seed: 21244\n/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:43: FutureWarning: Importing `DiffusionPipeline` or `ImagePipelineOutput` from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead.\nfrom diffusers.pipeline_utils import DiffusionPipeline\nloaded temporal unet's pretrained weights from /src/pretrained_models/stable-diffusion-v1-5/unet ...\n### missing keys: 560;\n### unexpected keys: 0;\n### Temporal Module Parameters: 417.1376 M\nThe config attributes {'addition_embed_type': None, 'addition_embed_type_num_heads': 64, 'addition_time_embed_dim': None, 'conditioning_channels': 3, 'encoder_hid_dim': None, 'encoder_hid_dim_type': None, 'global_pool_conditions': False, 'num_attention_heads': None, 'transformer_layers_per_block': 1} were passed to ControlNetModel, but are not expected and will be ignored. Please verify your config.json configuration file.\nIt is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.\nIt is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.\nIt is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.\nIt is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.\n/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:103: FutureWarning: The configuration file of this scheduler: DDIMScheduler {\n\"_class_name\": \"DDIMScheduler\",\n\"_diffusers_version\": \"0.21.4\",\n\"beta_end\": 0.012,\n\"beta_schedule\": \"linear\",\n\"beta_start\": 0.00085,\n\"clip_sample\": true,\n\"clip_sample_range\": 1.0,\n\"dynamic_thresholding_ratio\": 0.995,\n\"num_train_timesteps\": 1000,\n\"prediction_type\": \"epsilon\",\n\"rescale_betas_zero_snr\": false,\n\"sample_max_value\": 1.0,\n\"set_alpha_to_one\": true,\n\"steps_offset\": 0,\n\"thresholding\": false,\n\"timestep_spacing\": \"leading\",\n\"trained_betas\": null\n}\nis outdated. `steps_offset` should be set to 1 instead of 0. Please make sure to update the config accordingly as leaving `steps_offset` might led to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file\ndeprecate(\"steps_offset!=1\", \"1.0.0\", deprecation_message, standard_warn=False)\n/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:116: FutureWarning: The configuration file of this scheduler: DDIMScheduler {\n\"_class_name\": \"DDIMScheduler\",\n\"_diffusers_version\": \"0.21.4\",\n\"beta_end\": 0.012,\n\"beta_schedule\": \"linear\",\n\"beta_start\": 0.00085,\n\"clip_sample\": true,\n\"clip_sample_range\": 1.0,\n\"dynamic_thresholding_ratio\": 0.995,\n\"num_train_timesteps\": 1000,\n\"prediction_type\": \"epsilon\",\n\"rescale_betas_zero_snr\": false,\n\"sample_max_value\": 1.0,\n\"set_alpha_to_one\": true,\n\"steps_offset\": 1,\n\"thresholding\": false,\n\"timestep_spacing\": \"leading\",\n\"trained_betas\": null\n}\nhas not set the configuration `clip_sample`. `clip_sample` should be set to False in the configuration file. Please make sure to update the config accordingly as not setting `clip_sample` in the config might lead to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file\ndeprecate(\"clip_sample not set\", \"1.0.0\", deprecation_message, standard_warn=False)\ncurrent seed: 21244\n  0%|          | 0/1 [00:00<?, ?it/s]/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:624: FutureWarning: Accessing config attribute `in_channels` directly via 'UNet3DConditionModel' object attribute is deprecated. Please access 'in_channels' over 'UNet3DConditionModel's config object instead, e.g. 'unet.config.in_channels'.\nnum_channels_latents = self.unet.in_channels\n  0%|          | 0/25 [00:00<?, ?it/s]\u001b[A\n  4%|▍         | 1/25 [00:02<00:52,  2.21s/it]\u001b[A\n  8%|▊         | 2/25 [00:04<00:48,  2.09s/it]\u001b[A\n 12%|█▏        | 3/25 [00:06<00:45,  2.05s/it]\u001b[A\n 16%|█▌        | 4/25 [00:08<00:42,  2.05s/it]\u001b[A\n 20%|██        | 5/25 [00:10<00:40,  2.04s/it]\u001b[A\n 24%|██▍       | 6/25 [00:12<00:38,  2.03s/it]\u001b[A\n 28%|██▊       | 7/25 [00:14<00:36,  2.02s/it]\u001b[A\n 32%|███▏      | 8/25 [00:16<00:34,  2.02s/it]\u001b[A\n 36%|███▌      | 9/25 [00:18<00:32,  2.02s/it]\u001b[A\n 40%|████      | 10/25 [00:20<00:30,  2.02s/it]\u001b[A\n 44%|████▍     | 11/25 [00:22<00:28,  2.02s/it]\u001b[A\n 48%|████▊     | 12/25 [00:24<00:26,  2.01s/it]\u001b[A\n 52%|█████▏    | 13/25 [00:26<00:24,  2.02s/it]\u001b[A\n 56%|█████▌    | 14/25 [00:28<00:22,  2.01s/it]\u001b[A\n 60%|██████    | 15/25 [00:30<00:20,  2.01s/it]\u001b[A\n 64%|██████▍   | 16/25 [00:32<00:18,  2.02s/it]\u001b[A\n 68%|██████▊   | 17/25 [00:34<00:16,  2.02s/it]\u001b[A\n 72%|███████▏  | 18/25 [00:36<00:14,  2.02s/it]\u001b[A\n 76%|███████▌  | 19/25 [00:38<00:12,  2.02s/it]\u001b[A\n 80%|████████  | 20/25 [00:40<00:10,  2.02s/it]\u001b[A\n 84%|████████▍ | 21/25 [00:42<00:08,  2.02s/it]\u001b[A\n 88%|████████▊ | 22/25 [00:44<00:06,  2.02s/it]\u001b[A\n 92%|█████████▏| 23/25 [00:46<00:04,  2.02s/it]\u001b[A\n 96%|█████████▌| 24/25 [00:48<00:02,  2.02s/it]\u001b[A\n100%|██████████| 25/25 [00:50<00:00,  2.01s/it]\u001b[A\n100%|██████████| 25/25 [00:50<00:00,  2.03s/it]\n  0%|          | 0/48 [00:00<?, ?it/s]\u001b[A\n 12%|█▎        | 6/48 [00:00<00:00, 49.40it/s]\u001b[A\n 23%|██▎       | 11/48 [00:00<00:01, 33.66it/s]\u001b[A\n 31%|███▏      | 15/48 [00:00<00:01, 30.52it/s]\u001b[A\n 40%|███▉      | 19/48 [00:00<00:01, 28.99it/s]\u001b[A\n 46%|████▌     | 22/48 [00:00<00:00, 28.27it/s]\u001b[A\n 52%|█████▏    | 25/48 [00:00<00:00, 27.77it/s]\u001b[A\n 58%|█████▊    | 28/48 [00:00<00:00, 27.40it/s]\u001b[A\n 65%|██████▍   | 31/48 [00:01<00:00, 27.15it/s]\u001b[A\n 71%|███████   | 34/48 [00:01<00:00, 26.97it/s]\u001b[A\n 77%|███████▋  | 37/48 [00:01<00:00, 26.84it/s]\u001b[A\n 83%|████████▎ | 40/48 [00:01<00:00, 26.75it/s]\u001b[A\n 90%|████████▉ | 43/48 [00:01<00:00, 26.69it/s]\u001b[A\n 96%|█████████▌| 46/48 [00:01<00:00, 26.65it/s]\u001b[A\n100%|██████████| 48/48 [00:01<00:00, 28.17it/s]\nIMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (1544, 516) to (1552, 528) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility).\n[swscaler @ 0x6185740] Warning: data is not aligned! This can lead to a speed loss\n100%|██████████| 1/1 [00:55<00:00, 55.93s/it]\n100%|██████████| 1/1 [00:55<00:00, 55.93s/it]\n/src/magic-animate/samples/config-2023-12-05T04-46-07/videos/tmpaf72liowdalle8_tmp868xbl66running.mp4",
  "metrics": {
    "predict_time": 87.278525,
    "total_time": 87.295103
  },
  "output": "https://replicate.delivery/pbxt/HwMGXs27BUIRLNacoAcS6P93YUL6r1lNfaiOfhvhFUxh1FfjA/tmpaf72liowdalle8_tmp868xbl66running.mp4",
  "started_at": "2023-12-05T04:46:02.789532Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/ja5opnrbi3pcoa5fzxxxxercje",
    "cancel": "https://api.replicate.com/v1/predictions/ja5opnrbi3pcoa5fzxxxxercje/cancel"
  },
  "version": "e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714"
}

Generated in

87.3 seconds

Tweak itReport

Using seed: 21244
/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:43: FutureWarning: Importing `DiffusionPipeline` or `ImagePipelineOutput` from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead.
from diffusers.pipeline_utils import DiffusionPipeline
loaded temporal unet's pretrained weights from /src/pretrained_models/stable-diffusion-v1-5/unet ...
### missing keys: 560;
### unexpected keys: 0;
### Temporal Module Parameters: 417.1376 M
The config attributes {'addition_embed_type': None, 'addition_embed_type_num_heads': 64, 'addition_time_embed_dim': None, 'conditioning_channels': 3, 'encoder_hid_dim': None, 'encoder_hid_dim_type': None, 'global_pool_conditions': False, 'num_attention_heads': None, 'transformer_layers_per_block': 1} were passed to ControlNetModel, but are not expected and will be ignored. Please verify your config.json configuration file.
It is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.
It is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.
It is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.
It is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.
/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:103: FutureWarning: The configuration file of this scheduler: DDIMScheduler {
"_class_name": "DDIMScheduler",
"_diffusers_version": "0.21.4",
"beta_end": 0.012,
"beta_schedule": "linear",
"beta_start": 0.00085,
"clip_sample": true,
"clip_sample_range": 1.0,
"dynamic_thresholding_ratio": 0.995,
"num_train_timesteps": 1000,
"prediction_type": "epsilon",
"rescale_betas_zero_snr": false,
"sample_max_value": 1.0,
"set_alpha_to_one": true,
"steps_offset": 0,
"thresholding": false,
"timestep_spacing": "leading",
"trained_betas": null
}
is outdated. `steps_offset` should be set to 1 instead of 0. Please make sure to update the config accordingly as leaving `steps_offset` might led to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file
deprecate("steps_offset!=1", "1.0.0", deprecation_message, standard_warn=False)
/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:116: FutureWarning: The configuration file of this scheduler: DDIMScheduler {
"_class_name": "DDIMScheduler",
"_diffusers_version": "0.21.4",
"beta_end": 0.012,
"beta_schedule": "linear",
"beta_start": 0.00085,
"clip_sample": true,
"clip_sample_range": 1.0,
"dynamic_thresholding_ratio": 0.995,
"num_train_timesteps": 1000,
"prediction_type": "epsilon",
"rescale_betas_zero_snr": false,
"sample_max_value": 1.0,
"set_alpha_to_one": true,
"steps_offset": 1,
"thresholding": false,
"timestep_spacing": "leading",
"trained_betas": null
}
has not set the configuration `clip_sample`. `clip_sample` should be set to False in the configuration file. Please make sure to update the config accordingly as not setting `clip_sample` in the config might lead to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file
deprecate("clip_sample not set", "1.0.0", deprecation_message, standard_warn=False)
current seed: 21244
  0%|          | 0/1 [00:00<?, ?it/s]/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:624: FutureWarning: Accessing config attribute `in_channels` directly via 'UNet3DConditionModel' object attribute is deprecated. Please access 'in_channels' over 'UNet3DConditionModel's config object instead, e.g. 'unet.config.in_channels'.
num_channels_latents = self.unet.in_channels
  0%|          | 0/25 [00:00<?, ?it/s]
  4%|▍         | 1/25 [00:02<00:52,  2.21s/it]
  8%|▊         | 2/25 [00:04<00:48,  2.09s/it]
 12%|█▏        | 3/25 [00:06<00:45,  2.05s/it]
 16%|█▌        | 4/25 [00:08<00:42,  2.05s/it]
 20%|██        | 5/25 [00:10<00:40,  2.04s/it]
 24%|██▍       | 6/25 [00:12<00:38,  2.03s/it]
 28%|██▊       | 7/25 [00:14<00:36,  2.02s/it]
 32%|███▏      | 8/25 [00:16<00:34,  2.02s/it]
 36%|███▌      | 9/25 [00:18<00:32,  2.02s/it]
 40%|████      | 10/25 [00:20<00:30,  2.02s/it]
 44%|████▍     | 11/25 [00:22<00:28,  2.02s/it]
 48%|████▊     | 12/25 [00:24<00:26,  2.01s/it]
 52%|█████▏    | 13/25 [00:26<00:24,  2.02s/it]
 56%|█████▌    | 14/25 [00:28<00:22,  2.01s/it]
 60%|██████    | 15/25 [00:30<00:20,  2.01s/it]
 64%|██████▍   | 16/25 [00:32<00:18,  2.02s/it]
 68%|██████▊   | 17/25 [00:34<00:16,  2.02s/it]
 72%|███████▏  | 18/25 [00:36<00:14,  2.02s/it]
 76%|███████▌  | 19/25 [00:38<00:12,  2.02s/it]
 80%|████████  | 20/25 [00:40<00:10,  2.02s/it]
 84%|████████▍ | 21/25 [00:42<00:08,  2.02s/it]
 88%|████████▊ | 22/25 [00:44<00:06,  2.02s/it]
 92%|█████████▏| 23/25 [00:46<00:04,  2.02s/it]
 96%|█████████▌| 24/25 [00:48<00:02,  2.02s/it]
100%|██████████| 25/25 [00:50<00:00,  2.01s/it]
100%|██████████| 25/25 [00:50<00:00,  2.03s/it]
  0%|          | 0/48 [00:00<?, ?it/s]
 12%|█▎        | 6/48 [00:00<00:00, 49.40it/s]
 23%|██▎       | 11/48 [00:00<00:01, 33.66it/s]
 31%|███▏      | 15/48 [00:00<00:01, 30.52it/s]
 40%|███▉      | 19/48 [00:00<00:01, 28.99it/s]
 46%|████▌     | 22/48 [00:00<00:00, 28.27it/s]
 52%|█████▏    | 25/48 [00:00<00:00, 27.77it/s]
 58%|█████▊    | 28/48 [00:00<00:00, 27.40it/s]
 65%|██████▍   | 31/48 [00:01<00:00, 27.15it/s]
 71%|███████   | 34/48 [00:01<00:00, 26.97it/s]
 77%|███████▋  | 37/48 [00:01<00:00, 26.84it/s]
 83%|████████▎ | 40/48 [00:01<00:00, 26.75it/s]
 90%|████████▉ | 43/48 [00:01<00:00, 26.69it/s]
 96%|█████████▌| 46/48 [00:01<00:00, 26.65it/s]
100%|██████████| 48/48 [00:01<00:00, 28.17it/s]
IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (1544, 516) to (1552, 528) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility).
[swscaler @ 0x6185740] Warning: data is not aligned! This can lead to a speed loss
100%|██████████| 1/1 [00:55<00:00, 55.93s/it]
100%|██████████| 1/1 [00:55<00:00, 55.93s/it]
/src/magic-animate/samples/config-2023-12-05T04-46-07/videos/tmpaf72liowdalle8_tmp868xbl66running.mp4

Examples

View more examples

Run time and cost

This model costs approximately $0.34 to run on Replicate, or 2 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia A100 (80GB) GPU hardware. Predictions typically complete within 5 minutes. The predict time for this model varies significantly based on the inputs.

Readme

Implementation of MagicAnimate

Note: For now, please use JPG images (not PNG)

@inproceedings{xu2023magicanimate,
    author    = {Xu, Zhongcong and Zhang, Jianfeng and Liew, Jun Hao and Yan, Hanshu and Liu, Jia-Wei and Zhang, Chenxu and Feng, Jiashi and Shou, Mike Zheng},
    title     = {MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model},
    booktitle = {arXiv},
    year      = {2023}
}