lucataco / magic-animate
MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model
Prediction
lucataco/magic-animate:e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714IDja5opnrbi3pcoa5fzxxxxercjeStatusSucceededSourceWebHardwareA100 (40GB)Total durationCreatedInput
- image
- video
- guidance_scale
- 7.5
- num_inference_steps
- 25
{ "image": "https://replicate.delivery/pbxt/JzgM3WOPIopVwRj4Pe7KPxC5DYYC3GUZx4GhjRtCOXwOoBoT/dalle8.jpeg", "video": "https://replicate.delivery/pbxt/JzgM380MyPyDjbgkCYGGPg1g2NWiV50W4PmD52JPIsjh9R9G/running.mp4", "guidance_scale": 7.5, "num_inference_steps": 25 }Install Replicate’s Node.js client library:npm install replicateImport and set up the client:import Replicate from "replicate"; const replicate = new Replicate({ auth: process.env.REPLICATE_API_TOKEN, });Run lucataco/magic-animate using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
const output = await replicate.run( "lucataco/magic-animate:e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714", { input: { image: "https://replicate.delivery/pbxt/JzgM3WOPIopVwRj4Pe7KPxC5DYYC3GUZx4GhjRtCOXwOoBoT/dalle8.jpeg", video: "https://replicate.delivery/pbxt/JzgM380MyPyDjbgkCYGGPg1g2NWiV50W4PmD52JPIsjh9R9G/running.mp4", guidance_scale: 7.5, num_inference_steps: 25 } } ); // To access the file URL: console.log(output.url()); //=> "http://example.com" // To write the file to disk: fs.writeFile("my-image.png", output);To learn more, take a look at the guide on getting started with Node.js.
Install Replicate’s Python client library:pip install replicateImport the client:import replicateRun lucataco/magic-animate using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
output = replicate.run( "lucataco/magic-animate:e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714", input={ "image": "https://replicate.delivery/pbxt/JzgM3WOPIopVwRj4Pe7KPxC5DYYC3GUZx4GhjRtCOXwOoBoT/dalle8.jpeg", "video": "https://replicate.delivery/pbxt/JzgM380MyPyDjbgkCYGGPg1g2NWiV50W4PmD52JPIsjh9R9G/running.mp4", "guidance_scale": 7.5, "num_inference_steps": 25 } ) print(output)To learn more, take a look at the guide on getting started with Python.
Run lucataco/magic-animate using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
curl -s -X POST \ -H "Authorization: Bearer $REPLICATE_API_TOKEN" \ -H "Content-Type: application/json" \ -H "Prefer: wait" \ -d $'{ "version": "lucataco/magic-animate:e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714", "input": { "image": "https://replicate.delivery/pbxt/JzgM3WOPIopVwRj4Pe7KPxC5DYYC3GUZx4GhjRtCOXwOoBoT/dalle8.jpeg", "video": "https://replicate.delivery/pbxt/JzgM380MyPyDjbgkCYGGPg1g2NWiV50W4PmD52JPIsjh9R9G/running.mp4", "guidance_scale": 7.5, "num_inference_steps": 25 } }' \ https://api.replicate.com/v1/predictionsTo learn more, take a look at Replicate’s HTTP API reference docs.
Output
{ "completed_at": "2023-12-05T04:47:30.068057Z", "created_at": "2023-12-05T04:46:02.772954Z", "data_removed": false, "error": null, "id": "ja5opnrbi3pcoa5fzxxxxercje", "input": { "image": "https://replicate.delivery/pbxt/JzgM3WOPIopVwRj4Pe7KPxC5DYYC3GUZx4GhjRtCOXwOoBoT/dalle8.jpeg", "video": "https://replicate.delivery/pbxt/JzgM380MyPyDjbgkCYGGPg1g2NWiV50W4PmD52JPIsjh9R9G/running.mp4", "guidance_scale": 7.5, "num_inference_steps": 25 }, "logs": "Using seed: 21244\n/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:43: FutureWarning: Importing `DiffusionPipeline` or `ImagePipelineOutput` from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead.\nfrom diffusers.pipeline_utils import DiffusionPipeline\nloaded temporal unet's pretrained weights from /src/pretrained_models/stable-diffusion-v1-5/unet ...\n### missing keys: 560;\n### unexpected keys: 0;\n### Temporal Module Parameters: 417.1376 M\nThe config attributes {'addition_embed_type': None, 'addition_embed_type_num_heads': 64, 'addition_time_embed_dim': None, 'conditioning_channels': 3, 'encoder_hid_dim': None, 'encoder_hid_dim_type': None, 'global_pool_conditions': False, 'num_attention_heads': None, 'transformer_layers_per_block': 1} were passed to ControlNetModel, but are not expected and will be ignored. Please verify your config.json configuration file.\nIt is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.\nIt is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.\nIt is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.\nIt is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.\n/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:103: FutureWarning: The configuration file of this scheduler: DDIMScheduler {\n\"_class_name\": \"DDIMScheduler\",\n\"_diffusers_version\": \"0.21.4\",\n\"beta_end\": 0.012,\n\"beta_schedule\": \"linear\",\n\"beta_start\": 0.00085,\n\"clip_sample\": true,\n\"clip_sample_range\": 1.0,\n\"dynamic_thresholding_ratio\": 0.995,\n\"num_train_timesteps\": 1000,\n\"prediction_type\": \"epsilon\",\n\"rescale_betas_zero_snr\": false,\n\"sample_max_value\": 1.0,\n\"set_alpha_to_one\": true,\n\"steps_offset\": 0,\n\"thresholding\": false,\n\"timestep_spacing\": \"leading\",\n\"trained_betas\": null\n}\nis outdated. `steps_offset` should be set to 1 instead of 0. Please make sure to update the config accordingly as leaving `steps_offset` might led to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file\ndeprecate(\"steps_offset!=1\", \"1.0.0\", deprecation_message, standard_warn=False)\n/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:116: FutureWarning: The configuration file of this scheduler: DDIMScheduler {\n\"_class_name\": \"DDIMScheduler\",\n\"_diffusers_version\": \"0.21.4\",\n\"beta_end\": 0.012,\n\"beta_schedule\": \"linear\",\n\"beta_start\": 0.00085,\n\"clip_sample\": true,\n\"clip_sample_range\": 1.0,\n\"dynamic_thresholding_ratio\": 0.995,\n\"num_train_timesteps\": 1000,\n\"prediction_type\": \"epsilon\",\n\"rescale_betas_zero_snr\": false,\n\"sample_max_value\": 1.0,\n\"set_alpha_to_one\": true,\n\"steps_offset\": 1,\n\"thresholding\": false,\n\"timestep_spacing\": \"leading\",\n\"trained_betas\": null\n}\nhas not set the configuration `clip_sample`. `clip_sample` should be set to False in the configuration file. Please make sure to update the config accordingly as not setting `clip_sample` in the config might lead to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file\ndeprecate(\"clip_sample not set\", \"1.0.0\", deprecation_message, standard_warn=False)\ncurrent seed: 21244\n 0%| | 0/1 [00:00<?, ?it/s]/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:624: FutureWarning: Accessing config attribute `in_channels` directly via 'UNet3DConditionModel' object attribute is deprecated. Please access 'in_channels' over 'UNet3DConditionModel's config object instead, e.g. 'unet.config.in_channels'.\nnum_channels_latents = self.unet.in_channels\n 0%| | 0/25 [00:00<?, ?it/s]\u001b[A\n 4%|▍ | 1/25 [00:02<00:52, 2.21s/it]\u001b[A\n 8%|▊ | 2/25 [00:04<00:48, 2.09s/it]\u001b[A\n 12%|█▏ | 3/25 [00:06<00:45, 2.05s/it]\u001b[A\n 16%|█▌ | 4/25 [00:08<00:42, 2.05s/it]\u001b[A\n 20%|██ | 5/25 [00:10<00:40, 2.04s/it]\u001b[A\n 24%|██▍ | 6/25 [00:12<00:38, 2.03s/it]\u001b[A\n 28%|██▊ | 7/25 [00:14<00:36, 2.02s/it]\u001b[A\n 32%|███▏ | 8/25 [00:16<00:34, 2.02s/it]\u001b[A\n 36%|███▌ | 9/25 [00:18<00:32, 2.02s/it]\u001b[A\n 40%|████ | 10/25 [00:20<00:30, 2.02s/it]\u001b[A\n 44%|████▍ | 11/25 [00:22<00:28, 2.02s/it]\u001b[A\n 48%|████▊ | 12/25 [00:24<00:26, 2.01s/it]\u001b[A\n 52%|█████▏ | 13/25 [00:26<00:24, 2.02s/it]\u001b[A\n 56%|█████▌ | 14/25 [00:28<00:22, 2.01s/it]\u001b[A\n 60%|██████ | 15/25 [00:30<00:20, 2.01s/it]\u001b[A\n 64%|██████▍ | 16/25 [00:32<00:18, 2.02s/it]\u001b[A\n 68%|██████▊ | 17/25 [00:34<00:16, 2.02s/it]\u001b[A\n 72%|███████▏ | 18/25 [00:36<00:14, 2.02s/it]\u001b[A\n 76%|███████▌ | 19/25 [00:38<00:12, 2.02s/it]\u001b[A\n 80%|████████ | 20/25 [00:40<00:10, 2.02s/it]\u001b[A\n 84%|████████▍ | 21/25 [00:42<00:08, 2.02s/it]\u001b[A\n 88%|████████▊ | 22/25 [00:44<00:06, 2.02s/it]\u001b[A\n 92%|█████████▏| 23/25 [00:46<00:04, 2.02s/it]\u001b[A\n 96%|█████████▌| 24/25 [00:48<00:02, 2.02s/it]\u001b[A\n100%|██████████| 25/25 [00:50<00:00, 2.01s/it]\u001b[A\n100%|██████████| 25/25 [00:50<00:00, 2.03s/it]\n 0%| | 0/48 [00:00<?, ?it/s]\u001b[A\n 12%|█▎ | 6/48 [00:00<00:00, 49.40it/s]\u001b[A\n 23%|██▎ | 11/48 [00:00<00:01, 33.66it/s]\u001b[A\n 31%|███▏ | 15/48 [00:00<00:01, 30.52it/s]\u001b[A\n 40%|███▉ | 19/48 [00:00<00:01, 28.99it/s]\u001b[A\n 46%|████▌ | 22/48 [00:00<00:00, 28.27it/s]\u001b[A\n 52%|█████▏ | 25/48 [00:00<00:00, 27.77it/s]\u001b[A\n 58%|█████▊ | 28/48 [00:00<00:00, 27.40it/s]\u001b[A\n 65%|██████▍ | 31/48 [00:01<00:00, 27.15it/s]\u001b[A\n 71%|███████ | 34/48 [00:01<00:00, 26.97it/s]\u001b[A\n 77%|███████▋ | 37/48 [00:01<00:00, 26.84it/s]\u001b[A\n 83%|████████▎ | 40/48 [00:01<00:00, 26.75it/s]\u001b[A\n 90%|████████▉ | 43/48 [00:01<00:00, 26.69it/s]\u001b[A\n 96%|█████████▌| 46/48 [00:01<00:00, 26.65it/s]\u001b[A\n100%|██████████| 48/48 [00:01<00:00, 28.17it/s]\nIMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (1544, 516) to (1552, 528) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility).\n[swscaler @ 0x6185740] Warning: data is not aligned! This can lead to a speed loss\n100%|██████████| 1/1 [00:55<00:00, 55.93s/it]\n100%|██████████| 1/1 [00:55<00:00, 55.93s/it]\n/src/magic-animate/samples/config-2023-12-05T04-46-07/videos/tmpaf72liowdalle8_tmp868xbl66running.mp4", "metrics": { "predict_time": 87.278525, "total_time": 87.295103 }, "output": "https://replicate.delivery/pbxt/HwMGXs27BUIRLNacoAcS6P93YUL6r1lNfaiOfhvhFUxh1FfjA/tmpaf72liowdalle8_tmp868xbl66running.mp4", "started_at": "2023-12-05T04:46:02.789532Z", "status": "succeeded", "urls": { "get": "https://api.replicate.com/v1/predictions/ja5opnrbi3pcoa5fzxxxxercje", "cancel": "https://api.replicate.com/v1/predictions/ja5opnrbi3pcoa5fzxxxxercje/cancel" }, "version": "e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714" }Generated inUsing seed: 21244 /src/magic-animate/magicanimate/pipelines/pipeline_animation.py:43: FutureWarning: Importing `DiffusionPipeline` or `ImagePipelineOutput` from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead. from diffusers.pipeline_utils import DiffusionPipeline loaded temporal unet's pretrained weights from /src/pretrained_models/stable-diffusion-v1-5/unet ... ### missing keys: 560; ### unexpected keys: 0; ### Temporal Module Parameters: 417.1376 M The config attributes {'addition_embed_type': None, 'addition_embed_type_num_heads': 64, 'addition_time_embed_dim': None, 'conditioning_channels': 3, 'encoder_hid_dim': None, 'encoder_hid_dim_type': None, 'global_pool_conditions': False, 'num_attention_heads': None, 'transformer_layers_per_block': 1} were passed to ControlNetModel, but are not expected and will be ignored. Please verify your config.json configuration file. It is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8. It is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8. It is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8. It is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8. /src/magic-animate/magicanimate/pipelines/pipeline_animation.py:103: FutureWarning: The configuration file of this scheduler: DDIMScheduler { "_class_name": "DDIMScheduler", "_diffusers_version": "0.21.4", "beta_end": 0.012, "beta_schedule": "linear", "beta_start": 0.00085, "clip_sample": true, "clip_sample_range": 1.0, "dynamic_thresholding_ratio": 0.995, "num_train_timesteps": 1000, "prediction_type": "epsilon", "rescale_betas_zero_snr": false, "sample_max_value": 1.0, "set_alpha_to_one": true, "steps_offset": 0, "thresholding": false, "timestep_spacing": "leading", "trained_betas": null } is outdated. `steps_offset` should be set to 1 instead of 0. Please make sure to update the config accordingly as leaving `steps_offset` might led to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file deprecate("steps_offset!=1", "1.0.0", deprecation_message, standard_warn=False) /src/magic-animate/magicanimate/pipelines/pipeline_animation.py:116: FutureWarning: The configuration file of this scheduler: DDIMScheduler { "_class_name": "DDIMScheduler", "_diffusers_version": "0.21.4", "beta_end": 0.012, "beta_schedule": "linear", "beta_start": 0.00085, "clip_sample": true, "clip_sample_range": 1.0, "dynamic_thresholding_ratio": 0.995, "num_train_timesteps": 1000, "prediction_type": "epsilon", "rescale_betas_zero_snr": false, "sample_max_value": 1.0, "set_alpha_to_one": true, "steps_offset": 1, "thresholding": false, "timestep_spacing": "leading", "trained_betas": null } has not set the configuration `clip_sample`. `clip_sample` should be set to False in the configuration file. Please make sure to update the config accordingly as not setting `clip_sample` in the config might lead to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file deprecate("clip_sample not set", "1.0.0", deprecation_message, standard_warn=False) current seed: 21244 0%| | 0/1 [00:00<?, ?it/s]/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:624: FutureWarning: Accessing config attribute `in_channels` directly via 'UNet3DConditionModel' object attribute is deprecated. Please access 'in_channels' over 'UNet3DConditionModel's config object instead, e.g. 'unet.config.in_channels'. num_channels_latents = self.unet.in_channels 0%| | 0/25 [00:00<?, ?it/s] 4%|▍ | 1/25 [00:02<00:52, 2.21s/it] 8%|▊ | 2/25 [00:04<00:48, 2.09s/it] 12%|█▏ | 3/25 [00:06<00:45, 2.05s/it] 16%|█▌ | 4/25 [00:08<00:42, 2.05s/it] 20%|██ | 5/25 [00:10<00:40, 2.04s/it] 24%|██▍ | 6/25 [00:12<00:38, 2.03s/it] 28%|██▊ | 7/25 [00:14<00:36, 2.02s/it] 32%|███▏ | 8/25 [00:16<00:34, 2.02s/it] 36%|███▌ | 9/25 [00:18<00:32, 2.02s/it] 40%|████ | 10/25 [00:20<00:30, 2.02s/it] 44%|████▍ | 11/25 [00:22<00:28, 2.02s/it] 48%|████▊ | 12/25 [00:24<00:26, 2.01s/it] 52%|█████▏ | 13/25 [00:26<00:24, 2.02s/it] 56%|█████▌ | 14/25 [00:28<00:22, 2.01s/it] 60%|██████ | 15/25 [00:30<00:20, 2.01s/it] 64%|██████▍ | 16/25 [00:32<00:18, 2.02s/it] 68%|██████▊ | 17/25 [00:34<00:16, 2.02s/it] 72%|███████▏ | 18/25 [00:36<00:14, 2.02s/it] 76%|███████▌ | 19/25 [00:38<00:12, 2.02s/it] 80%|████████ | 20/25 [00:40<00:10, 2.02s/it] 84%|████████▍ | 21/25 [00:42<00:08, 2.02s/it] 88%|████████▊ | 22/25 [00:44<00:06, 2.02s/it] 92%|█████████▏| 23/25 [00:46<00:04, 2.02s/it] 96%|█████████▌| 24/25 [00:48<00:02, 2.02s/it] 100%|██████████| 25/25 [00:50<00:00, 2.01s/it] 100%|██████████| 25/25 [00:50<00:00, 2.03s/it] 0%| | 0/48 [00:00<?, ?it/s] 12%|█▎ | 6/48 [00:00<00:00, 49.40it/s] 23%|██▎ | 11/48 [00:00<00:01, 33.66it/s] 31%|███▏ | 15/48 [00:00<00:01, 30.52it/s] 40%|███▉ | 19/48 [00:00<00:01, 28.99it/s] 46%|████▌ | 22/48 [00:00<00:00, 28.27it/s] 52%|█████▏ | 25/48 [00:00<00:00, 27.77it/s] 58%|█████▊ | 28/48 [00:00<00:00, 27.40it/s] 65%|██████▍ | 31/48 [00:01<00:00, 27.15it/s] 71%|███████ | 34/48 [00:01<00:00, 26.97it/s] 77%|███████▋ | 37/48 [00:01<00:00, 26.84it/s] 83%|████████▎ | 40/48 [00:01<00:00, 26.75it/s] 90%|████████▉ | 43/48 [00:01<00:00, 26.69it/s] 96%|█████████▌| 46/48 [00:01<00:00, 26.65it/s] 100%|██████████| 48/48 [00:01<00:00, 28.17it/s] IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (1544, 516) to (1552, 528) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). [swscaler @ 0x6185740] Warning: data is not aligned! This can lead to a speed loss 100%|██████████| 1/1 [00:55<00:00, 55.93s/it] 100%|██████████| 1/1 [00:55<00:00, 55.93s/it] /src/magic-animate/samples/config-2023-12-05T04-46-07/videos/tmpaf72liowdalle8_tmp868xbl66running.mp4Prediction
lucataco/magic-animate:e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714IDdpgwuujbayoh7psnb3w55hfohaStatusSucceededSourceWebHardwareA100 (40GB)Total durationCreatedInput
- image
- video
- guidance_scale
- 7.5
- num_inference_steps
- 25
{ "image": "https://replicate.delivery/pbxt/JzuGDlRZNTVFWwp8ND9OHwsswfhkxLsNQ6lqrdEYtuoge9rj/Shrek_%28character%29.jpeg", "video": "https://replicate.delivery/pbxt/JzuGDjyzx7sdybTacEo1BETL5KAVhTCHJPyvyuHigyshCMsH/demo4.mp4", "guidance_scale": 7.5, "num_inference_steps": 25 }Install Replicate’s Node.js client library:npm install replicateImport and set up the client:import Replicate from "replicate"; const replicate = new Replicate({ auth: process.env.REPLICATE_API_TOKEN, });Run lucataco/magic-animate using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
const output = await replicate.run( "lucataco/magic-animate:e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714", { input: { image: "https://replicate.delivery/pbxt/JzuGDlRZNTVFWwp8ND9OHwsswfhkxLsNQ6lqrdEYtuoge9rj/Shrek_%28character%29.jpeg", video: "https://replicate.delivery/pbxt/JzuGDjyzx7sdybTacEo1BETL5KAVhTCHJPyvyuHigyshCMsH/demo4.mp4", guidance_scale: 7.5, num_inference_steps: 25 } } ); // To access the file URL: console.log(output.url()); //=> "http://example.com" // To write the file to disk: fs.writeFile("my-image.png", output);To learn more, take a look at the guide on getting started with Node.js.
Install Replicate’s Python client library:pip install replicateImport the client:import replicateRun lucataco/magic-animate using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
output = replicate.run( "lucataco/magic-animate:e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714", input={ "image": "https://replicate.delivery/pbxt/JzuGDlRZNTVFWwp8ND9OHwsswfhkxLsNQ6lqrdEYtuoge9rj/Shrek_%28character%29.jpeg", "video": "https://replicate.delivery/pbxt/JzuGDjyzx7sdybTacEo1BETL5KAVhTCHJPyvyuHigyshCMsH/demo4.mp4", "guidance_scale": 7.5, "num_inference_steps": 25 } ) print(output)To learn more, take a look at the guide on getting started with Python.
Run lucataco/magic-animate using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
curl -s -X POST \ -H "Authorization: Bearer $REPLICATE_API_TOKEN" \ -H "Content-Type: application/json" \ -H "Prefer: wait" \ -d $'{ "version": "lucataco/magic-animate:e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714", "input": { "image": "https://replicate.delivery/pbxt/JzuGDlRZNTVFWwp8ND9OHwsswfhkxLsNQ6lqrdEYtuoge9rj/Shrek_%28character%29.jpeg", "video": "https://replicate.delivery/pbxt/JzuGDjyzx7sdybTacEo1BETL5KAVhTCHJPyvyuHigyshCMsH/demo4.mp4", "guidance_scale": 7.5, "num_inference_steps": 25 } }' \ https://api.replicate.com/v1/predictionsTo learn more, take a look at Replicate’s HTTP API reference docs.
Output
{ "completed_at": "2023-12-05T19:59:40.741536Z", "created_at": "2023-12-05T19:55:58.129702Z", "data_removed": false, "error": null, "id": "dpgwuujbayoh7psnb3w55hfoha", "input": { "image": "https://replicate.delivery/pbxt/JzuGDlRZNTVFWwp8ND9OHwsswfhkxLsNQ6lqrdEYtuoge9rj/Shrek_%28character%29.jpeg", "video": "https://replicate.delivery/pbxt/JzuGDjyzx7sdybTacEo1BETL5KAVhTCHJPyvyuHigyshCMsH/demo4.mp4", "guidance_scale": 7.5, "num_inference_steps": 25 }, "logs": "Using seed: 63307\n/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:43: FutureWarning: Importing `DiffusionPipeline` or `ImagePipelineOutput` from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead.\nfrom diffusers.pipeline_utils import DiffusionPipeline\nloaded temporal unet's pretrained weights from /src/pretrained_models/stable-diffusion-v1-5/unet ...\n### missing keys: 560;\n### unexpected keys: 0;\n### Temporal Module Parameters: 417.1376 M\nThe config attributes {'addition_embed_type': None, 'addition_embed_type_num_heads': 64, 'addition_time_embed_dim': None, 'conditioning_channels': 3, 'encoder_hid_dim': None, 'encoder_hid_dim_type': None, 'global_pool_conditions': False, 'num_attention_heads': None, 'transformer_layers_per_block': 1} were passed to ControlNetModel, but are not expected and will be ignored. Please verify your config.json configuration file.\nIt is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.\nIt is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.\nIt is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.\nIt is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.\n/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:103: FutureWarning: The configuration file of this scheduler: DDIMScheduler {\n\"_class_name\": \"DDIMScheduler\",\n\"_diffusers_version\": \"0.21.4\",\n\"beta_end\": 0.012,\n\"beta_schedule\": \"linear\",\n\"beta_start\": 0.00085,\n\"clip_sample\": true,\n\"clip_sample_range\": 1.0,\n\"dynamic_thresholding_ratio\": 0.995,\n\"num_train_timesteps\": 1000,\n\"prediction_type\": \"epsilon\",\n\"rescale_betas_zero_snr\": false,\n\"sample_max_value\": 1.0,\n\"set_alpha_to_one\": true,\n\"steps_offset\": 0,\n\"thresholding\": false,\n\"timestep_spacing\": \"leading\",\n\"trained_betas\": null\n}\nis outdated. `steps_offset` should be set to 1 instead of 0. Please make sure to update the config accordingly as leaving `steps_offset` might led to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file\ndeprecate(\"steps_offset!=1\", \"1.0.0\", deprecation_message, standard_warn=False)\n/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:116: FutureWarning: The configuration file of this scheduler: DDIMScheduler {\n\"_class_name\": \"DDIMScheduler\",\n\"_diffusers_version\": \"0.21.4\",\n\"beta_end\": 0.012,\n\"beta_schedule\": \"linear\",\n\"beta_start\": 0.00085,\n\"clip_sample\": true,\n\"clip_sample_range\": 1.0,\n\"dynamic_thresholding_ratio\": 0.995,\n\"num_train_timesteps\": 1000,\n\"prediction_type\": \"epsilon\",\n\"rescale_betas_zero_snr\": false,\n\"sample_max_value\": 1.0,\n\"set_alpha_to_one\": true,\n\"steps_offset\": 1,\n\"thresholding\": false,\n\"timestep_spacing\": \"leading\",\n\"trained_betas\": null\n}\nhas not set the configuration `clip_sample`. `clip_sample` should be set to False in the configuration file. Please make sure to update the config accordingly as not setting `clip_sample` in the config might lead to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file\ndeprecate(\"clip_sample not set\", \"1.0.0\", deprecation_message, standard_warn=False)\ncurrent seed: 63307\n 0%| | 0/1 [00:00<?, ?it/s]/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:624: FutureWarning: Accessing config attribute `in_channels` directly via 'UNet3DConditionModel' object attribute is deprecated. Please access 'in_channels' over 'UNet3DConditionModel's config object instead, e.g. 'unet.config.in_channels'.\nnum_channels_latents = self.unet.in_channels\n 0%| | 0/25 [00:00<?, ?it/s]\u001b[A\n 4%|▍ | 1/25 [00:05<02:01, 5.05s/it]\u001b[A\n 8%|▊ | 2/25 [00:09<01:53, 4.94s/it]\u001b[A\n 12%|█▏ | 3/25 [00:14<01:47, 4.90s/it]\u001b[A\n 16%|█▌ | 4/25 [00:19<01:42, 4.89s/it]\u001b[A\n 20%|██ | 5/25 [00:24<01:37, 4.88s/it]\u001b[A\n 24%|██▍ | 6/25 [00:29<01:32, 4.88s/it]\u001b[A\n 28%|██▊ | 7/25 [00:34<01:27, 4.88s/it]\u001b[A\n 32%|███▏ | 8/25 [00:39<01:22, 4.87s/it]\u001b[A\n 36%|███▌ | 9/25 [00:44<01:18, 4.88s/it]\u001b[A\n 40%|████ | 10/25 [00:48<01:13, 4.87s/it]\u001b[A\n 44%|████▍ | 11/25 [00:53<01:08, 4.88s/it]\u001b[A\n 48%|████▊ | 12/25 [00:58<01:03, 4.88s/it]\u001b[A\n 52%|█████▏ | 13/25 [01:03<00:58, 4.88s/it]\u001b[A\n 56%|█████▌ | 14/25 [01:08<00:53, 4.88s/it]\u001b[A\n 60%|██████ | 15/25 [01:13<00:48, 4.88s/it]\u001b[A\n 64%|██████▍ | 16/25 [01:18<00:43, 4.88s/it]\u001b[A\n 68%|██████▊ | 17/25 [01:23<00:39, 4.88s/it]\u001b[A\n 72%|███████▏ | 18/25 [01:27<00:34, 4.88s/it]\u001b[A\n 76%|███████▌ | 19/25 [01:32<00:29, 4.87s/it]\u001b[A\n 80%|████████ | 20/25 [01:37<00:24, 4.88s/it]\u001b[A\n 84%|████████▍ | 21/25 [01:42<00:19, 4.88s/it]\u001b[A\n 88%|████████▊ | 22/25 [01:47<00:14, 4.88s/it]\u001b[A\n 92%|█████████▏| 23/25 [01:52<00:09, 4.89s/it]\u001b[A\n 96%|█████████▌| 24/25 [01:57<00:04, 4.89s/it]\u001b[A\n100%|██████████| 25/25 [02:02<00:00, 4.89s/it]\u001b[A\n100%|██████████| 25/25 [02:02<00:00, 4.88s/it]\n 0%| | 0/112 [00:00<?, ?it/s]\u001b[A\n 5%|▌ | 6/112 [00:00<00:02, 48.50it/s]\u001b[A\n 10%|▉ | 11/112 [00:00<00:03, 33.46it/s]\u001b[A\n 13%|█▎ | 15/112 [00:00<00:03, 30.42it/s]\u001b[A\n 17%|█▋ | 19/112 [00:00<00:03, 28.93it/s]\u001b[A\n 20%|█▉ | 22/112 [00:00<00:03, 28.24it/s]\u001b[A\n 22%|██▏ | 25/112 [00:00<00:03, 27.74it/s]\u001b[A\n 25%|██▌ | 28/112 [00:00<00:03, 27.39it/s]\u001b[A\n 28%|██▊ | 31/112 [00:01<00:02, 27.14it/s]\u001b[A\n 30%|███ | 34/112 [00:01<00:02, 26.96it/s]\u001b[A\n 33%|███▎ | 37/112 [00:01<00:02, 26.84it/s]\u001b[A\n 36%|███▌ | 40/112 [00:01<00:02, 26.76it/s]\u001b[A\n 38%|███▊ | 43/112 [00:01<00:02, 26.70it/s]\u001b[A\n 41%|████ | 46/112 [00:01<00:02, 26.65it/s]\u001b[A\n 44%|████▍ | 49/112 [00:01<00:02, 26.63it/s]\u001b[A\n 46%|████▋ | 52/112 [00:01<00:02, 26.61it/s]\u001b[A\n 49%|████▉ | 55/112 [00:01<00:02, 26.59it/s]\u001b[A\n 52%|█████▏ | 58/112 [00:02<00:02, 26.58it/s]\u001b[A\n 54%|█████▍ | 61/112 [00:02<00:01, 26.57it/s]\u001b[A\n 57%|█████▋ | 64/112 [00:02<00:01, 26.57it/s]\u001b[A\n 60%|█████▉ | 67/112 [00:02<00:01, 26.57it/s]\u001b[A\n 62%|██████▎ | 70/112 [00:02<00:01, 26.56it/s]\u001b[A\n 65%|██████▌ | 73/112 [00:02<00:01, 26.56it/s]\u001b[A\n 68%|██████▊ | 76/112 [00:02<00:01, 26.56it/s]\u001b[A\n 71%|███████ | 79/112 [00:02<00:01, 26.56it/s]\u001b[A\n 73%|███████▎ | 82/112 [00:02<00:01, 26.56it/s]\u001b[A\n 76%|███████▌ | 85/112 [00:03<00:01, 26.56it/s]\u001b[A\n 79%|███████▊ | 88/112 [00:03<00:00, 26.55it/s]\u001b[A\n 81%|████████▏ | 91/112 [00:03<00:00, 26.55it/s]\u001b[A\n 84%|████████▍ | 94/112 [00:03<00:00, 26.56it/s]\u001b[A\n 87%|████████▋ | 97/112 [00:03<00:00, 26.56it/s]\u001b[A\n 89%|████████▉ | 100/112 [00:03<00:00, 26.56it/s]\u001b[A\n 92%|█████████▏| 103/112 [00:03<00:00, 26.56it/s]\u001b[A\n 95%|█████████▍| 106/112 [00:03<00:00, 26.57it/s]\u001b[A\n 97%|█████████▋| 109/112 [00:04<00:00, 26.57it/s]\u001b[A\n100%|██████████| 112/112 [00:04<00:00, 26.57it/s]\u001b[A\n100%|██████████| 112/112 [00:04<00:00, 27.21it/s]\nIMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (1544, 516) to (1552, 528) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility).\n[swscaler @ 0x5525740] Warning: data is not aligned! This can lead to a speed loss\n100%|██████████| 1/1 [02:12<00:00, 132.05s/it]\n100%|██████████| 1/1 [02:12<00:00, 132.05s/it]\n/src/magic-animate/samples/config-2023-12-05T19-57-02/videos/tmpkm6j0w5xShrek_(character)_tmpp6m3ibucdemo4.mp4", "metrics": { "predict_time": 162.648175, "total_time": 222.611834 }, "output": "https://replicate.delivery/pbxt/IeHubfyT1TjlwULL8WdAftmB4sZphfGoEHV5mxQekACmlZ6PC/tmpkm6j0w5xShrek_%28character%29_tmpp6m3ibucdemo4.mp4", "started_at": "2023-12-05T19:56:58.093361Z", "status": "succeeded", "urls": { "get": "https://api.replicate.com/v1/predictions/dpgwuujbayoh7psnb3w55hfoha", "cancel": "https://api.replicate.com/v1/predictions/dpgwuujbayoh7psnb3w55hfoha/cancel" }, "version": "e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714" }Generated inUsing seed: 63307 /src/magic-animate/magicanimate/pipelines/pipeline_animation.py:43: FutureWarning: Importing `DiffusionPipeline` or `ImagePipelineOutput` from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead. from diffusers.pipeline_utils import DiffusionPipeline loaded temporal unet's pretrained weights from /src/pretrained_models/stable-diffusion-v1-5/unet ... ### missing keys: 560; ### unexpected keys: 0; ### Temporal Module Parameters: 417.1376 M The config attributes {'addition_embed_type': None, 'addition_embed_type_num_heads': 64, 'addition_time_embed_dim': None, 'conditioning_channels': 3, 'encoder_hid_dim': None, 'encoder_hid_dim_type': None, 'global_pool_conditions': False, 'num_attention_heads': None, 'transformer_layers_per_block': 1} were passed to ControlNetModel, but are not expected and will be ignored. Please verify your config.json configuration file. It is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8. It is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8. It is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8. It is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8. /src/magic-animate/magicanimate/pipelines/pipeline_animation.py:103: FutureWarning: The configuration file of this scheduler: DDIMScheduler { "_class_name": "DDIMScheduler", "_diffusers_version": "0.21.4", "beta_end": 0.012, "beta_schedule": "linear", "beta_start": 0.00085, "clip_sample": true, "clip_sample_range": 1.0, "dynamic_thresholding_ratio": 0.995, "num_train_timesteps": 1000, "prediction_type": "epsilon", "rescale_betas_zero_snr": false, "sample_max_value": 1.0, "set_alpha_to_one": true, "steps_offset": 0, "thresholding": false, "timestep_spacing": "leading", "trained_betas": null } is outdated. `steps_offset` should be set to 1 instead of 0. Please make sure to update the config accordingly as leaving `steps_offset` might led to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file deprecate("steps_offset!=1", "1.0.0", deprecation_message, standard_warn=False) /src/magic-animate/magicanimate/pipelines/pipeline_animation.py:116: FutureWarning: The configuration file of this scheduler: DDIMScheduler { "_class_name": "DDIMScheduler", "_diffusers_version": "0.21.4", "beta_end": 0.012, "beta_schedule": "linear", "beta_start": 0.00085, "clip_sample": true, "clip_sample_range": 1.0, "dynamic_thresholding_ratio": 0.995, "num_train_timesteps": 1000, "prediction_type": "epsilon", "rescale_betas_zero_snr": false, "sample_max_value": 1.0, "set_alpha_to_one": true, "steps_offset": 1, "thresholding": false, "timestep_spacing": "leading", "trained_betas": null } has not set the configuration `clip_sample`. `clip_sample` should be set to False in the configuration file. Please make sure to update the config accordingly as not setting `clip_sample` in the config might lead to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file deprecate("clip_sample not set", "1.0.0", deprecation_message, standard_warn=False) current seed: 63307 0%| | 0/1 [00:00<?, ?it/s]/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:624: FutureWarning: Accessing config attribute `in_channels` directly via 'UNet3DConditionModel' object attribute is deprecated. Please access 'in_channels' over 'UNet3DConditionModel's config object instead, e.g. 'unet.config.in_channels'. num_channels_latents = self.unet.in_channels 0%| | 0/25 [00:00<?, ?it/s] 4%|▍ | 1/25 [00:05<02:01, 5.05s/it] 8%|▊ | 2/25 [00:09<01:53, 4.94s/it] 12%|█▏ | 3/25 [00:14<01:47, 4.90s/it] 16%|█▌ | 4/25 [00:19<01:42, 4.89s/it] 20%|██ | 5/25 [00:24<01:37, 4.88s/it] 24%|██▍ | 6/25 [00:29<01:32, 4.88s/it] 28%|██▊ | 7/25 [00:34<01:27, 4.88s/it] 32%|███▏ | 8/25 [00:39<01:22, 4.87s/it] 36%|███▌ | 9/25 [00:44<01:18, 4.88s/it] 40%|████ | 10/25 [00:48<01:13, 4.87s/it] 44%|████▍ | 11/25 [00:53<01:08, 4.88s/it] 48%|████▊ | 12/25 [00:58<01:03, 4.88s/it] 52%|█████▏ | 13/25 [01:03<00:58, 4.88s/it] 56%|█████▌ | 14/25 [01:08<00:53, 4.88s/it] 60%|██████ | 15/25 [01:13<00:48, 4.88s/it] 64%|██████▍ | 16/25 [01:18<00:43, 4.88s/it] 68%|██████▊ | 17/25 [01:23<00:39, 4.88s/it] 72%|███████▏ | 18/25 [01:27<00:34, 4.88s/it] 76%|███████▌ | 19/25 [01:32<00:29, 4.87s/it] 80%|████████ | 20/25 [01:37<00:24, 4.88s/it] 84%|████████▍ | 21/25 [01:42<00:19, 4.88s/it] 88%|████████▊ | 22/25 [01:47<00:14, 4.88s/it] 92%|█████████▏| 23/25 [01:52<00:09, 4.89s/it] 96%|█████████▌| 24/25 [01:57<00:04, 4.89s/it] 100%|██████████| 25/25 [02:02<00:00, 4.89s/it] 100%|██████████| 25/25 [02:02<00:00, 4.88s/it] 0%| | 0/112 [00:00<?, ?it/s] 5%|▌ | 6/112 [00:00<00:02, 48.50it/s] 10%|▉ | 11/112 [00:00<00:03, 33.46it/s] 13%|█▎ | 15/112 [00:00<00:03, 30.42it/s] 17%|█▋ | 19/112 [00:00<00:03, 28.93it/s] 20%|█▉ | 22/112 [00:00<00:03, 28.24it/s] 22%|██▏ | 25/112 [00:00<00:03, 27.74it/s] 25%|██▌ | 28/112 [00:00<00:03, 27.39it/s] 28%|██▊ | 31/112 [00:01<00:02, 27.14it/s] 30%|███ | 34/112 [00:01<00:02, 26.96it/s] 33%|███▎ | 37/112 [00:01<00:02, 26.84it/s] 36%|███▌ | 40/112 [00:01<00:02, 26.76it/s] 38%|███▊ | 43/112 [00:01<00:02, 26.70it/s] 41%|████ | 46/112 [00:01<00:02, 26.65it/s] 44%|████▍ | 49/112 [00:01<00:02, 26.63it/s] 46%|████▋ | 52/112 [00:01<00:02, 26.61it/s] 49%|████▉ | 55/112 [00:01<00:02, 26.59it/s] 52%|█████▏ | 58/112 [00:02<00:02, 26.58it/s] 54%|█████▍ | 61/112 [00:02<00:01, 26.57it/s] 57%|█████▋ | 64/112 [00:02<00:01, 26.57it/s] 60%|█████▉ | 67/112 [00:02<00:01, 26.57it/s] 62%|██████▎ | 70/112 [00:02<00:01, 26.56it/s] 65%|██████▌ | 73/112 [00:02<00:01, 26.56it/s] 68%|██████▊ | 76/112 [00:02<00:01, 26.56it/s] 71%|███████ | 79/112 [00:02<00:01, 26.56it/s] 73%|███████▎ | 82/112 [00:02<00:01, 26.56it/s] 76%|███████▌ | 85/112 [00:03<00:01, 26.56it/s] 79%|███████▊ | 88/112 [00:03<00:00, 26.55it/s] 81%|████████▏ | 91/112 [00:03<00:00, 26.55it/s] 84%|████████▍ | 94/112 [00:03<00:00, 26.56it/s] 87%|████████▋ | 97/112 [00:03<00:00, 26.56it/s] 89%|████████▉ | 100/112 [00:03<00:00, 26.56it/s] 92%|█████████▏| 103/112 [00:03<00:00, 26.56it/s] 95%|█████████▍| 106/112 [00:03<00:00, 26.57it/s] 97%|█████████▋| 109/112 [00:04<00:00, 26.57it/s] 100%|██████████| 112/112 [00:04<00:00, 26.57it/s] 100%|██████████| 112/112 [00:04<00:00, 27.21it/s] IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (1544, 516) to (1552, 528) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). [swscaler @ 0x5525740] Warning: data is not aligned! This can lead to a speed loss 100%|██████████| 1/1 [02:12<00:00, 132.05s/it] 100%|██████████| 1/1 [02:12<00:00, 132.05s/it] /src/magic-animate/samples/config-2023-12-05T19-57-02/videos/tmpkm6j0w5xShrek_(character)_tmpp6m3ibucdemo4.mp4Prediction
lucataco/magic-animate:e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714IDxqiik5zbjv6hvuhr2atlin42rmStatusSucceededSourceWebHardwareA100 (40GB)Total durationCreatedby @fofrInput
- image
- video
- guidance_scale
- 7.5
- num_inference_steps
- 25
{ "image": "https://replicate.delivery/pbxt/JzvM9XtqsMeLzNcMfO1AISTv6TndKGPB0Lq6GajEQd7Rqj70/meme.jpeg", "video": "https://replicate.delivery/pbxt/JzvM9RW3L86hkDTYb3RhvOr1YAstTktlOvN2TZ3jMsArFyUi/multi_dancing.mp4", "guidance_scale": 7.5, "num_inference_steps": 25 }Install Replicate’s Node.js client library:npm install replicateImport and set up the client:import Replicate from "replicate"; const replicate = new Replicate({ auth: process.env.REPLICATE_API_TOKEN, });Run lucataco/magic-animate using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
const output = await replicate.run( "lucataco/magic-animate:e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714", { input: { image: "https://replicate.delivery/pbxt/JzvM9XtqsMeLzNcMfO1AISTv6TndKGPB0Lq6GajEQd7Rqj70/meme.jpeg", video: "https://replicate.delivery/pbxt/JzvM9RW3L86hkDTYb3RhvOr1YAstTktlOvN2TZ3jMsArFyUi/multi_dancing.mp4", guidance_scale: 7.5, num_inference_steps: 25 } } ); // To access the file URL: console.log(output.url()); //=> "http://example.com" // To write the file to disk: fs.writeFile("my-image.png", output);To learn more, take a look at the guide on getting started with Node.js.
Install Replicate’s Python client library:pip install replicateImport the client:import replicateRun lucataco/magic-animate using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
output = replicate.run( "lucataco/magic-animate:e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714", input={ "image": "https://replicate.delivery/pbxt/JzvM9XtqsMeLzNcMfO1AISTv6TndKGPB0Lq6GajEQd7Rqj70/meme.jpeg", "video": "https://replicate.delivery/pbxt/JzvM9RW3L86hkDTYb3RhvOr1YAstTktlOvN2TZ3jMsArFyUi/multi_dancing.mp4", "guidance_scale": 7.5, "num_inference_steps": 25 } ) print(output)To learn more, take a look at the guide on getting started with Python.
Run lucataco/magic-animate using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
curl -s -X POST \ -H "Authorization: Bearer $REPLICATE_API_TOKEN" \ -H "Content-Type: application/json" \ -H "Prefer: wait" \ -d $'{ "version": "lucataco/magic-animate:e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714", "input": { "image": "https://replicate.delivery/pbxt/JzvM9XtqsMeLzNcMfO1AISTv6TndKGPB0Lq6GajEQd7Rqj70/meme.jpeg", "video": "https://replicate.delivery/pbxt/JzvM9RW3L86hkDTYb3RhvOr1YAstTktlOvN2TZ3jMsArFyUi/multi_dancing.mp4", "guidance_scale": 7.5, "num_inference_steps": 25 } }' \ https://api.replicate.com/v1/predictionsTo learn more, take a look at Replicate’s HTTP API reference docs.
Output
{ "completed_at": "2023-12-05T21:09:58.988081Z", "created_at": "2023-12-05T21:07:39.467889Z", "data_removed": false, "error": null, "id": "xqiik5zbjv6hvuhr2atlin42rm", "input": { "image": "https://replicate.delivery/pbxt/JzvM9XtqsMeLzNcMfO1AISTv6TndKGPB0Lq6GajEQd7Rqj70/meme.jpeg", "video": "https://replicate.delivery/pbxt/JzvM9RW3L86hkDTYb3RhvOr1YAstTktlOvN2TZ3jMsArFyUi/multi_dancing.mp4", "guidance_scale": 7.5, "num_inference_steps": 25 }, "logs": "Using seed: 3659\n/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:43: FutureWarning: Importing `DiffusionPipeline` or `ImagePipelineOutput` from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead.\nfrom diffusers.pipeline_utils import DiffusionPipeline\nloaded temporal unet's pretrained weights from /src/pretrained_models/stable-diffusion-v1-5/unet ...\n### missing keys: 560;\n### unexpected keys: 0;\n### Temporal Module Parameters: 417.1376 M\nThe config attributes {'addition_embed_type': None, 'addition_embed_type_num_heads': 64, 'addition_time_embed_dim': None, 'conditioning_channels': 3, 'encoder_hid_dim': None, 'encoder_hid_dim_type': None, 'global_pool_conditions': False, 'num_attention_heads': None, 'transformer_layers_per_block': 1} were passed to ControlNetModel, but are not expected and will be ignored. Please verify your config.json configuration file.\nIt is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.\nIt is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.\nIt is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.\nIt is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8.\n/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:103: FutureWarning: The configuration file of this scheduler: DDIMScheduler {\n\"_class_name\": \"DDIMScheduler\",\n\"_diffusers_version\": \"0.21.4\",\n\"beta_end\": 0.012,\n\"beta_schedule\": \"linear\",\n\"beta_start\": 0.00085,\n\"clip_sample\": true,\n\"clip_sample_range\": 1.0,\n\"dynamic_thresholding_ratio\": 0.995,\n\"num_train_timesteps\": 1000,\n\"prediction_type\": \"epsilon\",\n\"rescale_betas_zero_snr\": false,\n\"sample_max_value\": 1.0,\n\"set_alpha_to_one\": true,\n\"steps_offset\": 0,\n\"thresholding\": false,\n\"timestep_spacing\": \"leading\",\n\"trained_betas\": null\n}\nis outdated. `steps_offset` should be set to 1 instead of 0. Please make sure to update the config accordingly as leaving `steps_offset` might led to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file\ndeprecate(\"steps_offset!=1\", \"1.0.0\", deprecation_message, standard_warn=False)\n/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:116: FutureWarning: The configuration file of this scheduler: DDIMScheduler {\n\"_class_name\": \"DDIMScheduler\",\n\"_diffusers_version\": \"0.21.4\",\n\"beta_end\": 0.012,\n\"beta_schedule\": \"linear\",\n\"beta_start\": 0.00085,\n\"clip_sample\": true,\n\"clip_sample_range\": 1.0,\n\"dynamic_thresholding_ratio\": 0.995,\n\"num_train_timesteps\": 1000,\n\"prediction_type\": \"epsilon\",\n\"rescale_betas_zero_snr\": false,\n\"sample_max_value\": 1.0,\n\"set_alpha_to_one\": true,\n\"steps_offset\": 1,\n\"thresholding\": false,\n\"timestep_spacing\": \"leading\",\n\"trained_betas\": null\n}\nhas not set the configuration `clip_sample`. `clip_sample` should be set to False in the configuration file. Please make sure to update the config accordingly as not setting `clip_sample` in the config might lead to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file\ndeprecate(\"clip_sample not set\", \"1.0.0\", deprecation_message, standard_warn=False)\ncurrent seed: 3659\n 0%| | 0/1 [00:00<?, ?it/s]/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:624: FutureWarning: Accessing config attribute `in_channels` directly via 'UNet3DConditionModel' object attribute is deprecated. Please access 'in_channels' over 'UNet3DConditionModel's config object instead, e.g. 'unet.config.in_channels'.\nnum_channels_latents = self.unet.in_channels\n 0%| | 0/25 [00:00<?, ?it/s]\u001b[A\n 4%|▍ | 1/25 [00:04<01:38, 4.12s/it]\u001b[A\n 8%|▊ | 2/25 [00:08<01:32, 4.02s/it]\u001b[A\n 12%|█▏ | 3/25 [00:11<01:27, 3.97s/it]\u001b[A\n 16%|█▌ | 4/25 [00:15<01:23, 3.97s/it]\u001b[A\n 20%|██ | 5/25 [00:19<01:19, 3.97s/it]\u001b[A\n 24%|██▍ | 6/25 [00:23<01:15, 3.96s/it]\u001b[A\n 28%|██▊ | 7/25 [00:27<01:11, 3.96s/it]\u001b[A\n 32%|███▏ | 8/25 [00:31<01:07, 3.96s/it]\u001b[A\n 36%|███▌ | 9/25 [00:35<01:03, 3.96s/it]\u001b[A\n 40%|████ | 10/25 [00:39<00:59, 3.96s/it]\u001b[A\n 44%|████▍ | 11/25 [00:43<00:55, 3.96s/it]\u001b[A\n 48%|████▊ | 12/25 [00:47<00:51, 3.95s/it]\u001b[A\n 52%|█████▏ | 13/25 [00:51<00:47, 3.95s/it]\u001b[A\n 56%|█████▌ | 14/25 [00:55<00:43, 3.95s/it]\u001b[A\n 60%|██████ | 15/25 [00:59<00:39, 3.95s/it]\u001b[A\n 64%|██████▍ | 16/25 [01:03<00:35, 3.95s/it]\u001b[A\n 68%|██████▊ | 17/25 [01:07<00:31, 3.95s/it]\u001b[A\n 72%|███████▏ | 18/25 [01:11<00:27, 3.96s/it]\u001b[A\n 76%|███████▌ | 19/25 [01:15<00:23, 3.96s/it]\u001b[A\n 80%|████████ | 20/25 [01:19<00:19, 3.96s/it]\u001b[A\n 84%|████████▍ | 21/25 [01:23<00:15, 3.96s/it]\u001b[A\n 88%|████████▊ | 22/25 [01:27<00:11, 3.96s/it]\u001b[A\n 92%|█████████▏| 23/25 [01:31<00:07, 3.96s/it]\u001b[A\n 96%|█████████▌| 24/25 [01:35<00:03, 3.96s/it]\u001b[A\n100%|██████████| 25/25 [01:39<00:00, 3.97s/it]\u001b[A\n100%|██████████| 25/25 [01:39<00:00, 3.96s/it]\n 0%| | 0/96 [00:00<?, ?it/s]\u001b[A\n 6%|▋ | 6/96 [00:00<00:01, 46.92it/s]\u001b[A\n 11%|█▏ | 11/96 [00:00<00:02, 33.05it/s]\u001b[A\n 16%|█▌ | 15/96 [00:00<00:02, 30.18it/s]\u001b[A\n 20%|█▉ | 19/96 [00:00<00:02, 28.76it/s]\u001b[A\n 23%|██▎ | 22/96 [00:00<00:02, 28.10it/s]\u001b[A\n 26%|██▌ | 25/96 [00:00<00:02, 27.62it/s]\u001b[A\n 29%|██▉ | 28/96 [00:00<00:02, 27.29it/s]\u001b[A\n 32%|███▏ | 31/96 [00:01<00:02, 27.05it/s]\u001b[A\n 35%|███▌ | 34/96 [00:01<00:02, 26.88it/s]\u001b[A\n 39%|███▊ | 37/96 [00:01<00:02, 26.76it/s]\u001b[A\n 42%|████▏ | 40/96 [00:01<00:02, 26.68it/s]\u001b[A\n 45%|████▍ | 43/96 [00:01<00:01, 26.62it/s]\u001b[A\n 48%|████▊ | 46/96 [00:01<00:01, 26.58it/s]\u001b[A\n 51%|█████ | 49/96 [00:01<00:01, 26.55it/s]\u001b[A\n 54%|█████▍ | 52/96 [00:01<00:01, 26.54it/s]\u001b[A\n 57%|█████▋ | 55/96 [00:01<00:01, 26.52it/s]\u001b[A\n 60%|██████ | 58/96 [00:02<00:01, 26.51it/s]\u001b[A\n 64%|██████▎ | 61/96 [00:02<00:01, 26.50it/s]\u001b[A\n 67%|██████▋ | 64/96 [00:02<00:01, 26.50it/s]\u001b[A\n 70%|██████▉ | 67/96 [00:02<00:01, 26.49it/s]\u001b[A\n 73%|███████▎ | 70/96 [00:02<00:00, 26.49it/s]\u001b[A\n 76%|███████▌ | 73/96 [00:02<00:00, 26.49it/s]\u001b[A\n 79%|███████▉ | 76/96 [00:02<00:00, 26.49it/s]\u001b[A\n 82%|████████▏ | 79/96 [00:02<00:00, 26.49it/s]\u001b[A\n 85%|████████▌ | 82/96 [00:02<00:00, 26.49it/s]\u001b[A\n 89%|████████▊ | 85/96 [00:03<00:00, 26.48it/s]\u001b[A\n 92%|█████████▏| 88/96 [00:03<00:00, 26.49it/s]\u001b[A\n 95%|█████████▍| 91/96 [00:03<00:00, 26.49it/s]\u001b[A\n 98%|█████████▊| 94/96 [00:03<00:00, 26.49it/s]\u001b[A\n100%|██████████| 96/96 [00:03<00:00, 27.22it/s]\nIMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (1544, 516) to (1552, 528) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility).\n[swscaler @ 0x5baa740] Warning: data is not aligned! This can lead to a speed loss\n100%|██████████| 1/1 [01:48<00:00, 108.06s/it]\n100%|██████████| 1/1 [01:48<00:00, 108.06s/it]\n/src/magic-animate/samples/config-2023-12-05T21-07-44/videos/tmp3x9__2mjmeme_tmp8yfgquj_multi_dancing.mp4", "metrics": { "predict_time": 139.505846, "total_time": 139.520192 }, "output": "https://replicate.delivery/pbxt/qy0jsX7eYTyEBS452fKpLMvKIbCf9xR6RO3UYZwUYgpNdoeHB/tmp3x9__2mjmeme_tmp8yfgquj_multi_dancing.mp4", "started_at": "2023-12-05T21:07:39.482235Z", "status": "succeeded", "urls": { "get": "https://api.replicate.com/v1/predictions/xqiik5zbjv6hvuhr2atlin42rm", "cancel": "https://api.replicate.com/v1/predictions/xqiik5zbjv6hvuhr2atlin42rm/cancel" }, "version": "e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714" }Generated inUsing seed: 3659 /src/magic-animate/magicanimate/pipelines/pipeline_animation.py:43: FutureWarning: Importing `DiffusionPipeline` or `ImagePipelineOutput` from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead. from diffusers.pipeline_utils import DiffusionPipeline loaded temporal unet's pretrained weights from /src/pretrained_models/stable-diffusion-v1-5/unet ... ### missing keys: 560; ### unexpected keys: 0; ### Temporal Module Parameters: 417.1376 M The config attributes {'addition_embed_type': None, 'addition_embed_type_num_heads': 64, 'addition_time_embed_dim': None, 'conditioning_channels': 3, 'encoder_hid_dim': None, 'encoder_hid_dim_type': None, 'global_pool_conditions': False, 'num_attention_heads': None, 'transformer_layers_per_block': 1} were passed to ControlNetModel, but are not expected and will be ignored. Please verify your config.json configuration file. It is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8. It is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8. It is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8. It is recommended to provide `attention_head_dim` when calling `get_down_block`. Defaulting `attention_head_dim` to 8. /src/magic-animate/magicanimate/pipelines/pipeline_animation.py:103: FutureWarning: The configuration file of this scheduler: DDIMScheduler { "_class_name": "DDIMScheduler", "_diffusers_version": "0.21.4", "beta_end": 0.012, "beta_schedule": "linear", "beta_start": 0.00085, "clip_sample": true, "clip_sample_range": 1.0, "dynamic_thresholding_ratio": 0.995, "num_train_timesteps": 1000, "prediction_type": "epsilon", "rescale_betas_zero_snr": false, "sample_max_value": 1.0, "set_alpha_to_one": true, "steps_offset": 0, "thresholding": false, "timestep_spacing": "leading", "trained_betas": null } is outdated. `steps_offset` should be set to 1 instead of 0. Please make sure to update the config accordingly as leaving `steps_offset` might led to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file deprecate("steps_offset!=1", "1.0.0", deprecation_message, standard_warn=False) /src/magic-animate/magicanimate/pipelines/pipeline_animation.py:116: FutureWarning: The configuration file of this scheduler: DDIMScheduler { "_class_name": "DDIMScheduler", "_diffusers_version": "0.21.4", "beta_end": 0.012, "beta_schedule": "linear", "beta_start": 0.00085, "clip_sample": true, "clip_sample_range": 1.0, "dynamic_thresholding_ratio": 0.995, "num_train_timesteps": 1000, "prediction_type": "epsilon", "rescale_betas_zero_snr": false, "sample_max_value": 1.0, "set_alpha_to_one": true, "steps_offset": 1, "thresholding": false, "timestep_spacing": "leading", "trained_betas": null } has not set the configuration `clip_sample`. `clip_sample` should be set to False in the configuration file. Please make sure to update the config accordingly as not setting `clip_sample` in the config might lead to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file deprecate("clip_sample not set", "1.0.0", deprecation_message, standard_warn=False) current seed: 3659 0%| | 0/1 [00:00<?, ?it/s]/src/magic-animate/magicanimate/pipelines/pipeline_animation.py:624: FutureWarning: Accessing config attribute `in_channels` directly via 'UNet3DConditionModel' object attribute is deprecated. Please access 'in_channels' over 'UNet3DConditionModel's config object instead, e.g. 'unet.config.in_channels'. num_channels_latents = self.unet.in_channels 0%| | 0/25 [00:00<?, ?it/s] 4%|▍ | 1/25 [00:04<01:38, 4.12s/it] 8%|▊ | 2/25 [00:08<01:32, 4.02s/it] 12%|█▏ | 3/25 [00:11<01:27, 3.97s/it] 16%|█▌ | 4/25 [00:15<01:23, 3.97s/it] 20%|██ | 5/25 [00:19<01:19, 3.97s/it] 24%|██▍ | 6/25 [00:23<01:15, 3.96s/it] 28%|██▊ | 7/25 [00:27<01:11, 3.96s/it] 32%|███▏ | 8/25 [00:31<01:07, 3.96s/it] 36%|███▌ | 9/25 [00:35<01:03, 3.96s/it] 40%|████ | 10/25 [00:39<00:59, 3.96s/it] 44%|████▍ | 11/25 [00:43<00:55, 3.96s/it] 48%|████▊ | 12/25 [00:47<00:51, 3.95s/it] 52%|█████▏ | 13/25 [00:51<00:47, 3.95s/it] 56%|█████▌ | 14/25 [00:55<00:43, 3.95s/it] 60%|██████ | 15/25 [00:59<00:39, 3.95s/it] 64%|██████▍ | 16/25 [01:03<00:35, 3.95s/it] 68%|██████▊ | 17/25 [01:07<00:31, 3.95s/it] 72%|███████▏ | 18/25 [01:11<00:27, 3.96s/it] 76%|███████▌ | 19/25 [01:15<00:23, 3.96s/it] 80%|████████ | 20/25 [01:19<00:19, 3.96s/it] 84%|████████▍ | 21/25 [01:23<00:15, 3.96s/it] 88%|████████▊ | 22/25 [01:27<00:11, 3.96s/it] 92%|█████████▏| 23/25 [01:31<00:07, 3.96s/it] 96%|█████████▌| 24/25 [01:35<00:03, 3.96s/it] 100%|██████████| 25/25 [01:39<00:00, 3.97s/it] 100%|██████████| 25/25 [01:39<00:00, 3.96s/it] 0%| | 0/96 [00:00<?, ?it/s] 6%|▋ | 6/96 [00:00<00:01, 46.92it/s] 11%|█▏ | 11/96 [00:00<00:02, 33.05it/s] 16%|█▌ | 15/96 [00:00<00:02, 30.18it/s] 20%|█▉ | 19/96 [00:00<00:02, 28.76it/s] 23%|██▎ | 22/96 [00:00<00:02, 28.10it/s] 26%|██▌ | 25/96 [00:00<00:02, 27.62it/s] 29%|██▉ | 28/96 [00:00<00:02, 27.29it/s] 32%|███▏ | 31/96 [00:01<00:02, 27.05it/s] 35%|███▌ | 34/96 [00:01<00:02, 26.88it/s] 39%|███▊ | 37/96 [00:01<00:02, 26.76it/s] 42%|████▏ | 40/96 [00:01<00:02, 26.68it/s] 45%|████▍ | 43/96 [00:01<00:01, 26.62it/s] 48%|████▊ | 46/96 [00:01<00:01, 26.58it/s] 51%|█████ | 49/96 [00:01<00:01, 26.55it/s] 54%|█████▍ | 52/96 [00:01<00:01, 26.54it/s] 57%|█████▋ | 55/96 [00:01<00:01, 26.52it/s] 60%|██████ | 58/96 [00:02<00:01, 26.51it/s] 64%|██████▎ | 61/96 [00:02<00:01, 26.50it/s] 67%|██████▋ | 64/96 [00:02<00:01, 26.50it/s] 70%|██████▉ | 67/96 [00:02<00:01, 26.49it/s] 73%|███████▎ | 70/96 [00:02<00:00, 26.49it/s] 76%|███████▌ | 73/96 [00:02<00:00, 26.49it/s] 79%|███████▉ | 76/96 [00:02<00:00, 26.49it/s] 82%|████████▏ | 79/96 [00:02<00:00, 26.49it/s] 85%|████████▌ | 82/96 [00:02<00:00, 26.49it/s] 89%|████████▊ | 85/96 [00:03<00:00, 26.48it/s] 92%|█████████▏| 88/96 [00:03<00:00, 26.49it/s] 95%|█████████▍| 91/96 [00:03<00:00, 26.49it/s] 98%|█████████▊| 94/96 [00:03<00:00, 26.49it/s] 100%|██████████| 96/96 [00:03<00:00, 27.22it/s] IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (1544, 516) to (1552, 528) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). [swscaler @ 0x5baa740] Warning: data is not aligned! This can lead to a speed loss 100%|██████████| 1/1 [01:48<00:00, 108.06s/it] 100%|██████████| 1/1 [01:48<00:00, 108.06s/it] /src/magic-animate/samples/config-2023-12-05T21-07-44/videos/tmp3x9__2mjmeme_tmp8yfgquj_multi_dancing.mp4
Want to make some of these yourself?
Run this model
