Readme
Mostly a wrapper around ipiv’s beautiful video morphing ComfyUI workflow: https://civitai.com/models/372584/ipivs-morph-img2vid-animatediff-lcm
You can also add a style to the entire video.
fofr
/
video-morpher
Generate a video that morphs between subjects, with an optional style
Input
Run this model in Node.js with one line of code:
npx create-replicate --model=fofr/video-morpher
or set up a project from scratch
Install Replicate’s Node.js client library:
npm install replicateSet the
REPLICATE_API_TOKEN environment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Import and set up the client:
import Replicate from "replicate";
const replicate = new Replicate({
auth: process.env.REPLICATE_API_TOKEN,
});Run fofr/video-morpher using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
const output = await replicate.run(
"fofr/video-morpher:e70e975067d2b5dbe9e2d9022833d27230a1bdeb3f4af6fe6bb49a548a3039a7",
{
input: {
mode: "upscaled-and-interpolated",
prompt: "bright, vibrant, high contrast",
checkpoint: "3D",
style_image: "https://replicate.delivery/pbxt/KnxLVfe6BRRu2zHc3gT99mnwaemKfR4JzaZWxMCLsZYSTKzp/2024-03-05--06-47-29-u-q1-fofr_tropical_purple_beksinski_aaad09f0-d194-4e40-b312-51054fc4ebbf.png",
aspect_ratio: "3:4",
style_strength: 0.5,
use_controlnet: true,
negative_prompt: "dark, gloomy",
subject_image_1: "https://replicate.delivery/pbxt/KnxLX9qnA82YKCkScCJZf5VCX6hy5RuprGwLEAVjp3vu6Oh1/1.webp",
subject_image_2: "https://replicate.delivery/pbxt/KnxLWqPg086DnRTUvxDM7gCBYi3W3coIbn3Q8jXnTJsfvZzt/2.webp",
subject_image_3: "https://replicate.delivery/pbxt/KnxLWU6aHjc6kVW2QOF0xC4oen8zuPnlsK2o24GtOV2bJEn8/4.webp",
subject_image_4: "https://replicate.delivery/pbxt/KnxLVg54ySFfg8s78YyfzK0Qgse8YdTbSmEZKpnkFkmSFlsf/4.webp"
}
}
);
// To access the file URL:
console.log(output[0].url()); //=> "http://example.com"
// To write the file to disk:
fs.writeFile("my-image.png", output[0]);To learn more, take a look at the guide on getting started with Node.js.
Install Replicate’s Python client library:
pip install replicateSet the
REPLICATE_API_TOKEN environment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Import the client:
import replicateRun fofr/video-morpher using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
output = replicate.run(
"fofr/video-morpher:e70e975067d2b5dbe9e2d9022833d27230a1bdeb3f4af6fe6bb49a548a3039a7",
input={
"mode": "upscaled-and-interpolated",
"prompt": "bright, vibrant, high contrast",
"checkpoint": "3D",
"style_image": "https://replicate.delivery/pbxt/KnxLVfe6BRRu2zHc3gT99mnwaemKfR4JzaZWxMCLsZYSTKzp/2024-03-05--06-47-29-u-q1-fofr_tropical_purple_beksinski_aaad09f0-d194-4e40-b312-51054fc4ebbf.png",
"aspect_ratio": "3:4",
"style_strength": 0.5,
"use_controlnet": True,
"negative_prompt": "dark, gloomy",
"subject_image_1": "https://replicate.delivery/pbxt/KnxLX9qnA82YKCkScCJZf5VCX6hy5RuprGwLEAVjp3vu6Oh1/1.webp",
"subject_image_2": "https://replicate.delivery/pbxt/KnxLWqPg086DnRTUvxDM7gCBYi3W3coIbn3Q8jXnTJsfvZzt/2.webp",
"subject_image_3": "https://replicate.delivery/pbxt/KnxLWU6aHjc6kVW2QOF0xC4oen8zuPnlsK2o24GtOV2bJEn8/4.webp",
"subject_image_4": "https://replicate.delivery/pbxt/KnxLVg54ySFfg8s78YyfzK0Qgse8YdTbSmEZKpnkFkmSFlsf/4.webp"
}
)
print(output)To learn more, take a look at the guide on getting started with Python.
Set the
REPLICATE_API_TOKEN environment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Run fofr/video-morpher using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
curl -s -X POST \
-H "Authorization: Bearer $REPLICATE_API_TOKEN" \
-H "Content-Type: application/json" \
-H "Prefer: wait" \
-d $'{
"version": "e70e975067d2b5dbe9e2d9022833d27230a1bdeb3f4af6fe6bb49a548a3039a7",
"input": {
"mode": "upscaled-and-interpolated",
"prompt": "bright, vibrant, high contrast",
"checkpoint": "3D",
"style_image": "https://replicate.delivery/pbxt/KnxLVfe6BRRu2zHc3gT99mnwaemKfR4JzaZWxMCLsZYSTKzp/2024-03-05--06-47-29-u-q1-fofr_tropical_purple_beksinski_aaad09f0-d194-4e40-b312-51054fc4ebbf.png",
"aspect_ratio": "3:4",
"style_strength": 0.5,
"use_controlnet": true,
"negative_prompt": "dark, gloomy",
"subject_image_1": "https://replicate.delivery/pbxt/KnxLX9qnA82YKCkScCJZf5VCX6hy5RuprGwLEAVjp3vu6Oh1/1.webp",
"subject_image_2": "https://replicate.delivery/pbxt/KnxLWqPg086DnRTUvxDM7gCBYi3W3coIbn3Q8jXnTJsfvZzt/2.webp",
"subject_image_3": "https://replicate.delivery/pbxt/KnxLWU6aHjc6kVW2QOF0xC4oen8zuPnlsK2o24GtOV2bJEn8/4.webp",
"subject_image_4": "https://replicate.delivery/pbxt/KnxLVg54ySFfg8s78YyfzK0Qgse8YdTbSmEZKpnkFkmSFlsf/4.webp"
}
}' \
https://api.replicate.com/v1/predictionsTo learn more, take a look at Replicate’s HTTP API reference docs.
Add a payment method to run this model.
Sign in with GitHub
By signing in, you agree to our
terms of service and privacy policy
Output
{
"completed_at": "2024-11-28T11:08:35.822944Z",
"created_at": "2024-11-28T11:05:33.859000Z",
"data_removed": false,
"error": null,
"id": "ykjrhncvmdrm80cke9s9rhsqc4",
"input": {
"mode": "upscaled-and-interpolated",
"prompt": "bright, vibrant, high contrast",
"checkpoint": "3D",
"style_image": "https://replicate.delivery/pbxt/KnxLVfe6BRRu2zHc3gT99mnwaemKfR4JzaZWxMCLsZYSTKzp/2024-03-05--06-47-29-u-q1-fofr_tropical_purple_beksinski_aaad09f0-d194-4e40-b312-51054fc4ebbf.png",
"aspect_ratio": "3:4",
"style_strength": 0.5,
"use_controlnet": true,
"negative_prompt": "dark, gloomy",
"subject_image_1": "https://replicate.delivery/pbxt/KnxLX9qnA82YKCkScCJZf5VCX6hy5RuprGwLEAVjp3vu6Oh1/1.webp",
"subject_image_2": "https://replicate.delivery/pbxt/KnxLWqPg086DnRTUvxDM7gCBYi3W3coIbn3Q8jXnTJsfvZzt/2.webp",
"subject_image_3": "https://replicate.delivery/pbxt/KnxLWU6aHjc6kVW2QOF0xC4oen8zuPnlsK2o24GtOV2bJEn8/4.webp",
"subject_image_4": "https://replicate.delivery/pbxt/KnxLVg54ySFfg8s78YyfzK0Qgse8YdTbSmEZKpnkFkmSFlsf/4.webp"
},
"logs": "Random seed set to: 4049182242\nChecking inputs\n✅ /tmp/inputs/2.png\n✅ /tmp/inputs/1.png\n✅ /tmp/inputs/3.png\n✅ /tmp/inputs/4.png\n✅ /tmp/inputs/circles.mp4\n✅ /tmp/inputs/style.png\n====================================\nChecking weights\nIncluding weights for IPAdapter preset: PLUS (high strength)\n✅ RealESRGAN_x4.pth\n✅ film_net_fp32.pt\n✅ ip-adapter-plus_sdxl_vit-h.safetensors\n✅ CLIP-ViT-H-14-laion2B-s32B-b79K.safetensors\n✅ AnimateLCM_sd15_t2v.ckpt\n✅ control_v1p_sd15_qrcode_monster.safetensors\n✅ vae-ft-mse-840000-ema-pruned.safetensors\n✅ ip-adapter-plus_sd15.safetensors\n✅ AnimateLCM_sd15_t2v_lora.safetensors\n✅ rcnzCartoon3d_v20.safetensors\n====================================\nRunning workflow\ngot prompt\nExecuting node 746, title: Load Video (Upload) 🎥🅥🅗🅢, class type: VHS_LoadVideo\nExecuting node 461, title: 🔧 Simple Math, class type: SimpleMath+\nExecuting node 454, title: RepeatImageBatch, class type: RepeatImageBatch\nExecuting node 458, title: Split Image Batch 🎥🅥🅗🅢, class type: VHS_SplitImages\nExecuting node 125, title: Apply Advanced ControlNet 🛂🅐🅒🅝, class type: ACN_AdvancedControlNetApply\nExecuting node 80, title: KSampler, class type: KSampler\n[AnimateDiffEvo] - \u001b[0;32mINFO\u001b[0m - Sliding context window activated - latents passed in (96) greater than context_length 16.\n[AnimateDiffEvo] - \u001b[0;32mINFO\u001b[0m - Using motion module AnimateLCM_sd15_t2v.ckpt:v2.\nRequested to load ControlNet\nLoading 1 new model\n 0%| | 0/11 [00:00<?, ?it/s]\n 9%|▉ | 1/11 [00:01<00:11, 1.11s/it]\n 18%|█▊ | 2/11 [00:02<00:09, 1.11s/it]\n 27%|██▋ | 3/11 [00:03<00:09, 1.18s/it]\n 36%|███▋ | 4/11 [00:04<00:08, 1.21s/it]\n 45%|████▌ | 5/11 [00:05<00:06, 1.14s/it]\n 55%|█████▍ | 6/11 [00:06<00:05, 1.11s/it]\n 64%|██████▎ | 7/11 [00:07<00:04, 1.08s/it]\n 73%|███████▎ | 8/11 [00:09<00:03, 1.13s/it]\n 82%|████████▏ | 9/11 [00:10<00:02, 1.12s/it]\n 91%|█████████ | 10/11 [00:11<00:01, 1.09s/it]\n100%|██████████| 11/11 [00:12<00:00, 1.07s/it]\n100%|██████████| 11/11 [00:12<00:00, 1.11s/it]\nExecuting node 84, title: VAE Decode, class type: VAEDecode\nExecuting node 53, title: Video Combine 🎥🅥🅗🅢, class type: VHS_VideoCombine\nExecuting node 203, title: Upscale Image By, class type: ImageScaleBy\nExecuting node 204, title: VAE Encode, class type: VAEEncode\nExecuting node 198, title: KSampler, class type: KSampler\n[AnimateDiffEvo] - \u001b[0;32mINFO\u001b[0m - Sliding context window activated - latents passed in (96) greater than context_length 16.\n[AnimateDiffEvo] - \u001b[0;32mINFO\u001b[0m - Using motion module AnimateLCM_sd15_t2v.ckpt:v2.\n 0%| | 0/11 [00:00<?, ?it/s]\n 9%|▉ | 1/11 [00:02<00:21, 2.20s/it]\n 18%|█▊ | 2/11 [00:04<00:19, 2.19s/it]\n 27%|██▋ | 3/11 [00:06<00:17, 2.19s/it]\n 36%|███▋ | 4/11 [00:08<00:15, 2.19s/it]\n 45%|████▌ | 5/11 [00:10<00:13, 2.19s/it]\n 55%|█████▍ | 6/11 [00:13<00:10, 2.19s/it]\n 64%|██████▎ | 7/11 [00:15<00:08, 2.19s/it]\n 73%|███████▎ | 8/11 [00:17<00:06, 2.19s/it]\n 82%|████████▏ | 9/11 [00:19<00:04, 2.18s/it]\n 91%|█████████ | 10/11 [00:21<00:02, 2.18s/it]\n100%|██████████| 11/11 [00:24<00:00, 2.18s/it]\n100%|██████████| 11/11 [00:24<00:00, 2.19s/it]\nRequested to load AutoencoderKL\nLoading 1 new model\nExecuting node 201, title: VAE Decode, class type: VAEDecode\nExecuting node 205, title: Video Combine 🎥🅥🅗🅢, class type: VHS_VideoCombine\nExecuting node 271, title: Upscale Image (using Model), class type: ImageUpscaleWithModel\nExecuting node 279, title: Upscale Image, class type: ImageScale\nExecuting node 272, title: Video Combine 🎥🅥🅗🅢, class type: VHS_VideoCombine\nExecuting node 770, title: FILM VFI, class type: FILM VFI\nComfy-VFI: Clearing cache...\nComfy-VFI: Done cache clearing\nComfy-VFI: Clearing cache...\nComfy-VFI: Done cache clearing\nComfy-VFI: Clearing cache...\nComfy-VFI: Done cache clearing\nComfy-VFI: Clearing cache...\nComfy-VFI: Done cache clearing\nComfy-VFI: Clearing cache...\nComfy-VFI: Done cache clearing\nComfy-VFI: Clearing cache...\nComfy-VFI: Done cache clearing\nComfy-VFI: Clearing cache...\nComfy-VFI: Done cache clearing\nComfy-VFI: Clearing cache...\nComfy-VFI: Done cache clearing\nComfy-VFI: Clearing cache...\nComfy-VFI: Done cache clearing\nComfy-VFI: Final clearing cache...\nComfy-VFI: Done cache clearing\nExecuting node 219, title: Video Combine 🎥🅥🅗🅢, class type: VHS_VideoCombine\nPrompt executed in 176.75 seconds\noutputs: {'53': {'gifs': [{'filename': 'preview_00001.mp4', 'subfolder': '', 'type': 'output', 'format': 'video/h264-mp4'}]}, '205': {'gifs': [{'filename': 'upscaled_00001.mp4', 'subfolder': '', 'type': 'output', 'format': 'video/h264-mp4'}]}, '272': {'gifs': [{'filename': 'upscaled_model_00001.mp4', 'subfolder': '', 'type': 'output', 'format': 'video/h264-mp4'}]}, '219': {'gifs': [{'filename': 'interpolated_00001.mp4', 'subfolder': '', 'type': 'output', 'format': 'video/h264-mp4'}]}}\n====================================\npreview_00001.mp4\nupscaled_00001.mp4\nupscaled_model_00001.mp4\ninterpolated_00001.mp4",
"metrics": {
"predict_time": 181.956443524,
"total_time": 181.963944
},
"output": [
"https://replicate.delivery/xezq/fqOdvvfinVpSLUUMepVqdywzaSMyP2ru3DE33XUAAetPTAWPB/preview_00001.mp4",
"https://replicate.delivery/xezq/fFvJWDTii62RfEK5ExHpy88YYFFVuvYUoR2sqBtcBb0zEg1TA/upscaled_00001.mp4",
"https://replicate.delivery/xezq/oKSuEYjOC2rFF99foM6NVy0e1053GKPZn1AarSgHrAtzEg1TA/upscaled_model_00001.mp4",
"https://replicate.delivery/xezq/sZQRsofi4U2Yf0x5wfrwJzRr4IipUs8tuixRqdIRJM3mJArnA/interpolated_00001.mp4"
],
"started_at": "2024-11-28T11:05:33.866501Z",
"status": "succeeded",
"urls": {
"stream": "https://stream.replicate.com/v1/files/bcwr-tyikq3iqnxxvcbm44lu25xmm75evlu5c47aadad5o23zloamp4ka",
"get": "https://api.replicate.com/v1/predictions/ykjrhncvmdrm80cke9s9rhsqc4",
"cancel": "https://api.replicate.com/v1/predictions/ykjrhncvmdrm80cke9s9rhsqc4/cancel"
},
"version": "e70e975067d2b5dbe9e2d9022833d27230a1bdeb3f4af6fe6bb49a548a3039a7"
}Generated in
Random seed set to: 4049182242
Checking inputs
✅ /tmp/inputs/2.png
✅ /tmp/inputs/1.png
✅ /tmp/inputs/3.png
✅ /tmp/inputs/4.png
✅ /tmp/inputs/circles.mp4
✅ /tmp/inputs/style.png
====================================
Checking weights
Including weights for IPAdapter preset: PLUS (high strength)
✅ RealESRGAN_x4.pth
✅ film_net_fp32.pt
✅ ip-adapter-plus_sdxl_vit-h.safetensors
✅ CLIP-ViT-H-14-laion2B-s32B-b79K.safetensors
✅ AnimateLCM_sd15_t2v.ckpt
✅ control_v1p_sd15_qrcode_monster.safetensors
✅ vae-ft-mse-840000-ema-pruned.safetensors
✅ ip-adapter-plus_sd15.safetensors
✅ AnimateLCM_sd15_t2v_lora.safetensors
✅ rcnzCartoon3d_v20.safetensors
====================================
Running workflow
got prompt
Executing node 746, title: Load Video (Upload) 🎥🅥🅗🅢, class type: VHS_LoadVideo
Executing node 461, title: 🔧 Simple Math, class type: SimpleMath+
Executing node 454, title: RepeatImageBatch, class type: RepeatImageBatch
Executing node 458, title: Split Image Batch 🎥🅥🅗🅢, class type: VHS_SplitImages
Executing node 125, title: Apply Advanced ControlNet 🛂🅐🅒🅝, class type: ACN_AdvancedControlNetApply
Executing node 80, title: KSampler, class type: KSampler
[AnimateDiffEvo] - INFO - Sliding context window activated - latents passed in (96) greater than context_length 16.
[AnimateDiffEvo] - INFO - Using motion module AnimateLCM_sd15_t2v.ckpt:v2.
Requested to load ControlNet
Loading 1 new model
0%| | 0/11 [00:00<?, ?it/s]
9%|▉ | 1/11 [00:01<00:11, 1.11s/it]
18%|█▊ | 2/11 [00:02<00:09, 1.11s/it]
27%|██▋ | 3/11 [00:03<00:09, 1.18s/it]
36%|███▋ | 4/11 [00:04<00:08, 1.21s/it]
45%|████▌ | 5/11 [00:05<00:06, 1.14s/it]
55%|█████▍ | 6/11 [00:06<00:05, 1.11s/it]
64%|██████▎ | 7/11 [00:07<00:04, 1.08s/it]
73%|███████▎ | 8/11 [00:09<00:03, 1.13s/it]
82%|████████▏ | 9/11 [00:10<00:02, 1.12s/it]
91%|█████████ | 10/11 [00:11<00:01, 1.09s/it]
100%|██████████| 11/11 [00:12<00:00, 1.07s/it]
100%|██████████| 11/11 [00:12<00:00, 1.11s/it]
Executing node 84, title: VAE Decode, class type: VAEDecode
Executing node 53, title: Video Combine 🎥🅥🅗🅢, class type: VHS_VideoCombine
Executing node 203, title: Upscale Image By, class type: ImageScaleBy
Executing node 204, title: VAE Encode, class type: VAEEncode
Executing node 198, title: KSampler, class type: KSampler
[AnimateDiffEvo] - INFO - Sliding context window activated - latents passed in (96) greater than context_length 16.
[AnimateDiffEvo] - INFO - Using motion module AnimateLCM_sd15_t2v.ckpt:v2.
0%| | 0/11 [00:00<?, ?it/s]
9%|▉ | 1/11 [00:02<00:21, 2.20s/it]
18%|█▊ | 2/11 [00:04<00:19, 2.19s/it]
27%|██▋ | 3/11 [00:06<00:17, 2.19s/it]
36%|███▋ | 4/11 [00:08<00:15, 2.19s/it]
45%|████▌ | 5/11 [00:10<00:13, 2.19s/it]
55%|█████▍ | 6/11 [00:13<00:10, 2.19s/it]
64%|██████▎ | 7/11 [00:15<00:08, 2.19s/it]
73%|███████▎ | 8/11 [00:17<00:06, 2.19s/it]
82%|████████▏ | 9/11 [00:19<00:04, 2.18s/it]
91%|█████████ | 10/11 [00:21<00:02, 2.18s/it]
100%|██████████| 11/11 [00:24<00:00, 2.18s/it]
100%|██████████| 11/11 [00:24<00:00, 2.19s/it]
Requested to load AutoencoderKL
Loading 1 new model
Executing node 201, title: VAE Decode, class type: VAEDecode
Executing node 205, title: Video Combine 🎥🅥🅗🅢, class type: VHS_VideoCombine
Executing node 271, title: Upscale Image (using Model), class type: ImageUpscaleWithModel
Executing node 279, title: Upscale Image, class type: ImageScale
Executing node 272, title: Video Combine 🎥🅥🅗🅢, class type: VHS_VideoCombine
Executing node 770, title: FILM VFI, class type: FILM VFI
Comfy-VFI: Clearing cache...
Comfy-VFI: Done cache clearing
Comfy-VFI: Clearing cache...
Comfy-VFI: Done cache clearing
Comfy-VFI: Clearing cache...
Comfy-VFI: Done cache clearing
Comfy-VFI: Clearing cache...
Comfy-VFI: Done cache clearing
Comfy-VFI: Clearing cache...
Comfy-VFI: Done cache clearing
Comfy-VFI: Clearing cache...
Comfy-VFI: Done cache clearing
Comfy-VFI: Clearing cache...
Comfy-VFI: Done cache clearing
Comfy-VFI: Clearing cache...
Comfy-VFI: Done cache clearing
Comfy-VFI: Clearing cache...
Comfy-VFI: Done cache clearing
Comfy-VFI: Final clearing cache...
Comfy-VFI: Done cache clearing
Executing node 219, title: Video Combine 🎥🅥🅗🅢, class type: VHS_VideoCombine
Prompt executed in 176.75 seconds
outputs: {'53': {'gifs': [{'filename': 'preview_00001.mp4', 'subfolder': '', 'type': 'output', 'format': 'video/h264-mp4'}]}, '205': {'gifs': [{'filename': 'upscaled_00001.mp4', 'subfolder': '', 'type': 'output', 'format': 'video/h264-mp4'}]}, '272': {'gifs': [{'filename': 'upscaled_model_00001.mp4', 'subfolder': '', 'type': 'output', 'format': 'video/h264-mp4'}]}, '219': {'gifs': [{'filename': 'interpolated_00001.mp4', 'subfolder': '', 'type': 'output', 'format': 'video/h264-mp4'}]}}
====================================
preview_00001.mp4
upscaled_00001.mp4
upscaled_model_00001.mp4
interpolated_00001.mp4Examples
Run time and cost
This model costs approximately $0.12 to run on Replicate, or 8 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.
This model runs on Nvidia L40S GPU hardware. Predictions typically complete within 128 seconds. The predict time for this model varies significantly based on the inputs.