cjwbw / lavie

High-Quality Video Generation with Cascaded Latent Diffusion Models

Cold

Public
13.9K runs
A100 (80GB)
GitHub
Paper
License

Run with an API

Playground API Examples README Versions

Input

prompt

string

Shift + Return to add a new line

a Corgi walking in the park at sunrise, oil painting stylea Corgi walking in the park at sunrise, oil painting style

Prompt for video generation.

Default: "a Corgi walking in the park at sunrise, oil painting style"

sample_method

string

Choose a scheduler for sampling base video output.

Default: "ddpm"

width

integer

Width of output video.

Default: 512

height

integer

Height of output video

Default: 320

num_inference_steps

integer

Number of denoising steps

Default: 250

guidance_scale

number

Scale for classifier-free guidance

Default: 7

quality

integer

(minimum: 0, maximum: 10)

Quality of the output vide0

Default: 9

seed

integer

Random seed. Leave blank to randomize the seed

interpolation

boolean

Default output has 16 frames. Set interpolation to True to get 61 frames output.

Default: false

super_resolution

boolean

Super resolution 4x when set to True.

Default: false

video_fps

integer

Number of frames per second in the output video

Default: 8

Run this model in Node.js with one line of code:

npx create-replicate --model=cjwbw/lavie

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run cjwbw/lavie using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "cjwbw/lavie:0bca850c4928b6c30052541fa002f24cbb4b677259c461dd041d271ba9d3c517",
  {
    input: {
      width: 512,
      height: 320,
      prompt: "a Corgi walking in the park at sunrise, oil painting style",
      quality: 9,
      video_fps: 8,
      interpolation: false,
      sample_method: "ddpm",
      guidance_scale: 7,
      super_resolution: false,
      num_inference_steps: 50
    }
  }
);

// To access the file URL:
console.log(output.url()); //=> "http://example.com"

// To write the file to disk:
fs.writeFile("my-image.png", output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run cjwbw/lavie using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "cjwbw/lavie:0bca850c4928b6c30052541fa002f24cbb4b677259c461dd041d271ba9d3c517",
    input={
        "width": 512,
        "height": 320,
        "prompt": "a Corgi walking in the park at sunrise, oil painting style",
        "quality": 9,
        "video_fps": 8,
        "interpolation": False,
        "sample_method": "ddpm",
        "guidance_scale": 7,
        "super_resolution": False,
        "num_inference_steps": 50
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run cjwbw/lavie using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "cjwbw/lavie:0bca850c4928b6c30052541fa002f24cbb4b677259c461dd041d271ba9d3c517",
    "input": {
      "width": 512,
      "height": 320,
      "prompt": "a Corgi walking in the park at sunrise, oil painting style",
      "quality": 9,
      "video_fps": 8,
      "interpolation": false,
      "sample_method": "ddpm",
      "guidance_scale": 7,
      "super_resolution": false,
      "num_inference_steps": 50
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

{
  "completed_at": "2023-11-21T18:23:19.760824Z",
  "created_at": "2023-11-21T18:22:22.662591Z",
  "data_removed": false,
  "error": null,
  "id": "i2hg7h3bjjhlidpso4hozwlp4u",
  "input": {
    "width": 512,
    "height": 320,
    "prompt": "a Corgi walking in the park at sunrise, oil painting style",
    "quality": 9,
    "video_fps": 8,
    "interpolation": false,
    "sample_method": "ddpm",
    "guidance_scale": 7,
    "super_resolution": false,
    "num_inference_steps": 50
  },
  "logs": "Using seed: 27373\n  0%|          | 0/50 [00:00<?, ?it/s]\n  2%|▏         | 1/50 [00:00<00:29,  1.64it/s]\n  4%|▍         | 2/50 [00:00<00:17,  2.68it/s]\n  6%|▌         | 3/50 [00:01<00:13,  3.36it/s]\n  8%|▊         | 4/50 [00:01<00:12,  3.82it/s]\n 10%|█         | 5/50 [00:01<00:10,  4.12it/s]\n 12%|█▏        | 6/50 [00:01<00:10,  4.34it/s]\n 14%|█▍        | 7/50 [00:01<00:09,  4.49it/s]\n 16%|█▌        | 8/50 [00:02<00:09,  4.59it/s]\n 18%|█▊        | 9/50 [00:02<00:08,  4.66it/s]\n 20%|██        | 10/50 [00:02<00:08,  4.70it/s]\n 22%|██▏       | 11/50 [00:02<00:08,  4.73it/s]\n 24%|██▍       | 12/50 [00:02<00:07,  4.75it/s]\n 26%|██▌       | 13/50 [00:03<00:07,  4.77it/s]\n 28%|██▊       | 14/50 [00:03<00:07,  4.79it/s]\n 30%|███       | 15/50 [00:03<00:07,  4.79it/s]\n 32%|███▏      | 16/50 [00:03<00:07,  4.80it/s]\n 34%|███▍      | 17/50 [00:03<00:06,  4.81it/s]\n 36%|███▌      | 18/50 [00:04<00:06,  4.80it/s]\n 38%|███▊      | 19/50 [00:04<00:06,  4.81it/s]\n 40%|████      | 20/50 [00:04<00:06,  4.80it/s]\n 42%|████▏     | 21/50 [00:04<00:06,  4.80it/s]\n 44%|████▍     | 22/50 [00:04<00:05,  4.80it/s]\n 46%|████▌     | 23/50 [00:05<00:05,  4.80it/s]\n 48%|████▊     | 24/50 [00:05<00:05,  4.81it/s]\n 50%|█████     | 25/50 [00:05<00:05,  4.81it/s]\n 52%|█████▏    | 26/50 [00:05<00:04,  4.80it/s]\n 54%|█████▍    | 27/50 [00:06<00:04,  4.80it/s]\n 56%|█████▌    | 28/50 [00:06<00:04,  4.80it/s]\n 58%|█████▊    | 29/50 [00:06<00:04,  4.80it/s]\n 60%|██████    | 30/50 [00:06<00:04,  4.80it/s]\n 62%|██████▏   | 31/50 [00:06<00:03,  4.81it/s]\n 64%|██████▍   | 32/50 [00:07<00:03,  4.80it/s]\n 66%|██████▌   | 33/50 [00:07<00:03,  4.80it/s]\n 68%|██████▊   | 34/50 [00:07<00:03,  4.81it/s]\n 70%|███████   | 35/50 [00:07<00:03,  4.80it/s]\n 72%|███████▏  | 36/50 [00:07<00:02,  4.80it/s]\n 74%|███████▍  | 37/50 [00:08<00:02,  4.79it/s]\n 76%|███████▌  | 38/50 [00:08<00:02,  4.79it/s]\n 78%|███████▊  | 39/50 [00:08<00:02,  4.79it/s]\n 80%|████████  | 40/50 [00:08<00:02,  4.79it/s]\n 82%|████████▏ | 41/50 [00:08<00:01,  4.80it/s]\n 84%|████████▍ | 42/50 [00:09<00:01,  4.78it/s]\n 86%|████████▌ | 43/50 [00:09<00:01,  4.79it/s]\n 88%|████████▊ | 44/50 [00:09<00:01,  4.79it/s]\n 90%|█████████ | 45/50 [00:09<00:01,  4.78it/s]\n 92%|█████████▏| 46/50 [00:09<00:00,  4.79it/s]\n 94%|█████████▍| 47/50 [00:10<00:00,  4.79it/s]\n 96%|█████████▌| 48/50 [00:10<00:00,  4.78it/s]\n 98%|█████████▊| 49/50 [00:10<00:00,  4.79it/s]\n100%|██████████| 50/50 [00:10<00:00,  4.80it/s]\n100%|██████████| 50/50 [00:11<00:00,  4.50it/s]",
  "metrics": {
    "predict_time": 12.403431,
    "total_time": 57.098233
  },
  "output": "https://replicate.delivery/pbxt/RXCtleF9AJTea0LIS32LREtQ2uWHxirA07X5LhENFqIXeU1jA/out.mp4",
  "started_at": "2023-11-21T18:23:07.357393Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/i2hg7h3bjjhlidpso4hozwlp4u",
    "cancel": "https://api.replicate.com/v1/predictions/i2hg7h3bjjhlidpso4hozwlp4u/cancel"
  },
  "version": "0bca850c4928b6c30052541fa002f24cbb4b677259c461dd041d271ba9d3c517"
}

Generated in

12.4 seconds

Tweak itReport

Using seed: 27373
  0%|          | 0/50 [00:00<?, ?it/s]
  2%|▏         | 1/50 [00:00<00:29,  1.64it/s]
  4%|▍         | 2/50 [00:00<00:17,  2.68it/s]
  6%|▌         | 3/50 [00:01<00:13,  3.36it/s]
  8%|▊         | 4/50 [00:01<00:12,  3.82it/s]
 10%|█         | 5/50 [00:01<00:10,  4.12it/s]
 12%|█▏        | 6/50 [00:01<00:10,  4.34it/s]
 14%|█▍        | 7/50 [00:01<00:09,  4.49it/s]
 16%|█▌        | 8/50 [00:02<00:09,  4.59it/s]
 18%|█▊        | 9/50 [00:02<00:08,  4.66it/s]
 20%|██        | 10/50 [00:02<00:08,  4.70it/s]
 22%|██▏       | 11/50 [00:02<00:08,  4.73it/s]
 24%|██▍       | 12/50 [00:02<00:07,  4.75it/s]
 26%|██▌       | 13/50 [00:03<00:07,  4.77it/s]
 28%|██▊       | 14/50 [00:03<00:07,  4.79it/s]
 30%|███       | 15/50 [00:03<00:07,  4.79it/s]
 32%|███▏      | 16/50 [00:03<00:07,  4.80it/s]
 34%|███▍      | 17/50 [00:03<00:06,  4.81it/s]
 36%|███▌      | 18/50 [00:04<00:06,  4.80it/s]
 38%|███▊      | 19/50 [00:04<00:06,  4.81it/s]
 40%|████      | 20/50 [00:04<00:06,  4.80it/s]
 42%|████▏     | 21/50 [00:04<00:06,  4.80it/s]
 44%|████▍     | 22/50 [00:04<00:05,  4.80it/s]
 46%|████▌     | 23/50 [00:05<00:05,  4.80it/s]
 48%|████▊     | 24/50 [00:05<00:05,  4.81it/s]
 50%|█████     | 25/50 [00:05<00:05,  4.81it/s]
 52%|█████▏    | 26/50 [00:05<00:04,  4.80it/s]
 54%|█████▍    | 27/50 [00:06<00:04,  4.80it/s]
 56%|█████▌    | 28/50 [00:06<00:04,  4.80it/s]
 58%|█████▊    | 29/50 [00:06<00:04,  4.80it/s]
 60%|██████    | 30/50 [00:06<00:04,  4.80it/s]
 62%|██████▏   | 31/50 [00:06<00:03,  4.81it/s]
 64%|██████▍   | 32/50 [00:07<00:03,  4.80it/s]
 66%|██████▌   | 33/50 [00:07<00:03,  4.80it/s]
 68%|██████▊   | 34/50 [00:07<00:03,  4.81it/s]
 70%|███████   | 35/50 [00:07<00:03,  4.80it/s]
 72%|███████▏  | 36/50 [00:07<00:02,  4.80it/s]
 74%|███████▍  | 37/50 [00:08<00:02,  4.79it/s]
 76%|███████▌  | 38/50 [00:08<00:02,  4.79it/s]
 78%|███████▊  | 39/50 [00:08<00:02,  4.79it/s]
 80%|████████  | 40/50 [00:08<00:02,  4.79it/s]
 82%|████████▏ | 41/50 [00:08<00:01,  4.80it/s]
 84%|████████▍ | 42/50 [00:09<00:01,  4.78it/s]
 86%|████████▌ | 43/50 [00:09<00:01,  4.79it/s]
 88%|████████▊ | 44/50 [00:09<00:01,  4.79it/s]
 90%|█████████ | 45/50 [00:09<00:01,  4.78it/s]
 92%|█████████▏| 46/50 [00:09<00:00,  4.79it/s]
 94%|█████████▍| 47/50 [00:10<00:00,  4.79it/s]
 96%|█████████▌| 48/50 [00:10<00:00,  4.78it/s]
 98%|█████████▊| 49/50 [00:10<00:00,  4.79it/s]
100%|██████████| 50/50 [00:10<00:00,  4.80it/s]
100%|██████████| 50/50 [00:11<00:00,  4.50it/s]

Examples

View more examples

Run time and cost

This model costs approximately $0.015 to run on Replicate, or 66 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia A100 (80GB) GPU hardware. Predictions typically complete within 11 seconds.

Readme

LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models

LaVie is a Text-to-Video (T2V) generation framework, and main part of video generation system Vchitect.

BibTex

@article{wang2023lavie,
  title={LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models},
  author={Wang, Yaohui and Chen, Xinyuan and Ma, Xin and Zhou, Shangchen and Huang, Ziqi and Wang, Yi and Yang, Ceyuan and He, Yinan and Yu, Jiashuo and Yang, Peiqing and others},
  journal={arXiv preprint arXiv:2309.15103},
  year={2023}
}

Disclaimer

We disclaim responsibility for user-generated content. The model was not trained to realistically represent people or events, so using it to generate such content is beyond the model’s capabilities. It is prohibited for pornographic, violent and bloody content generation, and to generate content that is demeaning or harmful to people or their environment, culture, religion, etc. Users are solely liable for their actions. The project contributors are not legally affiliated with, nor accountable for users’ behaviors. Use the generative model responsibly, adhering to ethical and legal standards.

Contact Us

Yaohui Wang: wangyaohui@pjlab.org.cn
Xinyuan Chen: chenxinyuan@pjlab.org.cn
Xin Ma: xin.ma1@monash.edu

Acknowledgements

The code is built upon diffusers and Stable Diffusion, we thank all the contributors for open-sourcing.

License

The code is licensed under Apache-2.0, model weights are fully open for academic research and also allow free commercial usage. To apply for a commercial license, please contact vchitect@pjlab.org.cn.