andreasjansson / stable-diffusion-animation

Animate Stable Diffusion by interpolating between two prompts

Cold

Public
118.6K runs
A100 (80GB)
GitHub
License

Run with an API

Playground API Examples README Versions

Input

prompt_start

*string

Shift + Return to add a new line

tall rectangular black monolith, monkeys in the desert looking at a large tall monolith, a detailed matte painting by Wes Anderson, behance, light and space, reimagined by industrial light and magic, matte painting, criterion collectiontall rectangular black monolith, monkeys in the desert looking at a large tall monolith, a detailed matte painting by Wes Anderson, behance, light and space, reimagined by industrial light and magic, matte painting, criterion collection

Prompt to start the animation with

prompt_end

*string

Shift + Return to add a new line

tall rectangular black monolith, a white room in the future with a bed, victorian details and a tall black monolith, a detailed matte painting by Wes Anderson, behance, light and space, reimagined by industrial light and magic, matte painting, criterion collectiontall rectangular black monolith, a white room in the future with a bed, victorian details and a tall black monolith, a detailed matte painting by Wes Anderson, behance, light and space, reimagined by industrial light and magic, matte painting, criterion collection

Prompt to end the animation with. You can include multiple prompts by separating the prompts with | (the 'pipe' character)

width

integer

Width of output image

Default: 512

height

integer

Height of output image

Default: 512

num_inference_steps

integer

(minimum: 1, maximum: 500)

Number of denoising steps

Default: 50

prompt_strength

number

Lower prompt strength generates more coherent gifs, higher respects prompts more but can be jumpy

Default: 0.8

num_animation_frames

integer

(minimum: 2, maximum: 50)

Number of frames to animate

Default: 10

num_interpolation_steps

integer

(minimum: 1, maximum: 50)

Number of steps to interpolate between animation frames

Default: 5

guidance_scale

number

(minimum: 1, maximum: 20)

Scale for classifier-free guidance

Default: 7.5

gif_frames_per_second

integer

(minimum: 1, maximum: 50)

Frames/second in output GIF

Default: 20

gif_ping_pong

boolean

Whether to reverse the animation and go back to the beginning before looping

Default: false

film_interpolation

boolean

Whether to use FILM for between-frame interpolation (film-net.github.io)

Default: true

intermediate_output

boolean

Whether to display intermediate outputs during generation

Default: false

seed

integer

Random seed. Leave blank to randomize the seed

output_format

string

Output file format

Default: "gif"

Run this model in Node.js with one line of code:

npx create-replicate --model=andreasjansson/stable-diffusion-animation

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run andreasjansson/stable-diffusion-animation using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "andreasjansson/stable-diffusion-animation:ca1f5e306e5721e19c473e0d094e6603f0456fe759c10715fcd6c1b79242d4a5",
  {
    input: {
      width: 512,
      height: 512,
      prompt_end: "tall rectangular black monolith, a white room in the future with a bed, victorian details and a tall black monolith, a detailed matte painting by Wes Anderson, behance, light and space, reimagined by industrial light and magic, matte painting, criterion collection",
      prompt_start: "tall rectangular black monolith, monkeys in the desert looking at a large tall monolith, a detailed matte painting by Wes Anderson, behance, light and space, reimagined by industrial light and magic, matte painting, criterion collection",
      gif_ping_pong: true,
      output_format: "mp4",
      guidance_scale: 7.5,
      prompt_strength: 0.9,
      film_interpolation: true,
      intermediate_output: false,
      num_inference_steps: 50,
      num_animation_frames: 25,
      gif_frames_per_second: 20,
      num_interpolation_steps: 5
    }
  }
);
console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run andreasjansson/stable-diffusion-animation using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "andreasjansson/stable-diffusion-animation:ca1f5e306e5721e19c473e0d094e6603f0456fe759c10715fcd6c1b79242d4a5",
    input={
        "width": 512,
        "height": 512,
        "prompt_end": "tall rectangular black monolith, a white room in the future with a bed, victorian details and a tall black monolith, a detailed matte painting by Wes Anderson, behance, light and space, reimagined by industrial light and magic, matte painting, criterion collection",
        "prompt_start": "tall rectangular black monolith, monkeys in the desert looking at a large tall monolith, a detailed matte painting by Wes Anderson, behance, light and space, reimagined by industrial light and magic, matte painting, criterion collection",
        "gif_ping_pong": True,
        "output_format": "mp4",
        "guidance_scale": 7.5,
        "prompt_strength": 0.9,
        "film_interpolation": True,
        "intermediate_output": False,
        "num_inference_steps": 50,
        "num_animation_frames": 25,
        "gif_frames_per_second": 20,
        "num_interpolation_steps": 5
    }
)

# The andreasjansson/stable-diffusion-animation model can stream output as it's running.
# The predict method returns an iterator, and you can iterate over that output.
for item in output:
    # https://replicate.com/andreasjansson/stable-diffusion-animation/api#output-schema
    print(item)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run andreasjansson/stable-diffusion-animation using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "ca1f5e306e5721e19c473e0d094e6603f0456fe759c10715fcd6c1b79242d4a5",
    "input": {
      "width": 512,
      "height": 512,
      "prompt_end": "tall rectangular black monolith, a white room in the future with a bed, victorian details and a tall black monolith, a detailed matte painting by Wes Anderson, behance, light and space, reimagined by industrial light and magic, matte painting, criterion collection",
      "prompt_start": "tall rectangular black monolith, monkeys in the desert looking at a large tall monolith, a detailed matte painting by Wes Anderson, behance, light and space, reimagined by industrial light and magic, matte painting, criterion collection",
      "gif_ping_pong": true,
      "output_format": "mp4",
      "guidance_scale": 7.5,
      "prompt_strength": 0.9,
      "film_interpolation": true,
      "intermediate_output": false,
      "num_inference_steps": 50,
      "num_animation_frames": 25,
      "gif_frames_per_second": 20,
      "num_interpolation_steps": 5
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

{
  "completed_at": "2022-08-28T21:01:43Z",
  "created_at": "2022-08-28T20:56:18.476932Z",
  "data_removed": false,
  "error": "",
  "id": "wsdaibdifnhlnptfyeyq6dgm4q",
  "input": {
    "width": 512,
    "height": 512,
    "prompt_end": "tall rectangular black monolith, a white room in the future with a bed, victorian details and a tall black monolith, a detailed matte painting by Wes Anderson, behance, light and space, reimagined by industrial light and magic, matte painting, criterion collection",
    "prompt_start": "tall rectangular black monolith, monkeys in the desert looking at a large tall monolith, a detailed matte painting by Wes Anderson, behance, light and space, reimagined by industrial light and magic, matte painting, criterion collection",
    "gif_ping_pong": true,
    "output_format": "mp4",
    "guidance_scale": 7.5,
    "prompt_strength": 0.9,
    "film_interpolation": true,
    "num_inference_steps": 50,
    "num_animation_frames": "25",
    "gif_frames_per_second": 20,
    "num_interpolation_steps": 5
  },
  "logs": "Using seed: 52273\r\nGenerating initial latents for 4 steps\r\nGenerating frame 0\r\nGenerating frame 1\r\nGenerating frame 2\r\nGenerating frame 3\r\nGenerating frame 4\r\nGenerating frame 5\r\nGenerating frame 6\r\nGenerating frame 7\r\nGenerating frame 8\r\nGenerating frame 9\r\nGenerating frame 10\r\nGenerating frame 11\r\nGenerating frame 12\r\nGenerating frame 13\r\nGenerating frame 14\r\nGenerating frame 15\r\nGenerating frame 16\r\nGenerating frame 17\r\nGenerating frame 18\r\nGenerating frame 19\r\nGenerating frame 20\r\nGenerating frame 21\r\nGenerating frame 22\r\nGenerating frame 23\r\nGenerating frame 24\r\nInterpolating images with FILM\r\nSaving MP4",
  "metrics": {
    "predict_time": 133,
    "total_time": 324.523068
  },
  "output": "https://replicate.delivery/mgxm/56df19d7-3776-4119-a0f9-d278b26a4a8a/output.mp4",
  "started_at": "2022-08-28T20:59:30Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/wsdaibdifnhlnptfyeyq6dgm4q",
    "cancel": "https://api.replicate.com/v1/predictions/wsdaibdifnhlnptfyeyq6dgm4q/cancel"
  },
  "version": "a0cd8005509b772461dd2a4dc58d304bac1d150d93fa35fdc80dc5835f766d4d"
}

Generated in

2 minutes 13 seconds

Tweak it Report

This example was created by a different version, andreasjansson/stable-diffusion-animation:a0cd8005.

Examples

View more examples

Run time and cost

This model costs approximately $0.13 to run on Replicate, or 7 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia A100 (80GB) GPU hardware. Predictions typically complete within 90 seconds. The predict time for this model varies significantly based on the inputs.

Readme

Stable Diffusion Animation

Animate Stable Diffusion by interpolating between two prompts

Code: https://github.com/andreasjansson/cog-stable-diffusion/tree/animate

How does it work?

Starting with noise, we then use stable diffusion to denoise for n steps towards the mid-point between the start prompt and end prompt, where n = num_inference_steps * (1 - prompt_strength). The higher the prompt strength, the fewer steps towards the mid-point.

We then denoise from that intermediate noisy output towards num_animation_frames interpolation points between the start and end prompts. By starting with an intermediate output, the model will generate samples that are similar to each other, resulting in a smoother animation.

Finally, the generated samples are interpolated with Google’s FILM (Frame Interpolation for Large Scene Motion) for extra smoothness.