andreasjansson / stable-diffusion-animation

Animate Stable Diffusion by interpolating between two prompts

  • Public
  • 118.8K runs
  • A100 (80GB)
  • GitHub
  • License

Input

Set the REPLICATE_API_TOKEN environment variable:
export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run andreasjansson/stable-diffusion-animation using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "ca1f5e306e5721e19c473e0d094e6603f0456fe759c10715fcd6c1b79242d4a5",
    "input": {
      "width": 512,
      "height": 512,
      "prompt_end": "tall rectangular black monolith, a white room in the future with a bed, victorian details and a tall black monolith, a detailed matte painting by Wes Anderson, behance, light and space, reimagined by industrial light and magic, matte painting, criterion collection",
      "prompt_start": "tall rectangular black monolith, monkeys in the desert looking at a large tall monolith, a detailed matte painting by Wes Anderson, behance, light and space, reimagined by industrial light and magic, matte painting, criterion collection",
      "gif_ping_pong": true,
      "output_format": "mp4",
      "guidance_scale": 7.5,
      "prompt_strength": 0.9,
      "film_interpolation": true,
      "intermediate_output": false,
      "num_inference_steps": 50,
      "num_animation_frames": 25,
      "gif_frames_per_second": 20,
      "num_interpolation_steps": 5
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

Generated in

This output was created using a different version of the model, andreasjansson/stable-diffusion-animation:a0cd8005.

Run time and cost

This model costs approximately $0.13 to run on Replicate, or 7 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia A100 (80GB) GPU hardware. Predictions typically complete within 90 seconds. The predict time for this model varies significantly based on the inputs.

Readme

Stable Diffusion Animation

Animate Stable Diffusion by interpolating between two prompts

Code: https://github.com/andreasjansson/cog-stable-diffusion/tree/animate

How does it work?

Starting with noise, we then use stable diffusion to denoise for n steps towards the mid-point between the start prompt and end prompt, where n = num_inference_steps * (1 - prompt_strength). The higher the prompt strength, the fewer steps towards the mid-point.

We then denoise from that intermediate noisy output towards num_animation_frames interpolation points between the start and end prompts. By starting with an intermediate output, the model will generate samples that are similar to each other, resulting in a smoother animation.

Finally, the generated samples are interpolated with Google’s FILM (Frame Interpolation for Large Scene Motion) for extra smoothness.