andreasjansson / tile-morph

Create tileable animations with seamless transitions (Updated 2 years, 4 months ago)

Cold

Public
529.3K runs
A100 (80GB)
GitHub
License

Iterate in playground

Run with an API

Playground API Examples README Versions

Input

prompt_start

*string

Shift + Return to add a new line

Prompt to start the animation with

prompt_end

*string

Shift + Return to add a new line

Prompt to end the animation with. You can include multiple prompts by separating the prompts with | (the 'pipe' character)

width

integer

Width of output video

Default: 512

height

integer

Height of output video

Default: 512

num_interpolation_steps

integer

(minimum: 0, maximum: 1000)

Number of steps to interpolate between animation frames

Default: 20

num_inference_steps

integer

(minimum: 1, maximum: 5000)

Number of denoising steps

Default: 50

num_animation_frames

integer

(minimum: 2, maximum: 50)

Number of frames to animate

Default: 10

guidance_scale

number

(minimum: 1, maximum: 20)

Scale for classifier-free guidance

Default: 7.5

frames_per_second

integer

(minimum: 1, maximum: 60)

Frames per second in output video

Default: 20

intermediate_output

boolean

Whether to display intermediate outputs during generation

Default: false

seed_start

integer

Random seed for first prompt. Leave blank to randomize the seed

seed_end

integer

Random seed for last prompt. Leave blank to randomize the seed

Run this model in Node.js with one line of code:

npx create-replicate --model=andreasjansson/tile-morph

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";
import fs from "node:fs";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run andreasjansson/tile-morph using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "andreasjansson/tile-morph:cbf059cce30a22d821a3c86309ae3b037dcd505dd2eba47f8ea6eba20adced85",
  {
    input: {
      width: 1024,
      height: 768,
      seed_end: 2,
      prompt_end: "tropical jungle, cgsociety",
      seed_start: 1,
      prompt_start: "colorful abstract patterns",
      guidance_scale: 7.5,
      frames_per_second: 20,
      intermediate_output: false,
      num_inference_steps: 20,
      num_animation_frames: 20,
      num_interpolation_steps: 25
    }
  }
);

// To access the file URL:
console.log(output[0].url()); //=> "http://example.com"

// To write the file to disk:
fs.writeFile("my-image.png", output[0]);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run andreasjansson/tile-morph using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "andreasjansson/tile-morph:cbf059cce30a22d821a3c86309ae3b037dcd505dd2eba47f8ea6eba20adced85",
    input={
        "width": 1024,
        "height": 768,
        "seed_end": 2,
        "prompt_end": "tropical jungle, cgsociety",
        "seed_start": 1,
        "prompt_start": "colorful abstract patterns",
        "guidance_scale": 7.5,
        "frames_per_second": 20,
        "intermediate_output": False,
        "num_inference_steps": 20,
        "num_animation_frames": 20,
        "num_interpolation_steps": 25
    }
)

# The andreasjansson/tile-morph model can stream output as it's running.
# The predict method returns an iterator, and you can iterate over that output.
for item in output:
    # https://replicate.com/andreasjansson/tile-morph/api#output-schema
    print(item)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run andreasjansson/tile-morph using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "andreasjansson/tile-morph:cbf059cce30a22d821a3c86309ae3b037dcd505dd2eba47f8ea6eba20adced85",
    "input": {
      "width": 1024,
      "height": 768,
      "seed_end": 2,
      "prompt_end": "tropical jungle, cgsociety",
      "seed_start": 1,
      "prompt_start": "colorful abstract patterns",
      "guidance_scale": 7.5,
      "frames_per_second": 20,
      "intermediate_output": false,
      "num_inference_steps": 20,
      "num_animation_frames": 20,
      "num_interpolation_steps": 25
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

{
  "completed_at": "2023-02-20T22:05:12.949739Z",
  "created_at": "2023-02-20T22:02:28.797000Z",
  "data_removed": false,
  "error": null,
  "id": "tbsl3qsevrap3a7y27y66emtbe",
  "input": {
    "width": "1024",
    "height": "768",
    "seed_end": 2,
    "prompt_end": "tropical jungle, cgsociety",
    "seed_start": 1,
    "prompt_start": "colorful abstract patterns",
    "guidance_scale": 7.5,
    "frames_per_second": 20,
    "num_inference_steps": "20",
    "num_animation_frames": "20",
    "num_interpolation_steps": "25"
  },
  "logs": "Using seeds: 1, 2\nGenerating first and last keyframes\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\nGenerating frame 1 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.153036117553711\nGenerating frame 2 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.1492583751678467\nGenerating frame 3 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.1709370613098145\nGenerating frame 4 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.1507010459899902\nGenerating frame 5 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.14638352394104\nGenerating frame 6 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.16998553276062\nGenerating frame 7 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.170248508453369\nGenerating frame 8 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.172081708908081\nGenerating frame 9 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.1681506633758545\nGenerating frame 10 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.1461799144744873\nGenerating frame 11 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.17413592338562\nGenerating frame 12 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.131671905517578\nGenerating frame 13 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.1551811695098877\nGenerating frame 14 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.1367413997650146\nGenerating frame 15 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.1480202674865723\nGenerating frame 16 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.143221139907837\nGenerating frame 17 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.140272855758667\nGenerating frame 18 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.1339914798736572\nGenerating frame 19 of keyframe 0\n  0%|          | 0/20 [00:00<?, ?it/s]\n0%|          | 0/20 [00:03<?, ?it/s]\ndenoise time.time() - t=3.1306416988372803\nInterpolating images from latents\nSaving MP4",
  "metrics": {
    "predict_time": 164.073466,
    "total_time": 164.152739
  },
  "output": [
    "https://replicate.delivery/pbxt/ZUZ5fk3s7X2RPaKecNyxmIefTkPftReHhooV2NWY0XJ9lgGIE/output.mp4"
  ],
  "started_at": "2023-02-20T22:02:28.876273Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/tbsl3qsevrap3a7y27y66emtbe",
    "cancel": "https://api.replicate.com/v1/predictions/tbsl3qsevrap3a7y27y66emtbe/cancel"
  },
  "version": "a819625e6d8b1884f0c5328cec39a7296737ab9bb4d1ea1d8a31a916cafa27bf"
}

Generated in

2 minutes 44 seconds

Tweak itReport View full prediction

Using seeds: 1, 2
Generating first and last keyframes
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
Generating frame 1 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.153036117553711
Generating frame 2 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.1492583751678467
Generating frame 3 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.1709370613098145
Generating frame 4 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.1507010459899902
Generating frame 5 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.14638352394104
Generating frame 6 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.16998553276062
Generating frame 7 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.170248508453369
Generating frame 8 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.172081708908081
Generating frame 9 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.1681506633758545
Generating frame 10 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.1461799144744873
Generating frame 11 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.17413592338562
Generating frame 12 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.131671905517578
Generating frame 13 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.1551811695098877
Generating frame 14 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.1367413997650146
Generating frame 15 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.1480202674865723
Generating frame 16 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.143221139907837
Generating frame 17 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.140272855758667
Generating frame 18 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.1339914798736572
Generating frame 19 of keyframe 0
  0%|          | 0/20 [00:00<?, ?it/s]
0%|          | 0/20 [00:03<?, ?it/s]
denoise time.time() - t=3.1306416988372803
Interpolating images from latents
Saving MP4

This output was created using a different version of the model, andreasjansson/tile-morph:a819625e.

Examples

View more examples

Run time and cost

This model runs on Nvidia A100 (80GB) GPU hardware. We don't yet have enough runs of this model to provide performance information.

Readme

TileMorph

TileMorph creates a tileable animation between two Stable Diffusion prompts. It uses the circular padding trick to generate images that wrap around the edges.

The animation effect is achieved by interpolating both in CLIP embedding space and latent space.

The number of CLIP interpolation steps is controlled by the num_animation_frames input. Each “animation frame” runs a full Stable Diffusion inference, which makes it slow but interesting.
The number of latent space interpolation steps between animation frames is controlled by the num_interpolation_steps input. Each interpolation step only runs a VAE inference, and is fast but less interesting. You can trade off interestingness versus prediction time by tweaking num_animation_frames and num_interpolation_steps
num_animation_frames * num_interpolation_steps = number of output frames
num_animation_frames * num_interpolation_steps / frames_per_second = output video length in seconds

This model supports seamless transitions between different generations. Set prompt_end and seed_end to the same value of video number n as prompt_start and seed_start of video number n + 1.