laion-ai / laionide-v3

GLIDE finetuned on LAION5B, then more on curated datasets.

Cold

Public
62K runs
T4
GitHub
Paper
License

Run with an API

Playground API Examples README Versions

Input

prompt

*string

Shift + Return to add a new line

The Lovers demon skull werewolf tentacle tarot card | Tarot card on artstation | eldritch cardThe Lovers demon skull werewolf tentacle tarot card | Tarot card on artstation | eldritch card

Text prompt to use.

batch_size

integer

(minimum: 1, maximum: 6)

Batch size. Number of generations to predict

Default: 3

side_x

integer

Must be multiple of 8. Going above 64 is not recommended. Actual image will be 4x larger.

Default: 64

side_y

integer

Must be multiple of 8. Going above 64 is not recommended. Actual image will be 4x larger.

Default: 64

upsample_stage

boolean

Performs prompt-aware upsampling by 4x base resolution

Default: true

guidance_scale

number

Classifier-free guidance scale. Higher values move further away from unconditional outputs. Lower values move closer to unconditional outputs. Negative values guide towards semantically opposite classes. 4-16 is a reasonable range.

Default: 4

upsample_temp

string

Upsample temperature. Consider lowering to ~0.997 for blurry images with fewer artifacts.

Default: "0.998"

timestep_respacing

string

Number of timesteps to use for base model. Going above 50 has diminishing returns.

Default: "40"

sr_timestep_respacing

string

Number of timesteps to use for base model. Going above 40 has diminishing returns.

Default: "17"

seed

integer

Seed for reproducibility

Default: 0

Run this model in Node.js with one line of code:

npx create-replicate --model=laion-ai/laionide-v3

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run laion-ai/laionide-v3 using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "laion-ai/laionide-v3:153af311cc39000fee1effcee0acbd5d954967193a5ab56368ffcd4766781de2",
  {
    input: {
      seed: 0,
      prompt: "The Lovers demon skull werewolf tentacle tarot card | Tarot card on artstation | eldritch card",
      side_x: 64,
      side_y: 64,
      batch_size: 3,
      upsample_temp: 0.998,
      guidance_scale: 3,
      upsample_stage: true,
      timestep_respacing: "25",
      sr_timestep_respacing: "27"
    }
  }
);
console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run laion-ai/laionide-v3 using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "laion-ai/laionide-v3:153af311cc39000fee1effcee0acbd5d954967193a5ab56368ffcd4766781de2",
    input={
        "seed": 0,
        "prompt": "The Lovers demon skull werewolf tentacle tarot card | Tarot card on artstation | eldritch card",
        "side_x": 64,
        "side_y": 64,
        "batch_size": 3,
        "upsample_temp": 0.998,
        "guidance_scale": 3,
        "upsample_stage": True,
        "timestep_respacing": "25",
        "sr_timestep_respacing": "27"
    }
)

# The laion-ai/laionide-v3 model can stream output as it's running.
# The predict method returns an iterator, and you can iterate over that output.
for item in output:
    # https://replicate.com/laion-ai/laionide-v3/api#output-schema
    print(item)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run laion-ai/laionide-v3 using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "153af311cc39000fee1effcee0acbd5d954967193a5ab56368ffcd4766781de2",
    "input": {
      "seed": 0,
      "prompt": "The Lovers demon skull werewolf tentacle tarot card | Tarot card on artstation | eldritch card",
      "side_x": 64,
      "side_y": 64,
      "batch_size": 3,
      "upsample_temp": 0.998,
      "guidance_scale": 3,
      "upsample_stage": true,
      "timestep_respacing": "25",
      "sr_timestep_respacing": "27"
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

{
  "completed_at": "2022-02-25T16:20:55.080250Z",
  "created_at": "2022-02-25T16:20:18.982110Z",
  "data_removed": false,
  "error": null,
  "id": "zheef7k26fhytoxwz7ceri2744",
  "input": {
    "prompt": "The Lovers demon skull werewolf tentacle tarot card | Tarot card on artstation | eldritch card",
    "side_x": 64,
    "side_y": 64,
    "batch_size": 3,
    "upsample_temp": 0.998,
    "guidance_scale": 3,
    "upsample_stage": true,
    "timestep_respacing": "25",
    "sr_timestep_respacing": "27"
  },
  "logs": "Generating 64x64 samples with 25 timesteps using GLIDE-base-64px...\n\n  0%|          | 0/23 [00:00<?, ?it/s]\n  4%|▍         | 1/23 [00:00<00:17,  1.27it/s]\n  9%|▊         | 2/23 [00:01<00:15,  1.34it/s]\n 13%|█▎        | 3/23 [00:02<00:14,  1.36it/s]\n 17%|█▋        | 4/23 [00:02<00:09,  1.93it/s]\n 22%|██▏       | 5/23 [00:02<00:07,  2.52it/s]\n 26%|██▌       | 6/23 [00:02<00:05,  3.09it/s]\n 30%|███       | 7/23 [00:02<00:04,  3.61it/s]\n 35%|███▍      | 8/23 [00:03<00:03,  4.06it/s]\n 39%|███▉      | 9/23 [00:03<00:03,  4.42it/s]\n 43%|████▎     | 10/23 [00:03<00:02,  4.72it/s]\n 48%|████▊     | 11/23 [00:03<00:02,  4.94it/s]\n 52%|█████▏    | 12/23 [00:03<00:02,  5.09it/s]\n 57%|█████▋    | 13/23 [00:04<00:01,  5.22it/s]\n 61%|██████    | 14/23 [00:04<00:01,  5.31it/s]\n 65%|██████▌   | 15/23 [00:04<00:01,  5.36it/s]\n 70%|██████▉   | 16/23 [00:04<00:01,  5.39it/s]\n 74%|███████▍  | 17/23 [00:04<00:01,  5.42it/s]\n 78%|███████▊  | 18/23 [00:04<00:00,  5.43it/s]\n 83%|████████▎ | 19/23 [00:05<00:00,  5.46it/s]\n 87%|████████▋ | 20/23 [00:05<00:00,  5.46it/s]\n 91%|█████████▏| 21/23 [00:05<00:00,  5.47it/s]\n 96%|█████████▌| 22/23 [00:05<00:00,  5.47it/s]\n100%|██████████| 23/23 [00:05<00:00,  5.46it/s]\n100%|██████████| 23/23 [00:05<00:00,  3.92it/s]\nUpsampling outputs from GLIDE-base 64x64 to 256x256 using 27 timesteps...\n\n  0%|          | 0/25 [00:00<?, ?it/s]\n  4%|▍         | 1/25 [00:01<00:47,  1.97s/it]\n  8%|▊         | 2/25 [00:03<00:45,  1.96s/it]\n 12%|█▏        | 3/25 [00:05<00:43,  1.96s/it]\n 16%|█▌        | 4/25 [00:06<00:28,  1.38s/it]\n 20%|██        | 5/25 [00:06<00:21,  1.06s/it]\n 24%|██▍       | 6/25 [00:07<00:16,  1.15it/s]\n 28%|██▊       | 7/25 [00:07<00:13,  1.34it/s]\n 32%|███▏      | 8/25 [00:08<00:11,  1.50it/s]\n 36%|███▌      | 9/25 [00:08<00:09,  1.63it/s]\n 40%|████      | 10/25 [00:09<00:08,  1.73it/s]\n 44%|████▍     | 11/25 [00:09<00:07,  1.81it/s]\n 48%|████▊     | 12/25 [00:10<00:06,  1.86it/s]\n 52%|█████▏    | 13/25 [00:10<00:06,  1.90it/s]\n 56%|█████▌    | 14/25 [00:11<00:05,  1.94it/s]\n 60%|██████    | 15/25 [00:11<00:05,  1.96it/s]\n 64%|██████▍   | 16/25 [00:12<00:04,  1.97it/s]\n 68%|██████▊   | 17/25 [00:12<00:04,  1.98it/s]\n 72%|███████▏  | 18/25 [00:13<00:03,  1.99it/s]\n 76%|███████▌  | 19/25 [00:13<00:03,  1.99it/s]\n 80%|████████  | 20/25 [00:14<00:02,  1.99it/s]\n 84%|████████▍ | 21/25 [00:14<00:02,  2.00it/s]\n 88%|████████▊ | 22/25 [00:15<00:01,  1.99it/s]\n 92%|█████████▏| 23/25 [00:15<00:01,  1.99it/s]\n 96%|█████████▌| 24/25 [00:16<00:00,  2.00it/s]\n100%|██████████| 25/25 [00:16<00:00,  2.00it/s]\n100%|██████████| 25/25 [00:16<00:00,  1.49it/s]",
  "metrics": {
    "predict_time": 35.933512,
    "total_time": 36.09814
  },
  "output": [
    {
      "file": "https://replicate.delivery/mgxm/4b4137a8-0dab-4c71-82ed-ef42219717da/upsample_predictions.png"
    }
  ],
  "started_at": "2022-02-25T16:20:19.146738Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/zheef7k26fhytoxwz7ceri2744",
    "cancel": "https://api.replicate.com/v1/predictions/zheef7k26fhytoxwz7ceri2744/cancel"
  },
  "version": "db44d8128b86ccbf770aa1c1d6cefa2f7dc7ee02333a4a3ab004e0de89d24f76"
}

Generated in

35.9 seconds

Tweak itReport

Generating 64x64 samples with 25 timesteps using GLIDE-base-64px...

  0%|          | 0/23 [00:00<?, ?it/s]
  4%|▍         | 1/23 [00:00<00:17,  1.27it/s]
  9%|▊         | 2/23 [00:01<00:15,  1.34it/s]
 13%|█▎        | 3/23 [00:02<00:14,  1.36it/s]
 17%|█▋        | 4/23 [00:02<00:09,  1.93it/s]
 22%|██▏       | 5/23 [00:02<00:07,  2.52it/s]
 26%|██▌       | 6/23 [00:02<00:05,  3.09it/s]
 30%|███       | 7/23 [00:02<00:04,  3.61it/s]
 35%|███▍      | 8/23 [00:03<00:03,  4.06it/s]
 39%|███▉      | 9/23 [00:03<00:03,  4.42it/s]
 43%|████▎     | 10/23 [00:03<00:02,  4.72it/s]
 48%|████▊     | 11/23 [00:03<00:02,  4.94it/s]
 52%|█████▏    | 12/23 [00:03<00:02,  5.09it/s]
 57%|█████▋    | 13/23 [00:04<00:01,  5.22it/s]
 61%|██████    | 14/23 [00:04<00:01,  5.31it/s]
 65%|██████▌   | 15/23 [00:04<00:01,  5.36it/s]
 70%|██████▉   | 16/23 [00:04<00:01,  5.39it/s]
 74%|███████▍  | 17/23 [00:04<00:01,  5.42it/s]
 78%|███████▊  | 18/23 [00:04<00:00,  5.43it/s]
 83%|████████▎ | 19/23 [00:05<00:00,  5.46it/s]
 87%|████████▋ | 20/23 [00:05<00:00,  5.46it/s]
 91%|█████████▏| 21/23 [00:05<00:00,  5.47it/s]
 96%|█████████▌| 22/23 [00:05<00:00,  5.47it/s]
100%|██████████| 23/23 [00:05<00:00,  5.46it/s]
100%|██████████| 23/23 [00:05<00:00,  3.92it/s]
Upsampling outputs from GLIDE-base 64x64 to 256x256 using 27 timesteps...

  0%|          | 0/25 [00:00<?, ?it/s]
  4%|▍         | 1/25 [00:01<00:47,  1.97s/it]
  8%|▊         | 2/25 [00:03<00:45,  1.96s/it]
 12%|█▏        | 3/25 [00:05<00:43,  1.96s/it]
 16%|█▌        | 4/25 [00:06<00:28,  1.38s/it]
 20%|██        | 5/25 [00:06<00:21,  1.06s/it]
 24%|██▍       | 6/25 [00:07<00:16,  1.15it/s]
 28%|██▊       | 7/25 [00:07<00:13,  1.34it/s]
 32%|███▏      | 8/25 [00:08<00:11,  1.50it/s]
 36%|███▌      | 9/25 [00:08<00:09,  1.63it/s]
 40%|████      | 10/25 [00:09<00:08,  1.73it/s]
 44%|████▍     | 11/25 [00:09<00:07,  1.81it/s]
 48%|████▊     | 12/25 [00:10<00:06,  1.86it/s]
 52%|█████▏    | 13/25 [00:10<00:06,  1.90it/s]
 56%|█████▌    | 14/25 [00:11<00:05,  1.94it/s]
 60%|██████    | 15/25 [00:11<00:05,  1.96it/s]
 64%|██████▍   | 16/25 [00:12<00:04,  1.97it/s]
 68%|██████▊   | 17/25 [00:12<00:04,  1.98it/s]
 72%|███████▏  | 18/25 [00:13<00:03,  1.99it/s]
 76%|███████▌  | 19/25 [00:13<00:03,  1.99it/s]
 80%|████████  | 20/25 [00:14<00:02,  1.99it/s]
 84%|████████▍ | 21/25 [00:14<00:02,  2.00it/s]
 88%|████████▊ | 22/25 [00:15<00:01,  1.99it/s]
 92%|█████████▏| 23/25 [00:15<00:01,  1.99it/s]
 96%|█████████▌| 24/25 [00:16<00:00,  2.00it/s]
100%|██████████| 25/25 [00:16<00:00,  2.00it/s]
100%|██████████| 25/25 [00:16<00:00,  1.49it/s]

This example was created by a different version, laion-ai/laionide-v3:db44d812.

Examples

View more examples

Run time and cost

This model runs on Nvidia T4 GPU hardware. We don't yet have enough runs of this model to provide performance information.

Readme

Laionide (version 3)

Direct comparison to OpenAI’s model using COCO captions

Shout out to stability.ai for donating the compute to laion needed for this to be possible.

Files: - laionide-v3-base.pt

Inference: - replicate - colab - locally

Results: - comparison to openai W&B report

Notes: - You can use laionide-v2-sr.pt to upscale the outputs from laionide-v3-base.pt. - There are watermarks in some outputs. You can try to prompt engineer this away, but it isn’t always possible. royalty free seems to work well.

Training details: - finetuned laionide-v2-base.pt for 9 epochs on a subset of CC12M (~1.5 million pairs), COCO (~100K pairs), virtual genome (~100K pairs), and open images localized annotations (~800K pairs). - 20% of unconditional/empty token, per the paper.