afiaka87 / glid-3-xl

CompVis `latent-diffusion text2im` finetuned for inpainting.

Cold

Public
8K runs
T4
GitHub
License

Run with an API

This model uses a mask. Visit the new playground to try this model with our mask editing tool.

Playground API Examples README Versions

Input

prompt

string

Shift + Return to add a new line

Your text prompt.

Default: ""

negative

string

Shift + Return to add a new line

(optional) Negate the model's prediction for this text from the model's prediction for the target text.

Default: ""

init_image

file

(optional) Initial image to use for the model's prediction. If provided alongside a mask, the image will be inpainted instead.

mask

file

a mask image for inpainting an init_image. white pixels = keep, black pixels = discard. resized to width = image width/8, height = image height/8

guidance_scale

number

(minimum: -20, maximum: 100)

Classifier-free guidance scale. Higher values will result in more guidance toward caption, with diminishing returns. Try values between 1.0 and 40.0. In general, going above 5.0 will introduce some artifacting.

Default: 5

steps

integer

(minimum: 15, maximum: 250)

Number of diffusion steps to run. Due to PLMS sampling, using more than 100 steps is unnecessary and may simply produce the exact same output.

Default: 50

batch_size

integer

(minimum: 1, maximum: 16)

Batch size. (higher = slower)

Default: 4

width

integer

Target width

Default: 256

height

integer

Target height

Default: 256

init_skip_fraction

number

(minimum: 0, maximum: 1)

Fraction of sampling steps to skip when using an init image. Defaults to 0.0 if init_image is not specified and 0.5 if init_image is specified.

Default: 0

aesthetic_rating

integer

Aesthetic rating (1-9) - embed to use.

Default: 9

aesthetic_weight

number

Aesthetic weight (0-1). How much to guide towards the aesthetic embed vs the prompt embed.

Default: 0.5

seed

integer

(minimum: -1, maximum: 4294967295)

Seed for random number generator. If -1, a random seed will be chosen.

Default: -1

intermediate_outputs

boolean

Whether to return intermediate outputs. Enable to visualize the diffusion process and/or debug the model. May slow down inference.

Default: false

Run this model in Node.js with one line of code:

npx create-replicate --model=afiaka87/glid-3-xl

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run afiaka87/glid-3-xl using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "afiaka87/glid-3-xl:d74db2a276065cf0d42fe9e2917219112ddf8c698f5d9acbe1cc353b58097dab",
  {
    input: {
      seed: -1,
      steps: 100,
      width: 256,
      height: 256,
      prompt: "pikachu rendered in pixar",
      negative: "",
      batch_size: 1,
      guidance_scale: 5,
      aesthetic_rating: 9,
      aesthetic_weight: 0.5,
      init_skip_fraction: 0,
      intermediate_outputs: false
    }
  }
);

console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run afiaka87/glid-3-xl using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "afiaka87/glid-3-xl:d74db2a276065cf0d42fe9e2917219112ddf8c698f5d9acbe1cc353b58097dab",
    input={
        "seed": -1,
        "steps": 100,
        "width": 256,
        "height": 256,
        "prompt": "pikachu rendered in pixar",
        "negative": "",
        "batch_size": 1,
        "guidance_scale": 5,
        "aesthetic_rating": 9,
        "aesthetic_weight": 0.5,
        "init_skip_fraction": 0,
        "intermediate_outputs": False
    }
)

# The afiaka87/glid-3-xl model can stream output as it's running.
# The predict method returns an iterator, and you can iterate over that output.
for item in output:
    # https://replicate.com/afiaka87/glid-3-xl/api#output-schema
    print(item)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run afiaka87/glid-3-xl using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "afiaka87/glid-3-xl:d74db2a276065cf0d42fe9e2917219112ddf8c698f5d9acbe1cc353b58097dab",
    "input": {
      "seed": -1,
      "steps": 100,
      "width": 256,
      "height": 256,
      "prompt": "pikachu rendered in pixar",
      "negative": "",
      "batch_size": 1,
      "guidance_scale": 5,
      "aesthetic_rating": 9,
      "aesthetic_weight": 0.5,
      "init_skip_fraction": 0,
      "intermediate_outputs": false
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

{
  "completed_at": "2022-07-13T06:36:13.121933Z",
  "created_at": "2022-07-13T06:36:02.238585Z",
  "data_removed": false,
  "error": null,
  "id": "n64nojk5qvgglgbqc4nbdu3zje",
  "input": {
    "seed": -1,
    "steps": 100,
    "width": 256,
    "height": 256,
    "prompt": "pikachu rendered in pixar",
    "batch_size": 1,
    "guidance_scale": 5,
    "aesthetic_rating": 9,
    "aesthetic_weight": 0.5
  },
  "logs": "Using seed 2882092835\nRunning simulation for pikachu rendered in pixar\nEncoding text embeddings with pikachu rendered in pixar dimensions\nUsing aesthetic embedding 9 with weight 0.5\nRunning diffusion...\n\n  0%|          | 0/100 [00:00<?, ?it/s]\n  1%|          | 1/100 [00:00<00:41,  2.37it/s]\n  2%|▏         | 2/100 [00:00<00:37,  2.64it/s]\n  3%|▎         | 3/100 [00:01<00:35,  2.74it/s]\n  5%|▌         | 5/100 [00:01<00:19,  4.79it/s]\n  7%|▋         | 7/100 [00:01<00:14,  6.47it/s]\n  9%|▉         | 9/100 [00:01<00:11,  7.79it/s]\n 11%|█         | 11/100 [00:01<00:10,  8.80it/s]\n 13%|█▎        | 13/100 [00:01<00:09,  9.56it/s]\n 15%|█▌        | 15/100 [00:02<00:08, 10.08it/s]\n 17%|█▋        | 17/100 [00:02<00:07, 10.42it/s]\n 19%|█▉        | 19/100 [00:02<00:07, 10.70it/s]\n 21%|██        | 21/100 [00:02<00:07, 10.84it/s]\n 23%|██▎       | 23/100 [00:02<00:06, 11.04it/s]\n 25%|██▌       | 25/100 [00:03<00:06, 11.11it/s]\n 27%|██▋       | 27/100 [00:03<00:06, 11.19it/s]\n 29%|██▉       | 29/100 [00:03<00:06, 11.25it/s]\n 31%|███       | 31/100 [00:03<00:06, 11.29it/s]\n 33%|███▎      | 33/100 [00:03<00:05, 11.31it/s]\n 35%|███▌      | 35/100 [00:03<00:05, 11.36it/s]\n 37%|███▋      | 37/100 [00:04<00:05, 11.34it/s]\n 39%|███▉      | 39/100 [00:04<00:05, 11.33it/s]\n 41%|████      | 41/100 [00:04<00:05, 11.32it/s]\n 43%|████▎     | 43/100 [00:04<00:05, 11.34it/s]\n 45%|████▌     | 45/100 [00:04<00:04, 11.32it/s]\n 47%|████▋     | 47/100 [00:04<00:04, 11.38it/s]\n 49%|████▉     | 49/100 [00:05<00:04, 11.40it/s]\n 51%|█████     | 51/100 [00:05<00:04, 11.36it/s]\n 53%|█████▎    | 53/100 [00:05<00:04, 11.37it/s]\n 55%|█████▌    | 55/100 [00:05<00:03, 11.39it/s]\n 57%|█████▋    | 57/100 [00:05<00:03, 11.37it/s]\n 59%|█████▉    | 59/100 [00:06<00:03, 11.36it/s]\n 61%|██████    | 61/100 [00:06<00:03, 11.39it/s]\n 63%|██████▎   | 63/100 [00:06<00:03, 11.36it/s]\n 65%|██████▌   | 65/100 [00:06<00:03, 11.38it/s]\n 67%|██████▋   | 67/100 [00:06<00:02, 11.36it/s]\n 69%|██████▉   | 69/100 [00:06<00:02, 11.35it/s]\n 71%|███████   | 71/100 [00:07<00:02, 11.33it/s]\n 73%|███████▎  | 73/100 [00:07<00:02, 11.32it/s]\n 75%|███████▌  | 75/100 [00:07<00:02, 11.35it/s]\n 77%|███████▋  | 77/100 [00:07<00:02, 11.31it/s]\n 79%|███████▉  | 79/100 [00:07<00:01, 11.30it/s]\n 81%|████████  | 81/100 [00:07<00:01, 11.30it/s]\n 83%|████████▎ | 83/100 [00:08<00:01, 11.30it/s]\n 85%|████████▌ | 85/100 [00:08<00:01, 11.28it/s]\n 87%|████████▋ | 87/100 [00:08<00:01, 11.31it/s]\n 89%|████████▉ | 89/100 [00:08<00:00, 11.28it/s]\n 91%|█████████ | 91/100 [00:08<00:00, 11.30it/s]\n 93%|█████████▎| 93/100 [00:09<00:00, 11.31it/s]\n 95%|█████████▌| 95/100 [00:09<00:00, 11.32it/s]\n 97%|█████████▋| 97/100 [00:09<00:00, 11.33it/s]\n 99%|█████████▉| 99/100 [00:09<00:00, 11.30it/s]\n100%|██████████| 100/100 [00:09<00:00, 10.34it/s]\nSaving final sample/s",
  "metrics": {
    "predict_time": 10.695204,
    "total_time": 10.883348
  },
  "output": [
    [
      "https://replicate.delivery/mgxm/804c6fe5-d74a-4796-8835-f411b3776dae/current_0.png"
    ]
  ],
  "started_at": "2022-07-13T06:36:02.426729Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/n64nojk5qvgglgbqc4nbdu3zje",
    "cancel": "https://api.replicate.com/v1/predictions/n64nojk5qvgglgbqc4nbdu3zje/cancel"
  },
  "version": "742660111a36a544bd6de5acab0b46db0115eae9b091267d5ec48d885d564671"
}

Generated in

10.7 seconds

Tweak itReport

Using seed 2882092835
Running simulation for pikachu rendered in pixar
Encoding text embeddings with pikachu rendered in pixar dimensions
Using aesthetic embedding 9 with weight 0.5
Running diffusion...

  0%|          | 0/100 [00:00<?, ?it/s]
  1%|          | 1/100 [00:00<00:41,  2.37it/s]
  2%|▏         | 2/100 [00:00<00:37,  2.64it/s]
  3%|▎         | 3/100 [00:01<00:35,  2.74it/s]
  5%|▌         | 5/100 [00:01<00:19,  4.79it/s]
  7%|▋         | 7/100 [00:01<00:14,  6.47it/s]
  9%|▉         | 9/100 [00:01<00:11,  7.79it/s]
 11%|█         | 11/100 [00:01<00:10,  8.80it/s]
 13%|█▎        | 13/100 [00:01<00:09,  9.56it/s]
 15%|█▌        | 15/100 [00:02<00:08, 10.08it/s]
 17%|█▋        | 17/100 [00:02<00:07, 10.42it/s]
 19%|█▉        | 19/100 [00:02<00:07, 10.70it/s]
 21%|██        | 21/100 [00:02<00:07, 10.84it/s]
 23%|██▎       | 23/100 [00:02<00:06, 11.04it/s]
 25%|██▌       | 25/100 [00:03<00:06, 11.11it/s]
 27%|██▋       | 27/100 [00:03<00:06, 11.19it/s]
 29%|██▉       | 29/100 [00:03<00:06, 11.25it/s]
 31%|███       | 31/100 [00:03<00:06, 11.29it/s]
 33%|███▎      | 33/100 [00:03<00:05, 11.31it/s]
 35%|███▌      | 35/100 [00:03<00:05, 11.36it/s]
 37%|███▋      | 37/100 [00:04<00:05, 11.34it/s]
 39%|███▉      | 39/100 [00:04<00:05, 11.33it/s]
 41%|████      | 41/100 [00:04<00:05, 11.32it/s]
 43%|████▎     | 43/100 [00:04<00:05, 11.34it/s]
 45%|████▌     | 45/100 [00:04<00:04, 11.32it/s]
 47%|████▋     | 47/100 [00:04<00:04, 11.38it/s]
 49%|████▉     | 49/100 [00:05<00:04, 11.40it/s]
 51%|█████     | 51/100 [00:05<00:04, 11.36it/s]
 53%|█████▎    | 53/100 [00:05<00:04, 11.37it/s]
 55%|█████▌    | 55/100 [00:05<00:03, 11.39it/s]
 57%|█████▋    | 57/100 [00:05<00:03, 11.37it/s]
 59%|█████▉    | 59/100 [00:06<00:03, 11.36it/s]
 61%|██████    | 61/100 [00:06<00:03, 11.39it/s]
 63%|██████▎   | 63/100 [00:06<00:03, 11.36it/s]
 65%|██████▌   | 65/100 [00:06<00:03, 11.38it/s]
 67%|██████▋   | 67/100 [00:06<00:02, 11.36it/s]
 69%|██████▉   | 69/100 [00:06<00:02, 11.35it/s]
 71%|███████   | 71/100 [00:07<00:02, 11.33it/s]
 73%|███████▎  | 73/100 [00:07<00:02, 11.32it/s]
 75%|███████▌  | 75/100 [00:07<00:02, 11.35it/s]
 77%|███████▋  | 77/100 [00:07<00:02, 11.31it/s]
 79%|███████▉  | 79/100 [00:07<00:01, 11.30it/s]
 81%|████████  | 81/100 [00:07<00:01, 11.30it/s]
 83%|████████▎ | 83/100 [00:08<00:01, 11.30it/s]
 85%|████████▌ | 85/100 [00:08<00:01, 11.28it/s]
 87%|████████▋ | 87/100 [00:08<00:01, 11.31it/s]
 89%|████████▉ | 89/100 [00:08<00:00, 11.28it/s]
 91%|█████████ | 91/100 [00:08<00:00, 11.30it/s]
 93%|█████████▎| 93/100 [00:09<00:00, 11.31it/s]
 95%|█████████▌| 95/100 [00:09<00:00, 11.32it/s]
 97%|█████████▋| 97/100 [00:09<00:00, 11.33it/s]
 99%|█████████▉| 99/100 [00:09<00:00, 11.30it/s]
100%|██████████| 100/100 [00:09<00:00, 10.34it/s]
Saving final sample/s

This output was created using a different version of the model, afiaka87/glid-3-xl:74266011.

Examples

View more examples

Run time and cost

This model costs approximately $0.0044 to run on Replicate, or 227 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia T4 GPU hardware. Predictions typically complete within 20 seconds. The predict time for this model varies significantly based on the inputs.

Readme

Latent Diffusion

CompVis’ 1.4B parameter Latent Diffusion Text-To-Image model finetuned for inpainting, logo generation, art generation and more.

This repo is modified from glid-3-xl. Aesthetic CLIP embeds are provided by aesthetic-predictor

Quick start (docker required)

Install docker
Install cog

The following command will download all weights and run a prediction with your inputs inside a proper docker container.

cog predict r8.im/laion-ai/erlich \
  -i prompt="an armchair in the form of an avocado" \
  -i negative="" \
  -i init_image=@path/to/image \
  -i mask=@path/to/mask \
  -i guidance_scale=5.0 \
  -i steps=100 \
  -i batch_size=4 \
  -i width=256 \
  -i height=256 \
  -i init_skip_fraction=0.0 \
  -i aesthetic_rating=9 \
  -i aesthetic_weight=0.5 \
  -i seed=-1 \
  -i intermediate_outputs=False

Valid remote image URL’s are:

Setup

Prerequisites

Please ensure the following dependencies are installed prior to building this repo:

build-essential
libopenmpi-dev
liblzma-dev
zlib1g-dev

Pytorch

It’s a good idea to use a virtual environment or a conda environment.

python3 -m venv .venv
source venv/bin/activate
(venv) $

Before installing, you should install pytorch manually by following the instructions at pytorch.org

In my instance, I needed the following for cuda 11.3.

(venv) $ pip install torch torchvision --extra-index-url https://download.pytorch.org/whl/cu113

To check your cuda version, run nvidia-smi.

Install ldm-finetune

You can now install this repo by running pip install -e . in the project directory.

(venv) $ git clone https://github.com/laion-ai/ldm-finetune.git
(venv) $ cd ldm-finetune
(venv) $ pip install -e .
(venv) $ pip install -r requirements.txt

Checkpoints

Foundation/Backbone models:

# CLIP-ONNX
wget -O textual.onnx 'https://clip-as-service.s3.us-east-2.amazonaws.com/models/onnx/ViT-L-14/textual.onnx'
wget -O visual.onnx 'https://clip-as-service.s3.us-east-2.amazonaws.com/models/onnx/ViT-L-14/visual.onnx'

### BERT Text Encoder
wget --continue https://dall-3.com/models/glid-3-xl/bert.pt

### kl-f8 VAE backbone
wget --continue https://dall-3.com/models/glid-3-xl/kl-f8.pt

Latent Diffusion Stage 2 (diffusion)

There are several stage 2 checkpoints to choose from:

(recommended) jack000 - `inpaint.pt`

The second finetune from jack000’s glid-3-xl adds support for inpainting and can be used for unconditional output as well by setting the inpaint image_embed to zeros. Additionally finetuned to use the CLIP text embed via cross-attention (similar to unCLIP).

wget –continue https://dall-3.com/models/glid-3-xl/inpaint.pt

LAION Finetuning Checkpoints

Laion also finetuned inpaint.pt with the aim of improving logo generation and painting generation.

Erlich

erlich is inpaint.pt finetuned on a dataset collected from LAION-5B named Large Logo Dataset. It consists of roughly 100K images of logos with captions generated via BLIP using aggressive re-ranking and filtering.

wget --continue -O erlich.pt https://huggingface.co/laion/erlich/resolve/main/model/ema_0.9999_120000.pt

“You know aviato?”

Ongo

Ongo is inpaint.pt finetuned on the Wikiart dataset consisting of about 100K paintings with captions generated via BLIP using aggressive re-ranking and filtering. We also make use of the original captions which contain the author name and the painting title.

wget https://huggingface.co/laion/ongo/resolve/main/ongo.pt

“Ongo Gablogian, the art collector. Charmed, I’m sure.”

LAION - `puck.pt`

puck has been trained on pixel art. While the underlying kl-f8 encoder seems to struggle somewhat with pixel art, results are still interesting.

wget https://huggingface.co/laion/puck/resolve/main/puck.pt

Other

### CompVis - `diffusion.pt`
# The original checkpoint from CompVis trained on `LAION-400M`. May output watermarks.
wget --continue https://dall-3.com/models/glid-3-xl/diffusion.pt

### jack000 - `finetune.pt`
# The first finetune from jack000's [glid-3-xl](https://github.com/jack000/glid-3-xl). Modified to accept a CLIP text embed and finetuned on curated data to help with watermarks. Doesn't support inpainting.
# wget https://dall-3.com/models/glid-3-xl/finetune.pt

Generating images

You can run prediction via python or docker. Currently the docker method is best supported.

Docker/cog

If you have access to a linux machine (or WSL2.0 on Windows 11) with docker installed, you can very easily run models by installing cog:

sudo curl -o /usr/local/bin/cog -L https://github.com/replicate/cog/releases/latest/download/cog_`uname -s`_`uname -m`
sudo chmod +x /usr/local/bin/cog

Modify the MODEL_PATH in cog_sample.py:

MODEL_PATH = "erlich.pt"  # Can be erlich, ongo, puck, etc.

Now you can run predictions via docker container using:

cog predict -i prompt="a logo of a fox made of fire"

Output will be returned as a base64 string at the end of generation and is also saved locally at current_{batch_idx}.png

Flask API

If you’d like to stand up your own ldm-finetune Flask API, you can run:

cog build -t my_ldm_image
docker run -d -p 5000:5000 --gpus all my_ldm_image

Predictions can then be accessed via HTTP:

curl http://localhost:5000/predictions -X POST \
    -H 'Content-Type: application/json' \
    -d '{"input": {"prompt": "a logo of a fox made of fire"}}'

The output from the API will be a list of base64 strings representing your generations.

Input

Output

Examples

Run time and cost

Readme

Latent Diffusion

Quick start (docker required)

Setup

Prerequisites

Pytorch

Install ldm-finetune

Checkpoints

Foundation/Backbone models:

Latent Diffusion Stage 2 (diffusion)

(recommended) jack000 - inpaint.pt

LAION Finetuning Checkpoints

Erlich

Ongo

LAION - puck.pt

Other

Generating images

Docker/cog

Flask API

(recommended) jack000 - `inpaint.pt`

LAION - `puck.pt`