cjwbw / lambda-eclipse

λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Space

Cold

Public
174 runs
L40S
Paper
License

Run with an API

Playground API Examples README Versions

Input

prompt

string

Shift + Return to add a new line

The prompt to guide the image generation.

Default: "a cat wearing glasses at the beach"

negative_prompt

string

Shift + Return to add a new line

over-exposure, under-exposure, saturated, duplicate, out of frame, lowres, cropped, worst quality, low quality, jpeg artifacts, morbid, mutilated, ugly, bad anatomy, bad proportions, deformed, blurryover-exposure, under-exposure, saturated, duplicate, out of frame, lowres, cropped, worst quality, low quality, jpeg artifacts, morbid, mutilated, ugly, bad anatomy, bad proportions, deformed, blurry

The prompt or prompts not to guide the image generation.

Default: "over-exposure, under-exposure, saturated, duplicate, out of frame, lowres, cropped, worst quality, low quality, jpeg artifacts, morbid, mutilated, ugly, bad anatomy, bad proportions, deformed, blurry"

image1

*file

Preview

The image for the first subject.

image2

file

Preview

Optional. The image for the first subject.

subject_category1

string

Shift + Return to add a new line

The subject category of the first image.

Default: "dog"

subject_category2

string

Shift + Return to add a new line

Optional. The subject category of the first image.

Default: ""

num_inference_steps

integer

(minimum: 1, maximum: 500)

The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference.

Default: 25

guidance_scale

number

(minimum: 1, maximum: 20)

Scale for classifier-free guidance.

Default: 7.5

seed

integer

Random seed. Leave blank to randomize the seed

Run this model in Node.js with one line of code:

npx create-replicate --model=cjwbw/lambda-eclipse

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run cjwbw/lambda-eclipse using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "cjwbw/lambda-eclipse:0e36b028d4ed70538c3e6ad369ebd2acf872b41e339b0adf1f6f7d095e248a55",
  {
    input: {
      image1: "https://replicate.delivery/pbxt/KNhvYb6Ux4ZbIu4C0W9W5Z2DDOkFvhwSl9h3O0zv8sbQuf5N/cat.png",
      image2: "https://replicate.delivery/pbxt/KNhvYqUGSU8ZuYWSX0UiJswZ0hLV63GSzeKT4CV0RRU1W4rc/blue_sunglasses.png",
      prompt: "a cat wearing glasses at the beach",
      guidance_scale: 4,
      negative_prompt: "over-exposure, under-exposure, saturated, duplicate, out of frame, lowres, cropped, worst quality, low quality, jpeg artifacts, morbid, mutilated, ugly, bad anatomy, bad proportions, deformed, blurry",
      subject_category1: "cat",
      subject_category2: "glasses",
      num_inference_steps: 50
    }
  }
);

// To access the file URL:
console.log(output.url()); //=> "http://example.com"

// To write the file to disk:
fs.writeFile("my-image.png", output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run cjwbw/lambda-eclipse using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "cjwbw/lambda-eclipse:0e36b028d4ed70538c3e6ad369ebd2acf872b41e339b0adf1f6f7d095e248a55",
    input={
        "image1": "https://replicate.delivery/pbxt/KNhvYb6Ux4ZbIu4C0W9W5Z2DDOkFvhwSl9h3O0zv8sbQuf5N/cat.png",
        "image2": "https://replicate.delivery/pbxt/KNhvYqUGSU8ZuYWSX0UiJswZ0hLV63GSzeKT4CV0RRU1W4rc/blue_sunglasses.png",
        "prompt": "a cat wearing glasses at the beach",
        "guidance_scale": 4,
        "negative_prompt": "over-exposure, under-exposure, saturated, duplicate, out of frame, lowres, cropped, worst quality, low quality, jpeg artifacts, morbid, mutilated, ugly, bad anatomy, bad proportions, deformed, blurry",
        "subject_category1": "cat",
        "subject_category2": "glasses",
        "num_inference_steps": 50
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run cjwbw/lambda-eclipse using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "0e36b028d4ed70538c3e6ad369ebd2acf872b41e339b0adf1f6f7d095e248a55",
    "input": {
      "image1": "https://replicate.delivery/pbxt/KNhvYb6Ux4ZbIu4C0W9W5Z2DDOkFvhwSl9h3O0zv8sbQuf5N/cat.png",
      "image2": "https://replicate.delivery/pbxt/KNhvYqUGSU8ZuYWSX0UiJswZ0hLV63GSzeKT4CV0RRU1W4rc/blue_sunglasses.png",
      "prompt": "a cat wearing glasses at the beach",
      "guidance_scale": 4,
      "negative_prompt": "over-exposure, under-exposure, saturated, duplicate, out of frame, lowres, cropped, worst quality, low quality, jpeg artifacts, morbid, mutilated, ugly, bad anatomy, bad proportions, deformed, blurry",
      "subject_category1": "cat",
      "subject_category2": "glasses",
      "num_inference_steps": 50
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

{
  "completed_at": "2024-02-10T23:28:31.887973Z",
  "created_at": "2024-02-10T23:28:24.848434Z",
  "data_removed": false,
  "error": null,
  "id": "io7rpm3bsnmvixw34b3buj33pq",
  "input": {
    "image1": "https://replicate.delivery/pbxt/KNhvYb6Ux4ZbIu4C0W9W5Z2DDOkFvhwSl9h3O0zv8sbQuf5N/cat.png",
    "image2": "https://replicate.delivery/pbxt/KNhvYqUGSU8ZuYWSX0UiJswZ0hLV63GSzeKT4CV0RRU1W4rc/blue_sunglasses.png",
    "prompt": "a cat wearing glasses at the beach",
    "guidance_scale": 4,
    "negative_prompt": "over-exposure, under-exposure, saturated, duplicate, out of frame, lowres, cropped, worst quality, low quality, jpeg artifacts, morbid, mutilated, ugly, bad anatomy, bad proportions, deformed, blurry",
    "subject_category1": "cat",
    "subject_category2": "glasses",
    "num_inference_steps": 50
  },
  "logs": "Using seed: 50866\n  0%|          | 0/50 [00:00<?, ?it/s]\n  4%|▍         | 2/50 [00:00<00:03, 12.38it/s]\n  8%|▊         | 4/50 [00:00<00:03, 12.44it/s]\n 12%|█▏        | 6/50 [00:00<00:03, 12.47it/s]\n 16%|█▌        | 8/50 [00:00<00:03, 12.49it/s]\n 20%|██        | 10/50 [00:00<00:03, 12.49it/s]\n 24%|██▍       | 12/50 [00:00<00:03, 12.52it/s]\n 28%|██▊       | 14/50 [00:01<00:02, 12.53it/s]\n 32%|███▏      | 16/50 [00:01<00:02, 12.56it/s]\n 36%|███▌      | 18/50 [00:01<00:02, 12.58it/s]\n 40%|████      | 20/50 [00:01<00:02, 12.60it/s]\n 44%|████▍     | 22/50 [00:01<00:02, 12.60it/s]\n 48%|████▊     | 24/50 [00:01<00:02, 12.55it/s]\n 52%|█████▏    | 26/50 [00:02<00:01, 12.56it/s]\n 56%|█████▌    | 28/50 [00:02<00:01, 12.58it/s]\n 60%|██████    | 30/50 [00:02<00:01, 12.59it/s]\n 64%|██████▍   | 32/50 [00:02<00:01, 12.58it/s]\n 68%|██████▊   | 34/50 [00:02<00:01, 12.60it/s]\n 72%|███████▏  | 36/50 [00:02<00:01, 12.61it/s]\n 76%|███████▌  | 38/50 [00:03<00:00, 12.61it/s]\n 80%|████████  | 40/50 [00:03<00:00, 12.61it/s]\n 84%|████████▍ | 42/50 [00:03<00:00, 12.61it/s]\n 88%|████████▊ | 44/50 [00:03<00:00, 12.61it/s]\n 92%|█████████▏| 46/50 [00:03<00:00, 12.60it/s]\n 96%|█████████▌| 48/50 [00:03<00:00, 12.60it/s]\n100%|██████████| 50/50 [00:03<00:00, 12.60it/s]\n100%|██████████| 50/50 [00:03<00:00, 12.57it/s]",
  "metrics": {
    "predict_time": 7.003064,
    "total_time": 7.039539
  },
  "output": "https://replicate.delivery/pbxt/hrAQiNWHRxZqAhezHKf5rSyN39iXFCgKmS711HqubX4eE3qkA/out.png",
  "started_at": "2024-02-10T23:28:24.884909Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/io7rpm3bsnmvixw34b3buj33pq",
    "cancel": "https://api.replicate.com/v1/predictions/io7rpm3bsnmvixw34b3buj33pq/cancel"
  },
  "version": "0e36b028d4ed70538c3e6ad369ebd2acf872b41e339b0adf1f6f7d095e248a55"
}

Generated in

7.0 seconds

Tweak itReport

Examples

View more examples

Run time and cost

This model costs approximately $0.060 to run on Replicate, or 16 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia L40S GPU hardware. Predictions typically complete within 62 seconds. The predict time for this model varies significantly based on the inputs.

Readme

λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Space

This repository contains the inference code for our paper, λ-ECLIPSE.

The λ-ECLIPSE model is a light weight support for multi-concept personalization. λ-ECLIPSE is tiny T2I prior model designed for Kandinsky v2.2 diffusion image generator.
λ-ECLIPSE model extends the ECLIPSE-Prior via incorporating the image-text interleaved data.
λ-ECLIPSE shows that we do not need to train the Personalized T2I (P-T2I) models on lot of resources. For instance, λ-ECLIPSE is trained on mere 74 GPU Hours (A100) compared to it’s couterparts BLIP-Diffusion (2304 GPU hours) and Kosmos-G (12300 GPU hours).

Qualitative Examples:

Quantitative Comparisons: Results

Acknowledgement

We would like to acknoweldge excellent open-source text-to-image models (Kalro and Kandinsky) without them this work would not have been possible. Also, we thank HuggingFace for streamlining the T2I models.