pharmapsychotic / clip-interrogator

The CLIP Interrogator is a prompt engineering tool that combines OpenAI's CLIP and Salesforce's BLIP to optimize text prompts to match a given image. Use the resulting prompts with text-to-image models like Stable Diffusion to create cool art!

Warm

Public
3.5M runs
T4
GitHub
License

Run with an API

Playground API Examples README Versions

Prediction

pharmapsychotic/clip-interrogator:41fdb702d3fbe06c9835e39ac6ddfb8dee872835b22d6e8a7e0a467c793155f9

Model

pharmapsychotic/clip-interrogator:41fdb702

6iveqabqxrf55lw2xez2owwn6e

Status

Succeeded

Source

Web

Hardware

–

Total duration

3m 36s

Created

over 2 years ago by @pharmapsychotic

Input

image
clip_model_name: ViT-L/14

{
  "image": "https://replicate.delivery/pbxt/HrXsgowfhbZi3dImGZoIcvnz7oZfMtFY4UAEU8vBIakTd8JQ/watercolour-4799014_960_720.jpg",
  "clip_model_name": "ViT-L/14"
}

Install Replicate’s Node.js client library:

npm install replicate

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run pharmapsychotic/clip-interrogator using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "pharmapsychotic/clip-interrogator:41fdb702d3fbe06c9835e39ac6ddfb8dee872835b22d6e8a7e0a467c793155f9",
  {
    input: {
      image: "https://replicate.delivery/pbxt/HrXsgowfhbZi3dImGZoIcvnz7oZfMtFY4UAEU8vBIakTd8JQ/watercolour-4799014_960_720.jpg",
      clip_model_name: "ViT-L/14"
    }
  }
);

console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Import the client:

import replicate

Run pharmapsychotic/clip-interrogator using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "pharmapsychotic/clip-interrogator:41fdb702d3fbe06c9835e39ac6ddfb8dee872835b22d6e8a7e0a467c793155f9",
    input={
        "image": "https://replicate.delivery/pbxt/HrXsgowfhbZi3dImGZoIcvnz7oZfMtFY4UAEU8vBIakTd8JQ/watercolour-4799014_960_720.jpg",
        "clip_model_name": "ViT-L/14"
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Run pharmapsychotic/clip-interrogator using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "pharmapsychotic/clip-interrogator:41fdb702d3fbe06c9835e39ac6ddfb8dee872835b22d6e8a7e0a467c793155f9",
    "input": {
      "image": "https://replicate.delivery/pbxt/HrXsgowfhbZi3dImGZoIcvnz7oZfMtFY4UAEU8vBIakTd8JQ/watercolour-4799014_960_720.jpg",
      "clip_model_name": "ViT-L/14"
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

You can run this model locally using Cog. First, install Cog:

brew install cog

If you don’t have Homebrew, there are other installation options available.

Run this to download the model and run it in your local environment:

cog predict r8.im/pharmapsychotic/clip-interrogator@sha256:41fdb702d3fbe06c9835e39ac6ddfb8dee872835b22d6e8a7e0a467c793155f9 \
  -i 'image="https://replicate.delivery/pbxt/HrXsgowfhbZi3dImGZoIcvnz7oZfMtFY4UAEU8vBIakTd8JQ/watercolour-4799014_960_720.jpg"' \
  -i 'clip_model_name="ViT-L/14"'

To learn more, take a look at the Cog documentation.

Run this to download the model and run it in your local environment:

docker run -d -p 5000:5000 --gpus=all r8.im/pharmapsychotic/clip-interrogator@sha256:41fdb702d3fbe06c9835e39ac6ddfb8dee872835b22d6e8a7e0a467c793155f9
curl -s -X POST \
  -H "Content-Type: application/json" \
  -d $'{
    "input": {
      "image": "https://replicate.delivery/pbxt/HrXsgowfhbZi3dImGZoIcvnz7oZfMtFY4UAEU8vBIakTd8JQ/watercolour-4799014_960_720.jpg",
      "clip_model_name": "ViT-L/14"
    }
  }' \
  http://localhost:5000/predictions

To learn more, take a look at the Cog documentation.

Output

a watercolor painting of a sea turtle, a digital painting, by Kubisi art, featured on dribbble, medibang, warm saturated palette, red and green tones, turquoise horizon, digital art h 9 6 0, detailed scenery —width 672, illustration:.4, spray art, artstatiom

{
  "completed_at": "2022-11-27T22:32:41.135591Z",
  "created_at": "2022-11-27T22:29:05.240885Z",
  "data_removed": false,
  "error": null,
  "id": "6iveqabqxrf55lw2xez2owwn6e",
  "input": {
    "image": "https://replicate.delivery/pbxt/HrXsgowfhbZi3dImGZoIcvnz7oZfMtFY4UAEU8vBIakTd8JQ/watercolour-4799014_960_720.jpg",
    "clip_model_name": "ViT-L/14"
  },
  "logs": "0%|          | 0/50 [00:00<?, ?it/s]\n 24%|██▍       | 12/50 [00:00<00:00, 119.71it/s]\n 48%|████▊     | 24/50 [00:00<00:00, 117.53it/s]\n 72%|███████▏  | 36/50 [00:00<00:00, 117.99it/s]\n 96%|█████████▌| 48/50 [00:00<00:00, 118.17it/s]\n100%|██████████| 50/50 [00:00<00:00, 119.05it/s]\n  0%|          | 0/6 [00:00<?, ?it/s]\n100%|██████████| 6/6 [00:00<00:00, 93.98it/s]\n  0%|          | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:00<00:00, 33.81it/s]\nFlavor chain:   0%|          | 0/25 [00:00<?, ?it/s]\nFlavor chain:   4%|▍         | 1/25 [00:03<01:12,  3.01s/it]\nFlavor chain:   8%|▊         | 2/25 [00:05<01:08,  2.97s/it]\nFlavor chain:  12%|█▏        | 3/25 [00:09<01:06,  3.02s/it]\nFlavor chain:  16%|█▌        | 4/25 [00:12<01:03,  3.05s/it]\nFlavor chain:  20%|██        | 5/25 [00:15<01:00,  3.04s/it]\nFlavor chain:  24%|██▍       | 6/25 [00:18<00:57,  3.05s/it]\nFlavor chain:  28%|██▊       | 7/25 [00:21<00:55,  3.09s/it]\nFlavor chain:  32%|███▏      | 8/25 [00:24<00:55,  3.25s/it]\nFlavor chain:  36%|███▌      | 9/25 [00:28<00:53,  3.33s/it]\nFlavor chain:  40%|████      | 10/25 [00:32<00:51,  3.45s/it]\nFlavor chain:  40%|████      | 10/25 [00:33<00:49,  3.31s/it]",
  "metrics": {
    "predict_time": 37.674196,
    "total_time": 215.894706
  },
  "output": "a watercolor painting of a sea turtle, a digital painting, by Kubisi art, featured on dribbble, medibang, warm saturated palette, red and green tones, turquoise horizon, digital art h 9 6 0, detailed scenery —width 672, illustration:.4, spray art, artstatiom",
  "started_at": "2022-11-27T22:32:03.461395Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/6iveqabqxrf55lw2xez2owwn6e",
    "cancel": "https://api.replicate.com/v1/predictions/6iveqabqxrf55lw2xez2owwn6e/cancel"
  },
  "version": "41fdb702d3fbe06c9835e39ac6ddfb8dee872835b22d6e8a7e0a467c793155f9"
}

Generated in

37.7 seconds

Tweak it Share Report

Prediction

pharmapsychotic/clip-interrogator:f7ec9b39f3e37a723fd9549a2183905f5a8d32357346972de6ecd804614f0fd7

Model

pharmapsychotic/clip-interrogator:f7ec9b39

mfj4quetsfcuhk24dtit2sbdpe

Status

Succeeded

Source

Web

Hardware

–

Total duration

1m 50s

Created

over 2 years ago by @pharmapsychotic

Input

image
clip_model_name: ViT-L-14/openai

{
  "image": "https://replicate.delivery/pbxt/HraYo2UMXhjKEjuAUHLyMeiEg3upV2bxy7LSHhYQ81dPHUtE/cdb2593a4ece73b83fd4271c4cb92e57.jpg",
  "clip_model_name": "ViT-L-14/openai"
}

Install Replicate’s Node.js client library:

npm install replicate

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run pharmapsychotic/clip-interrogator using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "pharmapsychotic/clip-interrogator:f7ec9b39f3e37a723fd9549a2183905f5a8d32357346972de6ecd804614f0fd7",
  {
    input: {
      image: "https://replicate.delivery/pbxt/HraYo2UMXhjKEjuAUHLyMeiEg3upV2bxy7LSHhYQ81dPHUtE/cdb2593a4ece73b83fd4271c4cb92e57.jpg",
      clip_model_name: "ViT-L-14/openai"
    }
  }
);

console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Import the client:

import replicate

Run pharmapsychotic/clip-interrogator using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "pharmapsychotic/clip-interrogator:f7ec9b39f3e37a723fd9549a2183905f5a8d32357346972de6ecd804614f0fd7",
    input={
        "image": "https://replicate.delivery/pbxt/HraYo2UMXhjKEjuAUHLyMeiEg3upV2bxy7LSHhYQ81dPHUtE/cdb2593a4ece73b83fd4271c4cb92e57.jpg",
        "clip_model_name": "ViT-L-14/openai"
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Run pharmapsychotic/clip-interrogator using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "pharmapsychotic/clip-interrogator:f7ec9b39f3e37a723fd9549a2183905f5a8d32357346972de6ecd804614f0fd7",
    "input": {
      "image": "https://replicate.delivery/pbxt/HraYo2UMXhjKEjuAUHLyMeiEg3upV2bxy7LSHhYQ81dPHUtE/cdb2593a4ece73b83fd4271c4cb92e57.jpg",
      "clip_model_name": "ViT-L-14/openai"
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

a painting of a field with green grass, by Vincent Van Gogh, featured on deviantart, windy day, virtuosic level detail, the front of a trading card, loosely cropped, expansive, tendrils in the background, photo courtesy museum of art, 1980s art, standing on a hill, creative commons attribution

{
  "completed_at": "2022-11-28T01:26:14.246401Z",
  "created_at": "2022-11-28T01:24:24.296679Z",
  "data_removed": false,
  "error": null,
  "id": "mfj4quetsfcuhk24dtit2sbdpe",
  "input": {
    "image": "https://replicate.delivery/pbxt/HraYo2UMXhjKEjuAUHLyMeiEg3upV2bxy7LSHhYQ81dPHUtE/cdb2593a4ece73b83fd4271c4cb92e57.jpg",
    "clip_model_name": "ViT-L-14/openai"
  },
  "logs": "0%|          | 0/50 [00:00<?, ?it/s]\n 38%|███▊      | 19/50 [00:00<00:00, 189.19it/s]\n 78%|███████▊  | 39/50 [00:00<00:00, 192.65it/s]\n100%|██████████| 50/50 [00:00<00:00, 197.35it/s]\n  0%|          | 0/6 [00:00<?, ?it/s]\n100%|██████████| 6/6 [00:00<00:00, 127.91it/s]\n  0%|          | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:00<00:00, 49.73it/s]\nFlavor chain:   0%|          | 0/32 [00:00<?, ?it/s]\nFlavor chain:   3%|▎         | 1/32 [00:03<01:33,  3.01s/it]\nFlavor chain:   6%|▋         | 2/32 [00:05<01:29,  2.99s/it]\nFlavor chain:   9%|▉         | 3/32 [00:08<01:26,  2.98s/it]\nFlavor chain:  12%|█▎        | 4/32 [00:11<01:23,  3.00s/it]\nFlavor chain:  16%|█▌        | 5/32 [00:14<01:21,  3.00s/it]\nFlavor chain:  19%|█▉        | 6/32 [00:18<01:18,  3.01s/it]\nFlavor chain:  22%|██▏       | 7/32 [00:21<01:15,  3.02s/it]\nFlavor chain:  25%|██▌       | 8/32 [00:24<01:13,  3.05s/it]\nFlavor chain:  28%|██▊       | 9/32 [00:27<01:10,  3.07s/it]\nFlavor chain:  31%|███▏      | 10/32 [00:30<01:07,  3.09s/it]\nFlavor chain:  31%|███▏      | 10/32 [00:33<01:13,  3.35s/it]",
  "metrics": {
    "predict_time": 35.386679,
    "total_time": 109.949722
  },
  "output": "a painting of a field with green grass, by Vincent Van Gogh, featured on deviantart, windy day, virtuosic level detail, the front of a trading card, loosely cropped, expansive, tendrils in the background, photo courtesy museum of art, 1980s art, standing on a hill, creative commons attribution",
  "started_at": "2022-11-28T01:25:38.859722Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/mfj4quetsfcuhk24dtit2sbdpe",
    "cancel": "https://api.replicate.com/v1/predictions/mfj4quetsfcuhk24dtit2sbdpe/cancel"
  },
  "version": "f7ec9b39f3e37a723fd9549a2183905f5a8d32357346972de6ecd804614f0fd7"
}

Generated in

35.4 seconds

Tweak it Share Report

Want to make some of these yourself?

Run this model

pharmapsychotic / clip-interrogator

Prediction

Input

Output

Prediction

Input

Output

Logs (6iveqabqxrf55lw2xez2owwn6e)

Logs (mfj4quetsfcuhk24dtit2sbdpe)