andreasjansson / flash-eval

Suite of models to evaluate the image quality of text-to-image models with respect to their input prompts. (Updated 8 months ago)

Cold

Public
1.9K runs
A100 (80GB)
GitHub
Paper

Run with an API

Playground API Examples README Versions

Prediction

andreasjansson/flash-eval:40774574b8be084585513ddb4762de5125e869f28ab5e4ea7f6c689d9c75507d

Model

andreasjansson/flash-eval:40774574

3r68csevc5rj60cjdj2975mv6w

Status

Succeeded

Source

Web

Hardware

A100 (80GB)

Total duration

2m 24s

Created

8 months ago

Input

models: ImageReward,Aesthetic,CLIP,BLIP,PickScore
prompts_and_images: a cat: https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp,https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png a dog: https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp,https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png

{
  "models": "ImageReward,Aesthetic,CLIP,BLIP,PickScore",
  "prompts_and_images": "a cat: https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp,https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png\na dog: https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp,https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png"
}

Install Replicate’s Node.js client library:

npm install replicate

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run andreasjansson/flash-eval using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "andreasjansson/flash-eval:40774574b8be084585513ddb4762de5125e869f28ab5e4ea7f6c689d9c75507d",
  {
    input: {
      models: "ImageReward,Aesthetic,CLIP,BLIP,PickScore",
      prompts_and_images: "a cat: https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp,https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png\na dog: https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp,https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png"
    }
  }
);

console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Import the client:

import replicate

Run andreasjansson/flash-eval using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "andreasjansson/flash-eval:40774574b8be084585513ddb4762de5125e869f28ab5e4ea7f6c689d9c75507d",
    input={
        "models": "ImageReward,Aesthetic,CLIP,BLIP,PickScore",
        "prompts_and_images": "a cat: https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp,https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png\na dog: https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp,https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png"
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Run andreasjansson/flash-eval using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "andreasjansson/flash-eval:40774574b8be084585513ddb4762de5125e869f28ab5e4ea7f6c689d9c75507d",
    "input": {
      "models": "ImageReward,Aesthetic,CLIP,BLIP,PickScore",
      "prompts_and_images": "a cat: https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp,https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png\\na dog: https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp,https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png"
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

{ "a cat": { "https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png": { "BLIP": 0.4326244592666626, "CLIP": 0.2218017578125, "Aesthetic": 6.099793434143066, "PickScore": 21.34879493713379, "ImageReward": 0.3194262385368347 }, "https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp": { "BLIP": 0.35405826568603516, "CLIP": 0.245849609375, "Aesthetic": 5.96907901763916, "PickScore": 21.581308364868164, "ImageReward": 0.637330174446106 } }, "a dog": { "https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png": { "BLIP": 0.35565632581710815, "CLIP": 0.181884765625, "Aesthetic": 5.137001991271973, "PickScore": 19.575040817260742, "ImageReward": -0.915699303150177 }, "https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp": { "BLIP": 0.399705708026886, "CLIP": 0.2403564453125, "Aesthetic": 6.546636581420898, "PickScore": 21.414165496826172, "ImageReward": 1.6094799041748047 } } }

{
  "completed_at": "2024-10-08T14:27:23.314034Z",
  "created_at": "2024-10-08T14:24:59.233000Z",
  "data_removed": false,
  "error": null,
  "id": "3r68csevc5rj60cjdj2975mv6w",
  "input": {
    "models": "ImageReward,Aesthetic,CLIP,BLIP,PickScore",
    "prompts_and_images": "a cat: https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp,https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png\na dog: https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp,https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png"
  },
  "logs": "\"a cat\"\nhttps://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp\nImageReward: 0.6373  (took 0.256sec)\nAesthetic : 5.9691  (took 0.193sec)\nCLIP      : 0.2458  (took 0.109sec)\nBLIP      : 0.3541  (took 0.098sec)\nPickScore : 21.5813  (took 0.0812sec)\nhttps://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png\nImageReward: 0.3194  (took 0.109sec)\nAesthetic : 6.0998  (took 0.0923sec)\nCLIP      : 0.2218  (took 0.0985sec)\nBLIP      : 0.4326  (took 0.103sec)\nPickScore : 21.3488  (took 0.0753sec)\n\"a dog\"\nhttps://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp\nImageReward: 1.6095  (took 0.105sec)\nAesthetic : 6.5466  (took 0.0752sec)\nCLIP      : 0.2404  (took 0.0403sec)\nBLIP      : 0.3997  (took 0.0446sec)\nPickScore : 21.4142  (took 0.0654sec)\nhttps://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png\nImageReward: -0.9157  (took 0.0527sec)\nAesthetic : 5.1370  (took 0.0362sec)\nCLIP      : 0.1819  (took 0.043sec)\nBLIP      : 0.3557  (took 0.0467sec)\nPickScore : 19.5750  (took 0.0682sec)",
  "metrics": {
    "predict_time": 2.671128577,
    "total_time": 144.081034
  },
  "output": {
    "a cat": {
      "https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png": {
        "BLIP": 0.4326244592666626,
        "CLIP": 0.2218017578125,
        "Aesthetic": 6.099793434143066,
        "PickScore": 21.34879493713379,
        "ImageReward": 0.3194262385368347
      },
      "https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp": {
        "BLIP": 0.35405826568603516,
        "CLIP": 0.245849609375,
        "Aesthetic": 5.96907901763916,
        "PickScore": 21.581308364868164,
        "ImageReward": 0.637330174446106
      }
    },
    "a dog": {
      "https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png": {
        "BLIP": 0.35565632581710815,
        "CLIP": 0.181884765625,
        "Aesthetic": 5.137001991271973,
        "PickScore": 19.575040817260742,
        "ImageReward": -0.915699303150177
      },
      "https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp": {
        "BLIP": 0.399705708026886,
        "CLIP": 0.2403564453125,
        "Aesthetic": 6.546636581420898,
        "PickScore": 21.414165496826172,
        "ImageReward": 1.6094799041748047
      }
    }
  },
  "started_at": "2024-10-08T14:27:20.642906Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/3r68csevc5rj60cjdj2975mv6w",
    "cancel": "https://api.replicate.com/v1/predictions/3r68csevc5rj60cjdj2975mv6w/cancel"
  },
  "version": "40774574b8be084585513ddb4762de5125e869f28ab5e4ea7f6c689d9c75507d"
}

Generated in

2.7 seconds

Tweak it Share Report View full prediction

Want to make some of these yourself?

Run this model