andreasjansson/flash-eval – Run with an API on Replicate

andreasjansson / flash-eval

Suite of models to evaluate the image quality of text-to-image models with respect to their input prompts.

Cold

Public
1.9K runs
A100 (80GB)
GitHub
Paper

Run with an API

Playground API Examples README Versions

Input

prompts_and_images

*string

Shift + Return to add a new line

a cat: https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp,https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png
a dog: https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp,https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.pnga cat: https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp,https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png
a dog: https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp,https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png

Newline-separated list of prompt/image URL pairs. Prompt/image URL pairs are formatted as <prompt>:<image1>[,<image2>[,<image3>[,...]]], i.e. a prompt followed by a colon followed by one or more comma-separated image URLs.

models

string

Shift + Return to add a new line

ImageReward,Aesthetic,CLIP,BLIP,PickScoreImageReward,Aesthetic,CLIP,BLIP,PickScore

Comma-separated list of models to use to evaluate. Valid models are ImageReward, Aesthetic, CLIP, BLIP, PickScore

Default: "ImageReward,Aesthetic,CLIP,BLIP,PickScore"

prompt_images_separator

string

Shift + Return to add a new line

Separator between prompt and list of images

Default: ":"

image_separator

string

Shift + Return to add a new line

Separator between image URLs

Default: ","

Run this model in Node.js with one line of code:

npx create-replicate --model=andreasjansson/flash-eval

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run andreasjansson/flash-eval using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "andreasjansson/flash-eval:ef9f9879404379fd05006e888a6bddcab189914c1045b77cbabc565735164ce9",
  {
    input: {
      models: "ImageReward,Aesthetic,CLIP,BLIP,PickScore",
      image_separator: ",",
      prompts_and_images: "a cat: https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp,https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png\na dog: https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp,https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png",
      prompt_images_separator: ":"
    }
  }
);

console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run andreasjansson/flash-eval using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "andreasjansson/flash-eval:ef9f9879404379fd05006e888a6bddcab189914c1045b77cbabc565735164ce9",
    input={
        "models": "ImageReward,Aesthetic,CLIP,BLIP,PickScore",
        "image_separator": ",",
        "prompts_and_images": "a cat: https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp,https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png\na dog: https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp,https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png",
        "prompt_images_separator": ":"
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run andreasjansson/flash-eval using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "andreasjansson/flash-eval:ef9f9879404379fd05006e888a6bddcab189914c1045b77cbabc565735164ce9",
    "input": {
      "models": "ImageReward,Aesthetic,CLIP,BLIP,PickScore",
      "image_separator": ",",
      "prompts_and_images": "a cat: https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp,https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png\\na dog: https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp,https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png",
      "prompt_images_separator": ":"
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

{ "a cat": { "https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png": { "BLIP": 0.4326244592666626, "CLIP": 0.2218017578125, "Aesthetic": 6.099793434143066, "PickScore": 21.34879493713379, "ImageReward": 0.3194262385368347 }, "https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp": { "BLIP": 0.35405826568603516, "CLIP": 0.245849609375, "Aesthetic": 5.96907901763916, "PickScore": 21.581308364868164, "ImageReward": 0.637330174446106 } }, "a dog": { "https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png": { "BLIP": 0.35565632581710815, "CLIP": 0.181884765625, "Aesthetic": 5.137001991271973, "PickScore": 19.575040817260742, "ImageReward": -0.915699303150177 }, "https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp": { "BLIP": 0.399705708026886, "CLIP": 0.2403564453125, "Aesthetic": 6.546636581420898, "PickScore": 21.414165496826172, "ImageReward": 1.6094799041748047 } } }

{
  "completed_at": "2024-10-08T14:27:23.314034Z",
  "created_at": "2024-10-08T14:24:59.233000Z",
  "data_removed": false,
  "error": null,
  "id": "3r68csevc5rj60cjdj2975mv6w",
  "input": {
    "models": "ImageReward,Aesthetic,CLIP,BLIP,PickScore",
    "prompts_and_images": "a cat: https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp,https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png\na dog: https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp,https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png"
  },
  "logs": "\"a cat\"\nhttps://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp\nImageReward: 0.6373  (took 0.256sec)\nAesthetic : 5.9691  (took 0.193sec)\nCLIP      : 0.2458  (took 0.109sec)\nBLIP      : 0.3541  (took 0.098sec)\nPickScore : 21.5813  (took 0.0812sec)\nhttps://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png\nImageReward: 0.3194  (took 0.109sec)\nAesthetic : 6.0998  (took 0.0923sec)\nCLIP      : 0.2218  (took 0.0985sec)\nBLIP      : 0.4326  (took 0.103sec)\nPickScore : 21.3488  (took 0.0753sec)\n\"a dog\"\nhttps://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp\nImageReward: 1.6095  (took 0.105sec)\nAesthetic : 6.5466  (took 0.0752sec)\nCLIP      : 0.2404  (took 0.0403sec)\nBLIP      : 0.3997  (took 0.0446sec)\nPickScore : 21.4142  (took 0.0654sec)\nhttps://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png\nImageReward: -0.9157  (took 0.0527sec)\nAesthetic : 5.1370  (took 0.0362sec)\nCLIP      : 0.1819  (took 0.043sec)\nBLIP      : 0.3557  (took 0.0467sec)\nPickScore : 19.5750  (took 0.0682sec)",
  "metrics": {
    "predict_time": 2.671128577,
    "total_time": 144.081034
  },
  "output": {
    "a cat": {
      "https://replicate.delivery/yhqm/M3MzBpeWPfg0qkNZcK1x4dMXr2boczHOqTHsxnEjtauJAvkTA/out-0.png": {
        "BLIP": 0.4326244592666626,
        "CLIP": 0.2218017578125,
        "Aesthetic": 6.099793434143066,
        "PickScore": 21.34879493713379,
        "ImageReward": 0.3194262385368347
      },
      "https://replicate.delivery/czjl/49VkOf3fu7sfgon0W2OMf5dlIxiwUpsBAicD1lveJP0e2cL5E/output.webp": {
        "BLIP": 0.35405826568603516,
        "CLIP": 0.245849609375,
        "Aesthetic": 5.96907901763916,
        "PickScore": 21.581308364868164,
        "ImageReward": 0.637330174446106
      }
    },
    "a dog": {
      "https://replicate.delivery/yhqm/yzKdMfHFMeq14kceu334KavdzLMcETTHiC12E2iMkfNgS8SOB/out-0.png": {
        "BLIP": 0.35565632581710815,
        "CLIP": 0.181884765625,
        "Aesthetic": 5.137001991271973,
        "PickScore": 19.575040817260742,
        "ImageReward": -0.915699303150177
      },
      "https://replicate.delivery/czjl/c2e693uAGuWIe0o6ZOqFF3R5uJt5hq7SPSsrkgfA6dz1JeSOB/output.webp": {
        "BLIP": 0.399705708026886,
        "CLIP": 0.2403564453125,
        "Aesthetic": 6.546636581420898,
        "PickScore": 21.414165496826172,
        "ImageReward": 1.6094799041748047
      }
    }
  },
  "started_at": "2024-10-08T14:27:20.642906Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/3r68csevc5rj60cjdj2975mv6w",
    "cancel": "https://api.replicate.com/v1/predictions/3r68csevc5rj60cjdj2975mv6w/cancel"
  },
  "version": "40774574b8be084585513ddb4762de5125e869f28ab5e4ea7f6c689d9c75507d"
}

Generated in

2.7 seconds

Tweak it Share Report

This output was created using a different version of the model, andreasjansson/flash-eval:40774574.

Run time and cost

This model costs approximately $0.37 to run on Replicate, or 2 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia A100 (80GB) GPU hardware. Predictions typically complete within 5 minutes. The predict time for this model varies significantly based on the inputs.

Readme

A suite of models to evaluate the image quality of text-to-image models with respect to their input prompts.

Github repo: https://github.com/thu-nics/FlashEval

Models

CLIP

Measures text-image alignment using CLIP. It evaluates how well the image matches the given text prompt.
Link

BLIP

Evaluates text-to-image alignment using BLIP, assessing how well the image matches the text description.
Link

Aesthetic

Assesses the aesthetic quality of an image, predicting how visually appealing it is to humans.
Link

ImageReward

A human preference model that predicts which images humans would prefer based on the given prompt.
Link

PickScore

A human preference model that predicts which images humans would prefer based on the given prompt.
Link