cjwbw / hard-prompts-made-easy

Gradient-Based Discrete Optimization for Prompt Tuning and Discovery

Cold

Public
650 runs
A100 (80GB)
GitHub
Paper
License

Run with an API

Playground API Examples README Versions

Input

image

file

Input image.

image_url

string

Shift + Return to add a new line

https://a.1stdibscdn.com/alexander-averin-paintings-pony-riding-on-the-beach-post-impressionist-style-oil-painting-for-sale-picture-6/a_7443/a_28523631526593507117/Promenade_detalle_5_master.JPG?disable=upscale&auto=webp&quality=60&width=1318https://a.1stdibscdn.com/alexander-averin-paintings-pony-riding-on-the-beach-post-impressionist-style-oil-painting-for-sale-picture-6/a_7443/a_28523631526593507117/Promenade_detalle_5_master.JPG?disable=upscale&auto=webp&quality=60&width=1318

Alternatively, you can provide a url link for an image. Ignored when image is uploaded.

customised_prompt

string

Shift + Return to add a new line

Optional input prompt. By default, a prompt will be learnt by the model for the input image and will be used for generation, but you can also provide your own customised prompt.

num_inference_steps

integer

(minimum: 1, maximum: 500)

Number of denoising steps

Default: 25

guidance_scale

number

(minimum: 1, maximum: 20)

Scale for classifier-free guidance

Default: 9

scheduler

string

Choose a scheduler.

Default: "DPMSolverMultistep"

Run this model in Node.js with one line of code:

npx create-replicate --model=cjwbw/hard-prompts-made-easy

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run cjwbw/hard-prompts-made-easy using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "cjwbw/hard-prompts-made-easy:40dc3d4f02fa895b3835fe669d83f196724cbe1bd6298a8dd951494221f43686",
  {
    input: {
      image_url: "https://a.1stdibscdn.com/alexander-averin-paintings-pony-riding-on-the-beach-post-impressionist-style-oil-painting-for-sale-picture-6/a_7443/a_28523631526593507117/Promenade_detalle_5_master.JPG?disable=upscale&auto=webp&quality=60&width=1318",
      scheduler: "DPMSolverMultistep",
      guidance_scale: 9,
      num_inference_steps: 50
    }
  }
);

console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run cjwbw/hard-prompts-made-easy using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "cjwbw/hard-prompts-made-easy:40dc3d4f02fa895b3835fe669d83f196724cbe1bd6298a8dd951494221f43686",
    input={
        "image_url": "https://a.1stdibscdn.com/alexander-averin-paintings-pony-riding-on-the-beach-post-impressionist-style-oil-painting-for-sale-picture-6/a_7443/a_28523631526593507117/Promenade_detalle_5_master.JPG?disable=upscale&auto=webp&quality=60&width=1318",
        "scheduler": "DPMSolverMultistep",
        "guidance_scale": 9,
        "num_inference_steps": 50
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run cjwbw/hard-prompts-made-easy using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "cjwbw/hard-prompts-made-easy:40dc3d4f02fa895b3835fe669d83f196724cbe1bd6298a8dd951494221f43686",
    "input": {
      "image_url": "https://a.1stdibscdn.com/alexander-averin-paintings-pony-riding-on-the-beach-post-impressionist-style-oil-painting-for-sale-picture-6/a_7443/a_28523631526593507117/Promenade_detalle_5_master.JPG?disable=upscale&auto=webp&quality=60&width=1318",
      "scheduler": "DPMSolverMultistep",
      "guidance_scale": 9,
      "num_inference_steps": 50
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

best_prompt

fulham children weymouth seaside gita octane equestrian artforsale artists surya pino victorian impressionism romantic impressionist westend

original_image

generated_images

{
  "completed_at": "2023-02-12T16:20:04.491361Z",
  "created_at": "2023-02-12T16:17:03.257235Z",
  "data_removed": false,
  "error": null,
  "id": "wsgaei3xuzcqnfs6fpz4sllx64",
  "input": {
    "image_url": "https://a.1stdibscdn.com/alexander-averin-paintings-pony-riding-on-the-beach-post-impressionist-style-oil-painting-for-sale-picture-6/a_7443/a_28523631526593507117/Promenade_detalle_5_master.JPG?disable=upscale&auto=webp&quality=60&width=1318",
    "scheduler": "DPMSolverMultistep",
    "guidance_scale": 9,
    "num_inference_steps": "50"
  },
  "logs": "step: 0, lr: 0.1, cosim: 0.136, text: publisher iche historicalmelville purcell foe fr inhifatally authorautomobiclassrooms chill ittleprawns ப\nstep: 100, lr: 0.1, cosim: 0.433, text: twinkle cuties spectators horseracing france primrose ltd impressionist auhorseback hd spectators painting beach seafopposite\nstep: 200, lr: 0.1, cosim: 0.379, text: children belle raffic demonstrate reviews mcenslump earnest eth sloane affair fineartamerica impressionist coastline marshmalharley\nstep: 300, lr: 0.1, cosim: 0.315, text: pupils reform chariot dressing mila oils yash burton �り�artistsoncarouoott horseracing portland troubles roseanne\nstep: 400, lr: 0.1, cosim: 0.441, text: children paints kensington onceuponedward paintings tdf donneinarte onetsy 🚩 thessal▬▬horses coastline malibu cymru\nstep: 500, lr: 0.1, cosim: 0.353, text: children pixelart trafalgar reinforleonardo vl chandler upstairs juliette automtunibartholpaddlebournemouth promenade leng\nstep: 600, lr: 0.1, cosim: 0.421, text: children epa voli aravimpressionist lesson osborne londonmarathon auction lourjommusique horse bournemouth acea 🔥🔗\nstep: 700, lr: 0.1, cosim: 0.359, text: children epa joanneflynn mucha wftv paintings makeinindia artist endeavcharleston tdf emphasibournemouth acea bicycle\nstep: 800, lr: 0.1, cosim: 0.470, text: juliette nats weymouth princesses downton juliette artist julian quarendeavwells ellen horses beach yokohama bicycle\nstep: 900, lr: 0.1, cosim: 0.454, text: nicolblossom weymouth children alecoscino orientimpressionist gerald warmhudson michael maya horse beach coastal bicycle\nstep: 1000, lr: 0.1, cosim: 0.427, text: joannekerala seaside children ⃣@ modernism mucha sor ktperljulian 🔥🔗 horse beach ½ jubilee\nstep: 1100, lr: 0.1, cosim: 0.258, text: vaus pier stgeorgepupils ⃣@ juliette sdcc george blueclane davidalecoscino southampton gail slatcarte\nstep: 1200, lr: 0.1, cosim: 0.392, text: horse oils paddington girls coomintereharcourt plein parasdubrovjennifer alecoscino paintings jyolyric�\nstep: 1300, lr: 0.1, cosim: 0.332, text: hanley tt seaside girls alecoscino sabaredwarpalette yoocelebrknuckalecoscino private alecoscino alecoscino jonathan\nstep: 1400, lr: 0.1, cosim: 0.376, text: horse edwarvenice children showblenparasantiques stgeorgepiccadhindus alecoscino artist alecoscino inspiring🚩\nstep: 1500, lr: 0.1, cosim: 0.402, text: horse ponies 🌊 children gallimpressionist parasonceuponfcblive stgeorgechildren alecoscino david weathercee alecoscino alecoscino\nstep: 1600, lr: 0.1, cosim: 0.406, text: horse victorian eastbourne sisters artistsondolldonneinarte antiques parasfertchildrenalecoscino main weathercee weathercee alecoscino\nstep: 1700, lr: 0.1, cosim: 0.306, text: british eid 🌊 gals janetguest rukh hagan impressionism paintings mermaids museumandyweathercee brun�️\nstep: 1800, lr: 0.1, cosim: 0.397, text: bathing ponies seaside girl braestevens impremishsuffravickchildren art andyweathercee weathercee alecoscino\nstep: 1900, lr: 0.1, cosim: 0.406, text: blonde igh weymouth children donald royo laghateparth joannebebe goodwood suffraart star weathercee plunarti\nstep: 2000, lr: 0.1, cosim: 0.352, text: promenade weymouth weymouth laghateparth clive cropped laghateparth dubrovnigan monet carriage art front weathercee tourist arti\nstep: 2100, lr: 0.1, cosim: 0.296, text: prima handicseaside liscavendish display harry plein braeqequestrian amelia front weathercee art art\nstep: 2200, lr: 0.1, cosim: 0.423, text: enipleasure seaside equestrian horseback harry phoebe ramsay rolls dolores children oprah fb weathercee art fine\nstep: 2300, lr: 0.1, cosim: 0.364, text: ponies promenade weymouth brighton whitehall lithograph pipe banning principeninsula peregrine oprah fb impending art jiu\nstep: 2400, lr: 0.1, cosim: 0.409, text: neve ponies waikiki equestrian madeleine mccann cyril almond fine zara yash �fb impending painting .\"-\nstep: 2500, lr: 0.1, cosim: 0.330, text: weymouth girls tapas yokohama hermann hoppaud gita edwarmango leighton voli fb victoria quanteau\nstep: 2600, lr: 0.1, cosim: 0.368, text: soyucuties weymouth weymouth clive grossdraujkstaged sienna suffraardo sunil victoria paintings eau\nstep: 2700, lr: 0.1, cosim: 0.392, text: ciara children beach brighton sureshpprabhu %nascardescripzara imp jane llesusan impressionist alty eid\nstep: 2800, lr: 0.1, cosim: 0.466, text: lottie girls weymouth weymouth nvfresco ponyoils artists jigsaw haydn victorian contemporary romantic artisan fiorentina\nstep: 2900, lr: 0.1, cosim: 0.435, text: boardwalk girls seaside weymouth premier equestrian ponies oils britannia parliamcuts fresco demonstration thomasjvskinny\nstep: 2999, lr: 0.1, cosim: 0.418, text: catalina children beach theartparliampixelart romantic paintings parliamambassadpixelart hd oil horses bonded short\nbest cosine sim: 0.505692183971405\nbest prompt: fulham children weymouth seaside gita octane equestrian artforsale artists surya pino victorian impressionism romantic impressionist westend\nGenerating images with prompt: fulham children weymouth seaside gita octane equestrian artforsale artists surya pino victorian impressionism romantic impressionist westend\n  0%|          | 0/50 [00:00<?, ?it/s]\n  4%|▍         | 2/50 [00:00<00:02, 17.35it/s]\n  8%|▊         | 4/50 [00:00<00:02, 17.82it/s]\n 12%|█▏        | 6/50 [00:00<00:02, 17.91it/s]\n 16%|█▌        | 8/50 [00:00<00:02, 18.04it/s]\n 20%|██        | 10/50 [00:00<00:02, 18.11it/s]\n 24%|██▍       | 12/50 [00:00<00:02, 18.13it/s]\n 28%|██▊       | 14/50 [00:00<00:01, 18.19it/s]\n 32%|███▏      | 16/50 [00:00<00:01, 17.94it/s]\n 36%|███▌      | 18/50 [00:00<00:01, 18.02it/s]\n 40%|████      | 20/50 [00:01<00:01, 17.98it/s]\n 44%|████▍     | 22/50 [00:01<00:01, 18.03it/s]\n 48%|████▊     | 24/50 [00:01<00:01, 18.14it/s]\n 52%|█████▏    | 26/50 [00:01<00:01, 18.22it/s]\n 56%|█████▌    | 28/50 [00:01<00:01, 18.26it/s]\n 60%|██████    | 30/50 [00:01<00:01, 18.24it/s]\n 64%|██████▍   | 32/50 [00:01<00:00, 18.25it/s]\n 68%|██████▊   | 34/50 [00:01<00:00, 18.12it/s]\n 72%|███████▏  | 36/50 [00:01<00:00, 18.13it/s]\n 76%|███████▌  | 38/50 [00:02<00:00, 17.91it/s]\n 80%|████████  | 40/50 [00:02<00:00, 17.99it/s]\n 84%|████████▍ | 42/50 [00:02<00:00, 18.04it/s]\n 88%|████████▊ | 44/50 [00:02<00:00, 18.05it/s]\n 92%|█████████▏| 46/50 [00:02<00:00, 18.14it/s]\n 96%|█████████▌| 48/50 [00:02<00:00, 18.23it/s]\n100%|██████████| 50/50 [00:02<00:00, 18.30it/s]\n100%|██████████| 50/50 [00:02<00:00, 18.10it/s]",
  "metrics": {
    "predict_time": 181.171184,
    "total_time": 181.234126
  },
  "output": {
    "best_prompt": "fulham children weymouth seaside gita octane equestrian artforsale artists surya pino victorian impressionism romantic impressionist westend ",
    "original_image": "https://replicate.delivery/pbxt/5tcF5wtUzfRUMiOE1sQwagMGj1PsyU4KjiRVl0eFL5jzOsdQA/orig_img.png",
    "generated_images": "https://replicate.delivery/pbxt/0RBhumGLhJJ4PZdiOhzYmMvF6fAza7PKVf0Eo7pKsunzOsdQA/out.png"
  },
  "started_at": "2023-02-12T16:17:03.320177Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/wsgaei3xuzcqnfs6fpz4sllx64",
    "cancel": "https://api.replicate.com/v1/predictions/wsgaei3xuzcqnfs6fpz4sllx64/cancel"
  },
  "version": "40dc3d4f02fa895b3835fe669d83f196724cbe1bd6298a8dd951494221f43686"
}

Generated in

3 minutes

Tweak itReport

step: 0, lr: 0.1, cosim: 0.136, text: publisher iche historicalmelville purcell foe fr inhifatally authorautomobiclassrooms chill ittleprawns ப
step: 100, lr: 0.1, cosim: 0.433, text: twinkle cuties spectators horseracing france primrose ltd impressionist auhorseback hd spectators painting beach seafopposite
step: 200, lr: 0.1, cosim: 0.379, text: children belle raffic demonstrate reviews mcenslump earnest eth sloane affair fineartamerica impressionist coastline marshmalharley
step: 300, lr: 0.1, cosim: 0.315, text: pupils reform chariot dressing mila oils yash burton �り�artistsoncarouoott horseracing portland troubles roseanne
step: 400, lr: 0.1, cosim: 0.441, text: children paints kensington onceuponedward paintings tdf donneinarte onetsy 🚩 thessal▬▬horses coastline malibu cymru
step: 500, lr: 0.1, cosim: 0.353, text: children pixelart trafalgar reinforleonardo vl chandler upstairs juliette automtunibartholpaddlebournemouth promenade leng
step: 600, lr: 0.1, cosim: 0.421, text: children epa voli aravimpressionist lesson osborne londonmarathon auction lourjommusique horse bournemouth acea 🔥🔗
step: 700, lr: 0.1, cosim: 0.359, text: children epa joanneflynn mucha wftv paintings makeinindia artist endeavcharleston tdf emphasibournemouth acea bicycle
step: 800, lr: 0.1, cosim: 0.470, text: juliette nats weymouth princesses downton juliette artist julian quarendeavwells ellen horses beach yokohama bicycle
step: 900, lr: 0.1, cosim: 0.454, text: nicolblossom weymouth children alecoscino orientimpressionist gerald warmhudson michael maya horse beach coastal bicycle
step: 1000, lr: 0.1, cosim: 0.427, text: joannekerala seaside children ⃣@ modernism mucha sor ktperljulian 🔥🔗 horse beach ½ jubilee
step: 1100, lr: 0.1, cosim: 0.258, text: vaus pier stgeorgepupils ⃣@ juliette sdcc george blueclane davidalecoscino southampton gail slatcarte
step: 1200, lr: 0.1, cosim: 0.392, text: horse oils paddington girls coomintereharcourt plein parasdubrovjennifer alecoscino paintings jyolyric�
step: 1300, lr: 0.1, cosim: 0.332, text: hanley tt seaside girls alecoscino sabaredwarpalette yoocelebrknuckalecoscino private alecoscino alecoscino jonathan
step: 1400, lr: 0.1, cosim: 0.376, text: horse edwarvenice children showblenparasantiques stgeorgepiccadhindus alecoscino artist alecoscino inspiring🚩
step: 1500, lr: 0.1, cosim: 0.402, text: horse ponies 🌊 children gallimpressionist parasonceuponfcblive stgeorgechildren alecoscino david weathercee alecoscino alecoscino
step: 1600, lr: 0.1, cosim: 0.406, text: horse victorian eastbourne sisters artistsondolldonneinarte antiques parasfertchildrenalecoscino main weathercee weathercee alecoscino
step: 1700, lr: 0.1, cosim: 0.306, text: british eid 🌊 gals janetguest rukh hagan impressionism paintings mermaids museumandyweathercee brun�️
step: 1800, lr: 0.1, cosim: 0.397, text: bathing ponies seaside girl braestevens impremishsuffravickchildren art andyweathercee weathercee alecoscino
step: 1900, lr: 0.1, cosim: 0.406, text: blonde igh weymouth children donald royo laghateparth joannebebe goodwood suffraart star weathercee plunarti
step: 2000, lr: 0.1, cosim: 0.352, text: promenade weymouth weymouth laghateparth clive cropped laghateparth dubrovnigan monet carriage art front weathercee tourist arti
step: 2100, lr: 0.1, cosim: 0.296, text: prima handicseaside liscavendish display harry plein braeqequestrian amelia front weathercee art art
step: 2200, lr: 0.1, cosim: 0.423, text: enipleasure seaside equestrian horseback harry phoebe ramsay rolls dolores children oprah fb weathercee art fine
step: 2300, lr: 0.1, cosim: 0.364, text: ponies promenade weymouth brighton whitehall lithograph pipe banning principeninsula peregrine oprah fb impending art jiu
step: 2400, lr: 0.1, cosim: 0.409, text: neve ponies waikiki equestrian madeleine mccann cyril almond fine zara yash �fb impending painting ."-
step: 2500, lr: 0.1, cosim: 0.330, text: weymouth girls tapas yokohama hermann hoppaud gita edwarmango leighton voli fb victoria quanteau
step: 2600, lr: 0.1, cosim: 0.368, text: soyucuties weymouth weymouth clive grossdraujkstaged sienna suffraardo sunil victoria paintings eau
step: 2700, lr: 0.1, cosim: 0.392, text: ciara children beach brighton sureshpprabhu %nascardescripzara imp jane llesusan impressionist alty eid
step: 2800, lr: 0.1, cosim: 0.466, text: lottie girls weymouth weymouth nvfresco ponyoils artists jigsaw haydn victorian contemporary romantic artisan fiorentina
step: 2900, lr: 0.1, cosim: 0.435, text: boardwalk girls seaside weymouth premier equestrian ponies oils britannia parliamcuts fresco demonstration thomasjvskinny
step: 2999, lr: 0.1, cosim: 0.418, text: catalina children beach theartparliampixelart romantic paintings parliamambassadpixelart hd oil horses bonded short
best cosine sim: 0.505692183971405
best prompt: fulham children weymouth seaside gita octane equestrian artforsale artists surya pino victorian impressionism romantic impressionist westend
Generating images with prompt: fulham children weymouth seaside gita octane equestrian artforsale artists surya pino victorian impressionism romantic impressionist westend
  0%|          | 0/50 [00:00<?, ?it/s]
  4%|▍         | 2/50 [00:00<00:02, 17.35it/s]
  8%|▊         | 4/50 [00:00<00:02, 17.82it/s]
 12%|█▏        | 6/50 [00:00<00:02, 17.91it/s]
 16%|█▌        | 8/50 [00:00<00:02, 18.04it/s]
 20%|██        | 10/50 [00:00<00:02, 18.11it/s]
 24%|██▍       | 12/50 [00:00<00:02, 18.13it/s]
 28%|██▊       | 14/50 [00:00<00:01, 18.19it/s]
 32%|███▏      | 16/50 [00:00<00:01, 17.94it/s]
 36%|███▌      | 18/50 [00:00<00:01, 18.02it/s]
 40%|████      | 20/50 [00:01<00:01, 17.98it/s]
 44%|████▍     | 22/50 [00:01<00:01, 18.03it/s]
 48%|████▊     | 24/50 [00:01<00:01, 18.14it/s]
 52%|█████▏    | 26/50 [00:01<00:01, 18.22it/s]
 56%|█████▌    | 28/50 [00:01<00:01, 18.26it/s]
 60%|██████    | 30/50 [00:01<00:01, 18.24it/s]
 64%|██████▍   | 32/50 [00:01<00:00, 18.25it/s]
 68%|██████▊   | 34/50 [00:01<00:00, 18.12it/s]
 72%|███████▏  | 36/50 [00:01<00:00, 18.13it/s]
 76%|███████▌  | 38/50 [00:02<00:00, 17.91it/s]
 80%|████████  | 40/50 [00:02<00:00, 17.99it/s]
 84%|████████▍ | 42/50 [00:02<00:00, 18.04it/s]
 88%|████████▊ | 44/50 [00:02<00:00, 18.05it/s]
 92%|█████████▏| 46/50 [00:02<00:00, 18.14it/s]
 96%|█████████▌| 48/50 [00:02<00:00, 18.23it/s]
100%|██████████| 50/50 [00:02<00:00, 18.30it/s]
100%|██████████| 50/50 [00:02<00:00, 18.10it/s]

Examples

View more examples

Run time and cost

This model costs approximately $0.48 to run on Replicate, or 2 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia A100 (80GB) GPU hardware. Predictions typically complete within 6 minutes. The predict time for this model varies significantly based on the inputs.

Readme

Hard Prompts Made Easy: Discrete Prompt Tuning for Language Models

This code is the official implementation of Hard Prompts Made Easy.

If you have any questions, feel free to email Yuxin (ywen@umd.edu).

About

From a given image, we first optimize a hard prompt using the PEZ algorithm and CLIP encoders. Then, we take the optimized prompts and feed them into Stable Diffusion to generate new images. The name PEZ (hard Prompts made EaZy) was inspired from the PEZ candy dispenser.