laion-ai / deep-image-diffusion-prior

Generate an image using text by visualizing CLIP features.

  • Public
  • 1.1K runs
  • T4
  • GitHub
  • License

Input

string
Shift + Return to add a new line

Prompt to generate

Default: ""

string

Offset type

Default: "none"

integer
(minimum: 1, maximum: 10)

Number of scales

Default: 6

number
(minimum: 0, maximum: 1)

Strength of input noise

Default: 0

number
(minimum: 0, maximum: 10)

Learning rate

Default: 0.001

number
(minimum: 0, maximum: 10)

Learning rate factor for offset

Default: 1

number
(minimum: 0, maximum: 1)

Learning rate decay

Default: 0.995

number
(minimum: 0, maximum: 1)

Strength of parameter noise

Default: 0

integer
(minimum: 0, maximum: 100)

Display frequency

Default: 25

integer
(minimum: 0, maximum: 1000)

Number of iterations

Default: 250

integer
(minimum: 1, maximum: 10)

Number of samples per batch

Default: 2

integer
(minimum: 4, maximum: 32)

Number of cutouts

Default: 8

number
(minimum: 0, maximum: 10)

Scale of conditioning

Default: 1

integer
(minimum: -1, maximum: 100000)

Random seed

Default: -1

Output

output
Generated in

Run time and cost

This model costs approximately $0.080 to run on Replicate, or 12 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia T4 GPU hardware. Predictions typically complete within 6 minutes. The predict time for this model varies significantly based on the inputs.

Readme

Deep Image Diffusion Prior

by @nousr

Model description

Inverts CLIP text embeds to image embeds and visualizes with deep-image-prior.

Acknowledgements

Code and weights by @nousr, with help from:

  • LAION for support, resources, and community

  • @RiversHaveWings for making me aware of this technique

  • Stability AI for compute which makes these models possible

  • lucidrains for spearheading the open-source replication of DALLE 2

Just to avoid any confusion, this research is a recreation of (one part of) OpenAI’s DALLE2 paper. It is not, “DALLE2”, the product/service from OpenAI you may have seen on the web.

Intended use

See the world “through CLIP’s eyes” by taking advantage of the diffusion prior as replicated by Laion to invert CLIP “ViT-L/14” text embeds to image embeds (as in unCLIP/DALLE2). After, a process known as deep-image-prior developed by Katherine Crowson is run to visualize the features in CLIP’s weights corresponding to activations from your prompt.

Caveats and recommendations

These visualizations can be quite abstract compared to other text-2-image models. However, you can often find a sort of dream like quality due to this. Many outputs are artistically fantastic because of this, but whether or not the visual matches your prompt as often is another matter.