Prompt-to-Prompt

Stable Diffusion Implementation

Code for the demo is here https://github.com/chenxwh/prompt-to-prompt

Tips for the demo input above:

Prompt-to-prompt enables editing a stable-diffusion generated image original_image, generated with original_prompt, by editing the original_prompt only edited_prompt.

If you do not already have an original_image - original_prompt pair to play around with the editing, you can generate one by only giving value for original_prompt, set to None. It is best to set a seed (or remember the random seed assigned), which will be used for generating images with edited_prompt .

Now with original_prompt (and original_image in mind), there are three options for editing the prompt. Refer to the instructions below for each type of the editing. If you choose Re-weight, in the edited_prompt field only provide the weights assigned for words from the original_prompt, in the format of [list of words] | [list of weights]. The example gallery may come helpful!

Additionally, there is the local_edit option, in the format of words in original_prompt | words in edited_prompt, which allows you to specify the only words (semantics) that will be edited.

Prompt Edits

In our notebooks, we perform our main logic by implementing the abstract class AttentionControl object, of the following form:

class AttentionControl(abc.ABC):
    @abc.abstractmethod
    def forward (self, attn, is_cross: bool, place_in_unet: str):
        raise NotImplementedError

The forward method is called in each attention layer of the diffusion model during the image generation, and we use it to modify the weights of the attention. Our method (See Section 3 of our paper) edits images with the procedure above, and each different prompt edit type modifies the weights of the attention in a different manner.

Replacement

In this case, the user swaps tokens of the original prompt with others, e.g., the editing the prompt "A painting of a squirrel eating a burger" to "A painting of a squirrel eating a lasagna" or "A painting of a lion eating a burger". For this we define the class AttentionReplace.

Refinement

In this case, the user adds new tokens to the prompt, e.g., editing the prompt "A painting of a squirrel eating a burger" to "A watercolor painting of a squirrel eating a burger". For this we define the class AttentionEditRefine.

Re-weight

In this case, the user changes the weight of certain tokens in the prompt, e.g., for the prompt "A photo of a poppy field at night", strengthen or weaken the extent to which the word night affects the resulting image. For this we define the class AttentionReweight.

Attention Control Options

• cross_replace_steps: specifies the fraction of steps to edit the cross attention maps. Can also be set to a dictionary [str:float] which specifies fractions for different words in the prompt.
• self_replace_steps: specifies the fraction of steps to replace the self attention maps.
• local_blend (optional): LocalBlend object which is used to make local edits. LocalBlend is initialized with the words from each prompt that correspond with the region in the image we want to edit.
• equalizer: used for attention Re-weighting only. A vector of coefficients to multiply each cross-attention weight

Citation

@article{hertz2022prompt,
  title = {Prompt-to-Prompt Image Editing with Cross Attention Control},
  author = {Hertz, Amir and Mokady, Ron and Tenenbaum, Jay and Aberman, Kfir and Pritch, Yael and Cohen-Or, Daniel},
  journal = {arXiv preprint arXiv:2208.01626},
  year = {2022},
}

Disclaimer

This is not an officially supported Google product.