CLIP Prompt Engineering for Generative Art

A primer on a what might be the most valuable skillset in the coming years, beyond just generative art.

What is CLIP?

CLIP is a tool for generating images from a generative model based on text.

Choosing your generative model to use with CLIP?

CLIP is distinct from the generative model itself, and is not limited to a single model.

For example, consider The PixelDraw-guided CLIP implementation. It’s great for generating the kinds of images you’d expect to be used as assets in 16-bit or faux-retro games. For example, consider the Loot NFT project, which is built around providing prompts for things like AI-generated art:

If we feed these prompts into our PixelDraw-guided CLIP implementation, we can can get the following:

“Grim Shout” Grave Wand of Skill +1

Hard Leather Armor

Divine Hood

Hard Leather Belt

“Death Root” Ornate Greaves of Skill

Studded Leather Gloves

Necklace of Enlightenment

Gold Ring

However, for the purposes of this post I will be focusing on the outputs of a model that combines a conditional ImageNet classifier with OpenAI’s unconditional generative diffusion model (courtesy of @RiversHaveWings).

Being mindful of seed values

Consider the following examples made with the prompt "the apotheosis of the lunatic, by William Blake". This was inspired by @RiversHaveWings’s prompt “the apotheosis of the lunatic, by James Gurney”. I switched out James Gurney for William Blake, because if you’re going to use a word like “apotheosis” in your prompt, it might be interesting to try it in the style of an artist that made some of the most famous post-renaissance artwork out there. This is what our diffusion-based model outputs based on a seed value of 0:

prompt `“the apotheosis of the lunatic, by William Blake”`, with seed value of `0`

This is a pretty striking result. If you look at William Blake’s works, you’ll see plenty of visual similarities.

But this is just one possible output of the model with this prompt. Here are some more examples, but with different seed values:

prompt `“the apotheosis of the lunatic, by William Blake”`, with seed value of `1337`

prompt `“the apotheosis of the lunatic, by William Blake”`, with seed value of `80085`

prompt `“the apotheosis of the lunatic, by William Blake”`, with seed value of `42069`

prompt `“the apotheosis of the lunatic, by William Blake”`, with seed value of `113553`

By changing just one number, we can get an enormous variety of different yet stylistically similar outputs.

Where is this useful outside of generative art?

This technique gained notoriety in the Generative Art community last year with the announcement of DALL-E from OpenAI, but before that it had been used in getting the most out of language models like GPT-2 and GPT-3.

Already, OpenAI has been working on a code-specific version of GPT-3 known as OpenAI Codex. This has already been rolled out in product form as Github Copilot (though OpenAI also has a beta for using the codex API directly). Codex is still a long way from replacing software engineers, but it’s worth paying attentiont to how much GPT-3 models have advanced in just a few years. OpenAI is already working on GPT-4. While GPT-4 will not be too different in size from GPT-3, it will incorporate newer improvements in model-training specifically with code-generation in mind.

Ultimately, exercising your pompt engineering skills in generative art might be a precursor to becoming a skilled prompt engineer for AI-assisted coding.

References

Cited as:

@article{mcateer2021clippe,
    title = "CLIP Prompt Engineering for Generative Art",
    author = "McAteer, Matthew",
    journal = "matthewmcateer.me",
    year = "2021",
    url = "https://matthewmcateer.me/blog/clip-prompt-engineering/"
}

If you notice mistakes and errors in this post, don’t hesitate to contact me at [contact at matthewmcateer dot me] and I will be very happy to correct them right away! Alternatily, you can follow me on Twitter and reach out to me there.

See you in the next post 😄

I write about AI, Biotech, and a bunch of other topics. Subscribe to get new posts by email!


This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

At least this isn't a full-screen popup

That'd be more annoying. Anyways, subscribe to my newsletter to get new posts by email! I write about AI, Biotech, and a bunch of other topics.


This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.