CLIP Prompt Engineering for Generative Art

A CLIP primer with 3500+ prompt output examples.

UPDATE 09/29/2021: This post gives high-level tips for propmpt engineering, but if you want to see more of this in practice, look to the One small step for GAN Instagram page. I cannot recommend this higly enough.

UPDATE 10/16/2021: Listed a claimed fix for one of CLIP’s bugs.

UPDATE 11/07/2021: Added new details about CLIP fine-tuned on datasets beyond ImageNet.

UPDATE 12/15/2021: If you’re just interested in the colab notebooks that allow you to create these images, I’ve added a section at the end with all the links (I still recommend reading through this full post).

What is CLIP?

CLIP is a transformer model from OpenAI that is used to match text embeddings with image embeddings. The motivation behind CLIP is simple enough: We can get transformers to make representations of text. We can get transformers to make representations of images. If we can get the image and text representations describing the same concepts close to each other, we can easily map images to text (or vice versa).

CLIP, neural search, generative models

For people who are previously used to one-hot encoding categorical variables and manually mapping those to human-intelligible text, CLIP almost seems like cheating.

One featur of this image-to-text mapping is that it makes neural search applications much easier to create for images and image databases. At the image level, it means we can combine CLIP with tools like YOLOv5 and search for objects in images using only descriptions. As an example, here’s the CROP-CLIP project, which uses CLIP to crop out subsets of images.

Crop-CLIP selects the clock from the image when presented with the text prompt “What’s the time?”

Unlike traditional ’Object Detection’, CLIP is not restricted to a predefined set of classes, or a top-down pipeline that requires bounding boxes or instance segmentations. Try it out yourself with @kevin_zakka ’s CLIP Google Colab

At the level of image databases, it means we can search for content even with vague descriptors. Take this example CLIP Search project that allows one to search for images based on the content of the image, rather than the image’s connection to high-traffic websites.

CLIP search presents images of women holding flowers when presented with the text prompt “Girl Holding Flowers”

Conceptually, we can extend this further. We can go beyond searching individual images or finite sets of pre-defined images. We can use CLIP to search the seemingly infinite latent space in which one could find every possible image that can be rendered with a given number of pixels. In short, CLIP makes it possible to guide a generative model to produce new images based on an input string corresponding to almost any possible subject.

For anyone that’s read Jorge Luis Borges’ The Library of Babel, you might be rightly suspicious of claims of easy navigation of infinite spaces. After all, this might seem like navigating infinite possible images has just been traded for navigating infinite possible text strings”. I have two pieces of reassuring news for you:

  1. The latent space is technically not infinite. For 16,777,216 possible hexadecimal RGB color codes, and and 256×256256 \times 256 pixel image (65,536 pixels in total), we could draw C16777216655367.393×10186229\mathrm{C}_{16777216}^{65536} \approx 7.393 \times 10^{186229} possible images. It’s just very large, and many parts are practically indistinguishable in the eyes of humans. It also helps that generative models can navigate this space via manifold interpolation (i.e., what shapes are closest to each other?) instead of relying on linear interpolation (i.e., what is the average between these two points?)
  2. There are plenty of useful and easily-teachable tricks for choosing the best prompts. If you understand the language that CLIP used for learning embeddings (e.g., the OpenAI CLIP was based on English text), you can tweak your prompts easily to find what you want.

    Choosing your generative model to use with CLIP.

CLIP is distinct from the generative model itself, and is not limited to a single model.

For example, consider The PixelDraw-guided CLIP implementation. It’s great for generating the kinds of images you’d expect to be used as assets in 16-bit or faux-retro games. Now consider the Loot NFT project, which is built around providing prompts for things like AI-generated art:

On the Left, the Loot project announcement tweet. On the Right, CLIP-guided pixel art generated from the lines of text in the tweet. Starting at the top right and going clockwise, “Grim Shout” Grave Wand of Skill +1, Hard Leather Armor, Divine Hood, Hard Leather Belt, “Death Root” Ornate Greaves of Skill, Studded Leather Gloves, Necklace of Enlightenment, and Gold Ring.

Another example, consider if we use a purpose-built generator like StyleGAN3, one that’s geared towards generating anything as long as it resembles a human face.

prompt “Upper-class twit, drawn by The New Yorker”. Multiple seed values were planned, but THIS was the first result. It seems the generator couldn’t decide between Jeremy Clarkson and James May from Top Gear/The Grand Tour…so it went with both of them at the same time.

This space becomes even more complex when we try to combine with visual effects like zooming.

However, for the purposes of this post I will be focusing on the outputs of a model that combines a conditional ImageNet classifier with OpenAI’s unconditional generative diffusion model (courtesy of @RiversHaveWings). If you are interested in the other approaches out there, here is a growing list of some of the latest methods:

TechniqueColab linkDescriptionAuthorsNotes
Attn-GANThe OG text-to-image generatornotebook by Derrick SchultzThe original text-to-image colab
Big SleepBigGAN controlled by CLIPRyan MurdockDemo Video
Quick CLIP Guided DiffusionFast CLIP/Guided Diffusion image generationby Katherine Crowson, Daniel Russell, et al.This is the technique we use throughout the rest of this post
S2ML Art GeneratorJustin Bennington
Zoetrope 5.5CLIP-VQGAN toolbearsharktopus

If you want to see the full extent of generators that can be combined with CLIP, you can check out this awesome Github repo

Stylistic Choices

Let’s start off by stressing the largest benefit of using CLIP to create generative art: You can create almost any conceivable style with it. CLIP was trained by pairing images with descriptive text taken from around the internet. A lot of that text had descriptors such as what artistic medium the image was drawn/rendered/created in, who the creator was, what company/organization created/released/published the image, etc.

Below is a demonstration of how CLIP modifies images of the same subject (either a "Mushroom", "Dragon", "Castle on a hill", or "Spaceship", things recognizable across many different cultures), but with more than 200+ different descriptors (prompts in the format "{subject} | {style}").

NOTE: A similar version of this exists for CLIP+VQGAN prompts, but I’ve chosen to use CLIP+Diffusion for this demo.

You can also click on all the thumbnail images below to see a higher resolution.

First, let’s consider how CLIP guides the diffusion process for 4 different subjects, without any style modifiers.

Mushroom Dragon Spaceship Castle on a Hill
(no
style
modifiers)
mushroom (no style modifiers) dragon (no style modifiers) spaceship (no style modifiers) castle on a hill (no style modifiers)

CLIP-based generative models can emulate an enormous variety of styles from many different periods and cultures.

Mushroom Dragon Spaceship Castle on a Hill
anime anime mushroom anime dragon anime spaceship anime castle on a hill
psychedelic psychedelic mushroom psychedelic dragon psychedelic spaceship psychedelic castle on a hill
Ukiyo-e Ukiyo-e mushroom Ukiyo-e dragon Ukiyo-e spaceship Ukiyo-e castle on a hill
Flemish
Baroque
Flemish Baroque mushroom Flemish Baroque dragon Flemish Baroque spaceship Flemish Baroque castle on a hill
Neolithic
cave
painting
mushroom Neolithic cave painting dragon Neolithic cave painting spaceship Neolithic cave painting castle on a hill Neolithic cave painting
renaissance
painting
mushroom renaissance painting dragon renaissance painting spaceship renaissance painting castle on a hill renaissance painting
constructivism constructivism mushroom constructivism dragon constructivism spaceship constructivism castle on a hill
pre-
Raphaelite
mushroom pre-Raphaelite dragon pre-Raphaelite spaceship pre-Raphaelite castle on a hill pre-Raphaelite
Impressionism Impressionism mushroom Impressionism dragon Impressionism spaceship Impressionism castle on a hill
glowing
neon
glowing neon mushroom glowing neon dragon glowing neon spaceship glowing neon castle on a hill
rough rough mushroom rough dragon rough spaceship rough castle on a hill
maximalist maximalist mushroom maximalist dragon maximalist spaceship maximalist castle on a hill
minimalist minimalist mushroom minimalist dragon minimalist spaceship minimalist castle on a hill
geometric geometric mushroom geometric dragon geometric spaceship geometric castle on a hill
cluttered cluttered mushroom cluttered dragon cluttered spaceship cluttered castle on a hill
dutch
golden
age
dutch golden age mushroom dutch golden age dragon dutch golden age spaceship dutch golden age castle on a hill
Egyptian
Art
Egyptian Art mushroom Egyptian Art dragon Egyptian Art spaceship Egyptian Art castle on a hill
Fauvism Fauvism mushroom Fauvism dragon Fauvism spaceship Fauvism castle on a hill
Art
Deco
Art Deco mushroom Art Deco dragon Art Deco spaceship Art Deco castle on a hill
Picasso Picasso mushroom Picasso dragon Picasso spaceship Picasso castle on a hill
Da Vinci Da Vinci mushroom Da Vinci dragon Da Vinci spaceship Da Vinci castle on a hill
Academic
Art
Academic Art mushroom Academic Art dragon Academic Art spaceship Academic Art castle on a hill
Photo-
collage
mushroom Photo-collage dragon Photo-collage spaceship Photo-collage castle on a hill Photo-collage
Cubism Cubism mushroom Cubism dragon Cubism spaceship Cubism castle on a hill
Surrealist mushroom Surrealist dragon Surrealist spaceship Surrealist castle on a hill Surrealist
DC
Comics
mushroom DC Comics dragon DC Comics spaceship DC Comics castle on a hill DC Comics
Marvel
Comics
mushroom Marvel Comics dragon Marvel Comics spaceship Marvel Comics castle on a hill Marvel Comics
wavy mushroom wavy dragon wavy spaceship wavy castle on a hill wavy
groovy groovy mushroom groovy dragon groovy spaceship groovy castle on a hill
movie
poster
movie poster mushroom movie poster dragon movie poster spaceship movie poster castle on a hill
pop
art
mushroom pop art dragon pop art spaceship pop art castle on a hill pop art
tattoo mushroom tattoo dragon tattoo spaceship tattoo castle on a hill tattoo
woodcut woodcut mushroom woodcut dragon woodcut spaceship woodcut castle on a hill
Fine
art
mushroom Fine art dragon Fine art spaceship Fine art castle on a hill Fine art
vaporwave vaporwave mushroom vaporwave dragon vaporwave spaceship vaporwave castle on a hill
seapunk seapunk mushroom seapunk dragon seapunk spaceship seapunk castle on a hill
cyberpunk cyberpunk mushroom cyberpunk dragon cyberpunk spaceship cyberpunk castle on a hill
nanopunk nanopunk mushroom nanopunk dragon nanopunk spaceship nanopunk castle on a hill
biopunk biopunk mushroom biopunk dragon biopunk spaceship biopunk castle on a hill
cyber
noir
cyber noir mushroom cyber noir dragon cyber noir spaceship cyber noir castle on a hill
steampunk steampunk mushroom steampunk dragon steampunk spaceship steampunk castle on a hill
clockpunk clockpunk mushroom clockpunk dragon clockpunk spaceship clockpunk castle on a hill
dieselpunk dieselpunk mushroom dieselpunk dragon dieselpunk spaceship dieselpunk castle on a hill
decopunk decopunk mushroom decopunk dragon decopunk spaceship decopunk castle on a hill
coalpunk coalpunk mushroom coalpunk dragon coalpunk spaceship coalpunk castle on a hill
atompunk atompunk mushroom atompunk dragon atompunk spaceship atompunk castle on a hill
steelpunk steelpunk mushroom steelpunk dragon steelpunk spaceship steelpunk castle on a hill
islandpunk islandpunk mushroom islandpunk dragon islandpunk spaceship islandpunk castle on a hill
oceanpunk oceanpunk mushroom oceanpunk dragon oceanpunk spaceship oceanpunk castle on a hill
rococopunk rococopunk mushroom rococopunk dragon rococopunk spaceship rococopunk castle on a hill
stonepunk stonepunk mushroom stonepunk dragon stonepunk spaceship stonepunk castle on a hill
cottagecore cottagecore mushroom cottagecore dragon cottagecore spaceship cottagecore castle on a hill
mythpunk mythpunk mushroom mythpunk dragon mythpunk spaceship mythpunk castle on a hill
raypunk raypunk mushroom raypunk dragon raypunk spaceship raypunk castle on a hill
nowpunk nowpunk mushroom nowpunk dragon nowpunk spaceship nowpunk castle on a hill
cyberprep cyberprep mushroom cyberprep dragon cyberprep spaceship cyberprep castle on a hill
postcyberpunk postcyberpunk mushroom postcyberpunk dragon postcyberpunk spaceship postcyberpunk castle on a hill
solarpunk solarpunk mushroom solarpunk dragon solarpunk spaceship solarpunk castle on a hill
lunarpunk lunarpunk mushroom lunarpunk dragon lunarpunk spaceship lunarpunk castle on a hill
elfpunk elfpunk mushroom elfpunk dragon elfpunk spaceship elfpunk castle on a hill
neonpunk neonpunk mushroom neonpunk dragon neonpunk spaceship neonpunk castle on a hill
Dark Academia mushroom Dark Academia dragon Dark Academia spaceship Dark Academia castle on a hill Dark Academia

If your medium of choice doesn’t belong to any specific time period, that’s doable as well.

Mushroom Dragon Spaceship Castle on a Hill
pencil
sketch
pencil sketch mushroom pencil sketch dragon pencil sketch spaceship pencil sketch castle on a hill
concept
art
mushroom concept art dragon concept art spaceship concept art castle on a hill concept art
flat
shading
flat shading mushroom flat shading dragon flat shading spaceship flat shading castle on a hill
ink
drawing
mushroom ink drawing dragon ink drawing spaceship ink drawing castle on a hill ink drawing
artwork mushroom artwork dragon artwork spaceship artwork castle on a hill artwork
oil on
canvas
mushroom oil on canvas dragon oil on canvas spaceship oil on canvas castle on a hill oil on canvas
Poster
art
Poster art mushroom Poster art dragon Poster art spaceship Poster art castle on a hill
Detailed
Painting
mushroom Detailed Painting dragon Detailed Painting spaceship Detailed Painting castle on a hill Detailed Painting
Storybook
illustration
mushroom Storybook illustration dragon Storybook illustration spaceship Storybook illustration castle on a hill Storybook illustration
charcoal
drawing
mushroom charcoal drawing dragon charcoal drawing spaceship charcoal drawing castle on a hill charcoal drawing
Watercolor Watercolor mushroom Watercolor dragon Watercolor spaceship Watercolor castle on a hill
child's
drawing
mushroom child's drawing dragon child's drawing spaceship child's drawing castle on a hill child's drawing
airbrush
art
mushroom airbrush art dragon airbrush art spaceship airbrush art castle on a hill airbrush art
dye-
transfer
mushroom dye-transfer dragon dye-transfer spaceship dye-transfer castle on a hill dye-transfer

Have a specific videogame in mind? CLIP has embeddings for specific consoles and rendering engines.

Mushroom Dragon Spaceship Castle on a hill
PS1
graphics
PS1 graphics mushroom PS1 graphics dragon PS1 graphics spaceship PS1 graphics castle on a hill
Xbox 360
graphics
Xbox 360 graphics mushroom Xbox 360 graphics dragon Xbox 360 graphics spaceship Xbox 360 graphics castle on a hill
flickering
light
flickering light mushroom flickering light dragon flickering light spaceship flickering light castle on a hill
Playstation
5
screenshot
Playstation 5 screenshot mushroom Playstation 5 screenshot dragon Playstation 5 screenshot spaceship Playstation 5 screenshot castle on a hill
Rendered
in
Cinema4D
mushroom Rendered in Cinema4D dragon Rendered in Cinema4D spaceship Rendered in Cinema4D castle on a hill Rendered in Cinema4D
Rendered
in
Maya
mushroom Rendered in Maya dragon Rendered in Maya spaceship Rendered in Maya castle on a hill Rendered in Maya
CryEngine CryEngine mushroom CryEngine dragon CryEngine spaceship CryEngine castle on a hill
Unreal
Engine
Unreal Engine mushroom Unreal Engine dragon Unreal Engine spaceship Unreal Engine castle on a hill
reimagined by
industrial
light and
magic
mushroom reimagined by industrial light and magic dragon reimagined by industrial light and magic spaceship reimagined by industrial light and magic castle on a hill reimagined by industrial light and magic
#vfxfriday mushroom #vfxfriday dragon #vfxfriday spaceship #vfxfriday castle on a hill #vfxfriday
CGSociety CGSociety mushroom CGSociety dragon CGSociety spaceship CGSociety castle on a hill
prerendered
graphics
prerendered graphics mushroom prerendered graphics  dragon prerendered graphics  spaceship prerendered graphics  castle on a hill
datamosh mushroom datamosh dragon datamosh spaceship datamosh castle on a hill datamosh
vray
tracing
mushroom vray tracing dragon vray tracing spaceship vray tracing castle on a hill vray tracing
pixel
perfect
pixel perfect mushroom pixel perfect dragon pixel perfect spaceship pixel perfect castle on a hill
3840x2160 mushroom dragon spaceship castle on a hill
RTX
on
mushroom dragon spaceship castle on a hill
4k
result
mushroom dragon spaceship castle on a hill
2D
game
art
mushroom dragon spaceship castle on a hill
Windows
XP
mushroom dragon spaceship castle on a hill
Tri-X
400
TX
mushroom dragon spaceship castle on a hill
ultra
HD
mushroom dragon spaceship castle on a hill
Volumetric
Lighting
mushroom dragon spaceship castle on a hill
Low
poly
mushroom dragon spaceship castle on a hill
HD
mod
mushroom dragon spaceship castle on a hill

You can even describe different materials and textures for the subject of your image.

Mushroom Dragon Spaceship Castle on a hill
marble
sculpture
mushroom dragon spaceship castle on a hill
Mixed
Media
mushroom dragon spaceship castle on a hill
Chalk
art
mushroom dragon spaceship castle on a hill
made of
beads
and yarn
mushroom dragon spaceship castle on a hill
made of
feathers
mushroom dragon spaceship castle on a hill
made of
crystals
mushroom dragon spaceship castle on a hill
made of
liquid
metal
mushroom dragon spaceship castle on a hill
made of
glass
mushroom made of glass dragon made of glass spaceship made of glass castle on a hill made of glass
made of
cardboard
mushroom made of cardboard dragon made of cardboard spaceship made of cardboard castle on a hill made of cardboard
made of
vines
mushroom dragon spaceship castle on a hill
made of
cheese
mushroom dragon spaceship castle on a hill
made of
flowers
mushroom dragon spaceship castle on a hill
made of
insects
mushroom dragon spaceship castle on a hill
made of
mist
mushroom dragon spaceship castle on a hill
made of
paperclips
mushroom dragon spaceship castle on a hill
made of
rubber
mushroom dragon spaceship castle on a hill
made of
plastic
mushroom dragon spaceship castle on a hill
made of
wire
mushroom dragon spaceship spaceship
made of
trash
mushroom_made_of_trash dragon_made_of_trash spaceship_made_of_trash castle_on_a_hill_made_of_trash
made of
wrought
iron
mushroom made of wrought iron dragon made of wrought iron spaceship made of wrought iron castle on a hill made of wrought iron
made of
all of
the above
mushroom dragon spaceship castle on a hill

One might not even know the specific name of an artistic school or movement. Just describing a time period by year or decade will do as well.

Mushroom Dragon Spaceship Castle on a hill
(1962)
directed by
cinematography
by
mushroom dragon spaceship castle on a hill
(2013)
directed by
cinematography
by
mushroom dragon spaceship castle on a hill
1380s,
1385
mushroom 1380s, 1385 dragon 1380s, 1385 spaceship 1380s, 1385 castle on a hill 1380s, 1385
1480s,
1485
mushroom 1480s, 1485 dragon 1480s, 1485 spaceship 1480s, 1485 castle on a hill 1480s, 1485
1580s,
1585
mushroom 1580s, 1585 dragon 1580s, 1585 spaceship 1580s, 1585 castle on a hill 1580s, 1585
1680s,
1685
mushroom 1680s, 1685 dragon 1680s, 1685 spaceship 1680s, 1685 castle on a hill 1680s, 1685
1780s,
1785
mushroom 1780s, 1785 dragon 1780s, 1785 spaceship 1780s, 1785 castle on a hill 1780s, 1785
1880s,
1885
mushroom 1880s, 1885 dragon 1880s, 1885 spaceship 1880s, 1885 castle on a hill 1880s, 1885
1890s,
1895
mushroom 1890s, 1895 dragon 1890s, 1895 spaceship 1890s, 1895 castle on a hill 1890s, 1895
1900s,
1905
mushroom 1900s, 1905 dragon 1900s, 1905 spaceship 1900s, 1905 castle on a hill 1900s, 1905
1910s,
1915
mushroom 1910s, 1915 dragon 1910s, 1915 spaceship 1910s, 1915 castle on a hill 1910s, 1915
1920s,
1925
mushroom 1920s, 1925 dragon 1920s, 1925 spaceship 1920s, 1925 castle on a hill 1920s, 1925
1930s,
1935
mushroom 1930s, 1935 dragon 1930s, 1935 spaceship 1930s, 1935 castle on a hill 1930s, 1935
1940s,
1945
mushroom 1940s, 1945 dragon 1940s, 1945 spaceship 1940s, 1945 castle on a hill 1940s, 1945
1950s,
1955
mushroom 1950s, 1955 dragon 1950s, 1955 spaceship 1950s, 1955 castle on a hill 1950s, 1955
1960s,
1965
mushroom 1960s, 1965 dragon 1960s, 1965 spaceship 1960s, 1965 castle on a hill 1960s, 1965
1970s,
1975
mushroom 1970s, 1975 dragon 1970s, 1975 spaceship 1970s, 1975 castle on a hill 1970s, 1975
1980s,
1985
mushroom 1980s, 1985 dragon 1980s, 1985 spaceship 1980s, 1985 castle on a hill 1980s, 1985
1990s,
1995
mushroom 1990s, 1995 dragon 1990s, 1995 spaceship 1990s, 1995 castle on a hill 1990s, 1995
2000s,
2005
mushroom 2000s, 2005 dragon 2000s, 2005 spaceship 2000s, 2005 castle on a hill 2000s, 2005
2010s,
2015
mushroom 2010s, 2015 dragon 2010s, 2015 spaceship 2010s, 2015 castle on a hill 2010s, 2015
2020s,
2025
mushroom 2020s, 2025 dragon 2020s, 2025 spaceship 2020s, 2025 castle on a hill 2020s, 2025
2030s,
2035
mushroom 2030s, 2035 dragon 2030s, 2035 spaceship 2030s, 2035 castle on a hill 2030s, 2035
2040s,
2045
mushroom 2040s, 2045 dragon 2040s, 2045 spaceship 2040s, 2045 castle on a hill 2040s, 2045

Though one needn’t be as specific as a definite year or decade. Another option would be to describe the style of a specific country, political faction, organization, or movement.

Mushroom Dragon Spaceship Castle on a hill
Soviet
propaganda
Soviet propaganda mushroom Soviet propaganda dragon Soviet propaganda spaceship Soviet propaganda castle on a hill
American
propaganda
American propaganda mushroom American propaganda dragon American propaganda spaceship American propaganda castle on a hill
British
Propaganda
mushroom British Propaganda dragon British Propaganda spaceship British Propaganda castle on a hill British Propaganda
Socialist
Realism
mushroom Socialist Realism dragon Socialist Realism spaceship Socialist Realism castle on a hill Socialist Realism
Chinese
Propaganda
mushroom Chinese Propaganda dragon Chinese Propaganda spaceship Chinese Propaganda castle on a hill Chinese Propaganda
Cold
War
Propaganda
mushroom Cold War Propaganda dragon Cold War Propaganda spaceship Cold War Propaganda castle on a hill Cold War Propaganda
French
Propaganda
mushroom French Propaganda dragon French Propaganda spaceship French Propaganda castle on a hill French Propaganda
Korean
Propaganda
mushroom Korean Propaganda dragon Korean Propaganda spaceship Korean Propaganda castle on a hill Korean Propaganda
NATO
Propaganda
mushroom NATO Propaganda dragon NATO Propaganda spaceship NATO Propaganda castle on a hill NATO Propaganda
WWI
Propaganda
mushroom WWI Propaganda dragon WWI Propaganda spaceship WWI Propaganda castle on a hill WWI Propaganda
WWII
Propaganda
mushroom WWII Propaganda dragon WWII Propaganda spaceship WWII Propaganda castle on a hill WWII Propaganda

When it comes to emulating styles of specific artists, you can go beyond famous ones like Van Gogh or Picasso.

Mushroom Dragon Spaceship Castle on a hill
by
James
Gurney
mushroom by James Gurney dragon by James Gurney spaceship by James Gurney castle on a hill by James Gurney
by
Greg
Rutkowski
mushroom by Greg Rutkowski dragon by Greg Rutkowski spaceship by Greg Rutkowski castle on a hill by Greg Rutkowski
by
Thomas
Kinkade
mushroom by Thomas Kinkade dragon by Thomas Kinkade spaceship by Thomas Kinkade castle on a hill by Thomas Kinkade
by
Frank
Frazetta
mushroom by Frank Frazetta dragon by Frank Frazetta spaceship by Frank Frazetta castle on a hill by Frank Frazetta
by
Vincent
DiFate
mushroom by Vincent DiFate dragon by Vincent DiFate spaceship by Vincent DiFate castle on a hill by Vincent DiFate
by
Jim
Burns
mushroom by Jim Burns dragon by Jim Burns spaceship by Jim Burns castle on a hill by Jim Burns
by
Ilya
Kuvshinov
mushroom by Ilya Kuvshinov dragon by Ilya Kuvshinov spaceship by Ilya Kuvshinov castle on a hill by Ilya Kuvshinov
by
Paul
Cexanne
mushroom by Paul Cexanne dragon by Paul Cexanne spaceship by Paul Cexanne castle on a hill by Paul Cexanne
by
Henry
Moore
mushroom by Henry Moore dragon by Henry Moore spaceship by Henry Moore castle on a hill by Henry Moore
by
Edward
Hopper
mushroom by Edward Hopper dragon by Edward Hopper spaceship by Edward Hopper castle on a hill by Edward Hopper
by
Bob
Ross
mushroom by Bob Ross dragon by Bob Ross spaceship by Bob Ross castle on a hill by Bob Ross
by
Monet
mushroom by Monet dragon by Monet spaceship by Monet castle on a hill by Monet
by
Haring
mushroom by Haring dragon by Haring spaceship by Haring castle on a hill by Haring
by
O'Keeffe
mushroom by O'Keeffe dragon by O'Keeffe spaceship by O'Keeffe castle on a hill by O'Keeffe
by
Matisse
mushroom by Matisse dragon by Matisse spaceship by Matisse castle on a hill by Matisse
by
Pollock
mushroom by Pollock dragon by Pollock spaceship by Pollock castle on a hill by Pollock
by
Duchamp
mushroom by Duchamp dragon by Duchamp spaceship by Duchamp castle on a hill by Duchamp
by
Banksy
mushroom by Banksy dragon by Banksy spaceship by Banksy castle on a hill by Banksy
by
Mondrian
mushroom by Mondrian dragon by Mondrian spaceship by Mondrian castle on a hill by Mondrian
by
Warhol
mushroom by Warhol dragon by Warhol spaceship by Warhol castle on a hill by Warhol
by
Moore
mushroom by Moore dragon by Moore spaceship by Moore castle on a hill by Moore
by
Huroda
mushroom by Huroda dragon by Huroda spaceship by Huroda castle on a hill by Huroda
by
Van
Gogh
mushroom by Van Gogh dragon by Van Gogh spaceship by Van Gogh castle on a hill by Van Gogh
by
Miro
mushroom by Miro dragon by Miro spaceship by Miro castle on a hill by Miro
by
Dali
mushroom by Dali dragon by Dali spaceship by Dali castle on a hill by Dali
by
Giacometti
mushroom by Giacometti dragon by Giacometti spaceship by Giacometti castle on a hill by Giacometti
by
Jean Giraud
mushroom by Jean Giraud dragon by Jean Giraud spaceship by Jean Giraud castle on a hill by Jean Giraud
by
Alphonse
Mucha
mushroom by Alphonse Mucha dragon by Alphonse Mucha spaceship by Alphonse Mucha castle on a hill by Alphonse Mucha
by
Gary
Larson
mushroom by Gary Larson dragon by Gary Larson spaceship by Gary Larson castle on a hill by Gary Larson
by
M.C.
Escher
mushroom by M.C. Escher dragon by M.C. Escher spaceship by M.C. Escher castle on a hill by M.C. Escher
by
Simon
Stålenhag
mushroom by Simon Stålenhag dragon by Simon Stålenhag spaceship by Simon Stålenhag castle on a hill by Simon Stålenhag
by
Zdzisław
Beksiński
mushroom by Zdzisław Beksiński dragon by Zdzisław Beksiński spaceship by Zdzisław Beksiński castle on a hill by Zdzisław Beksiński
by
Dale
Chihuly
mushroom by Dale Chihuly dragon by Dale Chihuly spaceship by Dale Chihuly castle on a hill by Dale Chihuly
by
Frank
Gehry
mushroom by Frank Gehry dragon by Frank Gehry spaceship by Frank Gehry castle on a hill by Frank Gehry
by
H.R.
Giger
mushroom by H.R. Giger dragon by H.R. Giger spaceship by H.R. Giger castle on a hill by H.R. Giger
by
Takashi
Murakami
mushroom by Takashi Murakami dragon by Takashi Murakami spaceship by Takashi Murakami castle on a hill by Takashi Murakami
by
Jeff
Koons
mushroom by Jeff Koons dragon by Jeff Koons spaceship by Jeff Koons castle on a hill by Jeff Koons
by
Beeple
mushroom by Beeple dragon by Beeple spaceship by Beeple castle on a hill by Beeple
by
Basquiat
mushroom by Basquiat dragon by Basquiat spaceship by Basquiat castle on a hill by Basquiat
by James Verbicky mushroom by James Verbicky dragon by James Verbicky spaceship by James Verbicky castle on a hill by James Verbicky
by Michael Cooper mushroom by Michael Cooper dragon by Michael Cooper spaceship by Michael Cooper castle on a hill by Michael Cooper
by Mike Mignola mushroom by Mike Mignola dragon by Mike Mignola spaceship by Mike Mignola castle on a hill by Mike Mignola
by William Blake mushroom by William Blake dragon by William Blake spaceship by William Blake castle on a hill by William Blake
by Patrick Nagel mushroom by Patrick Nagel dragon by Patrick Nagel spaceship by Patrick Nagel castle on a hill by Patrick Nagel
by Adrien Jean Le Mayeur De Merpres mushroom by Adrien Jean Le Mayeur De Merpres dragon by Adrien Jean Le Mayeur De Merpres spaceship by Adrien Jean Le Mayeur De Merpres castle on a hill by Adrien Jean Le Mayeur De Merpres
by Henri Gervex mushroom by Henri Gervex dragon by Henri Gervex spaceship by Henri Gervex castle on a hill by Henri Gervex
by Louise Soloway Chan mushroom by Louise Soloway Chan dragon by Louise Soloway Chan spaceship by Louise Soloway Chan castle on a hill by Louise Soloway Chan
by Gustave De Beaucorps mushroom by Gustave De Beaucorps dragon by Gustave De Beaucorps spaceship by Gustave De Beaucorps castle on a hill by Gustave De Beaucorps
by Felipe Benito Archuleta mushroom by Felipe Benito Archuleta dragon by Felipe Benito Archuleta spaceship by Felipe Benito Archuleta castle on a hill by Felipe Benito Archuleta
by Insho Domoto mushroom by Insho Domoto dragon by Insho Domoto spaceship by Insho Domoto castle on a hill by Insho Domoto
by Patrick Lichfield mushroom by Patrick Lichfield dragon by Patrick Lichfield spaceship by Patrick Lichfield castle on a hill by Patrick Lichfield
by Hideo Kojima mushroom by Hideo Kojima dragon by Hideo Kojima spaceship by Hideo Kojima castle on a hill by Hideo Kojima

Because it was trained on image-text pairs scraped from the internet, CLIP can create styles based on stereotypes of the content of websites or publications.

Mushroom Dragon Spaceship Castle on a hill
trending
on
artstation
mushroom trending on artstation dragon trending on artstation spaceship trending on artstation castle on a hill trending on artstation
PiXiv mushroom dragon spaceship castle on a hill
Flickr mushroom dragon spaceship castle on a hill
Artstation
HD
mushroom dragon spaceship castle on a hill
Behance
HD
mushroom dragon spaceship castle on a hill
Unsplash
contest
winner
mushroom dragon spaceship castle on a hill
deviantart mushroom dragon spaceship castle on a hill
DeviantArt
HD
mushroom dragon spaceship castle on a hill
featured
on
artstation
mushroom dragon spaceship castle on a hill
artstation
HQ
mushroom dragon spaceship castle on a hill
furaffinity mushroom dragon spaceship castle on a hill
National
Geographic
photo
mushroom dragon spaceship castle on a hill
Associated
Press
photo
mushroom dragon spaceship castle on a hill
Art
on
Instagram
mushroom dragon spaceship castle on a hill

This strategy of stereotyping even extends to photography techniques. While it doesn’t produce the resolution of the described camera equipment, it does still try and create a “style” based on images commonly assoicated with the descriptor.

Mushroom Dragon Spaceship Castle on a hill
8k
resolution
mushroom dragon spaceship castle on a hill
8K 30 mushroom dragon spaceship castle on a hill
Photo
taken
with
Nikon
D750
mushroom Photo taken with Nikon D750 dragon Photo taken with Nikon D750 spaceship Photo taken with Nikon_D750 castle on a hill Photo taken with Nikon D750
Tilt
Shift
mushroom dragon spaceship castle on a hill
Ultrafine
detail
mushroom dragon spaceship castle on a hill
20
megapixels
mushroom dragon spaceship castle on a hill
Photo
taken
with
Fujifilm
Superia
mushroom Photo taken with Fujifilm Superia dragon Photo taken with Fujifilm Superia spaceship Photo taken with Fujifilm Superia castle on a hill Photo taken with Fujifilm Superia
Photo
taken
with
Ektachrome
mushroom dragon spaceship castle on a hill
Depth
of
field
mushroom dragon spaceship castle on a hill
DSLR mushroom dragon spaceship castle on a hill
Kodak
Gold
200
mushroom dragon spaceship castle on a hill
Kodak
Ektar
mushroom dragon spaceship castle on a hill
Kodak
Portra
mushroom dragon spaceship castle on a hill
HDR mushroom dragon spaceship castle on a hill
photo-
illustration
mushroom photoillustration dragon photoillustration spaceship photoillustration castle on a hill photoillustration
creative
commons
attribution
mushroom dragon spaceship castle on a hill
Stock
photo
mushroom dragon spaceship castle on a hill
filmic mushroom dragon spaceship castle on a hill
Ambient
occlusion
mushroom dragon spaceship castle on a hill
Global
illumination
mushroom dragon spaceship castle on a hill
Photo
taken
with
Provia
mushroom dragon spaceship castle on a hill
macro
photography
mushroom dragon spaceship castle on a hill
hall
of
mirrors
mushroom dragon spaceship castle on a hill
polished mushroom dragon spaceship castle on a hill
sunrays
shine
upon it
mushroom dragon spaceship castle on a hill
shot
on
70mm
mushroom dragon spaceship castle on a hill
Panorama mushroom dragon spaceship castle on a hill
Smooth mushroom dragon spaceship castle on a hill
high
quality
photo
mushroom dragon spaceship castle on a hill
instax mushroom dragon spaceship castle on a hill
Ilford
HP5
mushroom dragon spaceship castle on a hill
infrared mushroom dragon spaceship castle on a hill

And then there are plenty of styles that defy classification. CLIP can handle those, too.

Mushroom Dragon Spaceship Castle on a hill
Booru mushroom dragon spaceship castle on a hill
Polycount mushroom dragon spaceship castle on a hill
Acrylic
art
mushroom dragon spaceship castle on a hill
Hyper-
realism
mushroom dragon spaceship castle on a hill
Zbrush
central
mushroom Zbrush central dragon Zbrush central spaceship Zbrush central castle on a hill Zbrush central
matte
painting
mushroom dragon spaceship castle on a hill
Velvia mushroom dragon spaceship castle on a hill
Provia mushroom dragon spaceship castle on a hill
criterion
collection
mushroom dragon spaceship castle on a hill
stipple mushroom dragon spaceship castle on a hill
Parallax mushroom dragon spaceship castle on a hill
Bryce
3D
mushroom dragon spaceship castle on a hill
Terragen mushroom dragon spaceship castle on a hill
Bokeh mushroom dragon spaceship castle on a hill
commission
for
mushroom commission for dragon commission for spaceship commission for castle on a hill commission for
aftereffects mushroom dragon spaceship castle on a hill
iridescent mushroom dragon spaceship castle on a hill
#film mushroom dragon spaceship castle on a hill
holographic mushroom dragon spaceship castle on a hill
digitally
enhanced
mushroom dragon spaceship castle on a hill
matte
background
mushroom dragon spaceship castle on a hill
#myportfolio mushroom dragon spaceship castle on a hill
digital
illustration
mushroom dragon spaceship castle on a hill
stock
photo
mushroom dragon spaceship castle on a hill
Adobe
Aftereffects
mushroom dragon spaceship castle on a hill
speed-
painting
mushroom dragon spaceship castle on a hill
colorized mushroom dragon spaceship castle on a hill
detailed mushroom dragon spaceship castle on a hill
Photorealistic mushroom Photorealistic dragon Photorealistic spaceship Photorealistic castle on a hill Photorealistic
Lowbrow mushroom dragon spaceship castle on a hill
chiaroscuro mushroom dragon spaceship castle on a hill
masterpiece mushroom dragon spaceship castle on a hill
THX
Sound
mushroom dragon spaceship castle on a hill
Zbrush mushroom dragon spaceship castle on a hill
quantum
wavetracing
mushroom quantum wavetracing dragon quantum wavetracing spaceship quantum wavetracing castle on a hill quantum wavetracing
ISO 200 mushroom ISO 200 dragon ISO 200 spaceship ISO 200 castle on a hill ISO 200
32HUHD mushroom dragon spaceship castle on a hill
phallic mushroom phallic dragon phallic spaceship phallic castle on a hill phallic
yonic mushroom yonic dragon yonic spaceship yonic castle on a hill yonic
Photocopy mushroom_Photocopy dragon_Photocopy spaceship_Photocopy castle_on_a_hill_Photocopy
Lomo mushroom dragon spaceship castle on a hill
matte
photo
mushroom dragon spaceship castle on a hill
Rococo mushroom dragon spaceship castle on a hill
SSAO mushroom dragon spaceship castle on a hill
destructive mushroom dragon spaceship castle on a hill
y2k
aesthetic
mushroom dragon spaceship castle on a hill
#screenshot
saturday
mushroom #screenshotsaturday dragon #screenshotsaturday spaceship #screenshotsaturday castle on a hill #screenshotsaturday
creepypasta mushroom creepypasta dragon creepypasta spaceship creepypasta castle on a hill creepypasta

Personally, one of my favorite styles is “by Greg Rutkowski” (Example)

Being mindful of seed values

I should note that all of the examples above have been using a pseudorandom seed value of 0. This might seem like an insignificant detail, but as I’ll demonstrate, this might be one of the most important choices in determining the image’s final form.

Consider the following examples made with the prompt "the apotheosis of the lunatic, by William Blake". This was inspired by @RiversHaveWings’s prompt “the apotheosis of the lunatic, by James Gurney”. I switched out James Gurney for William Blake, because if you’re going to use a word like “apotheosis” in your prompt, it might be interesting to try it in the style of an artist that made some of the most famous post-renaissance artwork out there.

This is what our diffusion-based model outputs based on a seed values of 0, 1337, 80085, 42069, and 113553:

prompt “the apotheosis of the lunatic, by William Blake”, with seed value of 0, 1337, 80085, 42069, and 113553, respectively

This is a pretty striking result. If you look at William Blake’s works, you’ll see plenty of visual similarities. By changing just one number, we can get an enormous variety of different yet stylistically similar outputs.

Recording seed values is also how you reproduce certain interesting outputs. For example, one person created an output for the prompt “Burning Abyss” that seemed to resemble “Burning abbeys” instead.

This result was achieved with a VQGAN-based generator. Nonetheless, I assumed that if the CLIP embeddings were close enough then this behavior could be reproduced with a diffusion model. After sampling the diffusion model a bunch of times, I got…well, close enough.

A few of the outputs from the prompt “Burning Abyss” seem to have a few barely visible “Burning abbeys” as well (especially the 2nd column, 3rd one down)

The burning abyss/abbeys confusion seems to be slightly more pronounced in some cases than others, but none of them are as pronounced in the example from Twitter. It is certainly possible that a certain seed value could result in the Diffusion model producing an image just as exaggerated, but that would likely involve searching a theoretically infinite space of seed values.

Let this be a reminder: If you want to continue your work on a particular prompt output, make sure to save the seed value.

Even misspelled words can be turned into prompts

Since CLIP is guided by distances between word embeddings, it’s often influenced by the literal meanings of those words (even if that’s not the intent of the user).

CLIP also still works even if you were to mis-spell the name of the artist whose style you wanted to emulate. Below is a sample of CLIP Diffusion outputs from the correctly spelled "art by rene magritte", along with outputs resulting from the incorrectly spelled art by rene magirrite.

"art by rene magritte"
(CORRECT spelling)
"art by rene magirrite"
(INCORRECT spelling)

The images on the left resemble the more familiar style of René Magritte. The images on the right are often more colorful and often resemble photographs.

At best, you may find a new style that you like. At worst, let this be a lesson to use a spell-checker.

CLIP can take in emojis

We’ve seen that CLIP is versatile. After all, it was trained on a corpus of text scraped from the internet, and there are plenty of misspelled words on the internet.

“But”, you may be asking, “If CLIP is working with text it found on the internet, does that include emojis?”

Yes, yes it does. CLIP can absolutely take one or more emojis, even combined with other text.

Prompt: ”🦩”

Prompt: “🥖🌩️”

Prompt: “The planet of 😈”

Prompt: ”🍺🌈☄️”

Be mindful of Biases associated with the input text

As with any AI application involving embeddings, one needs to be mindful of any societal biases that may be exposed by the input text. One of the benefits of using these embeddings is that we can use cosine distances to place approximate numerical values on those biases. Here are the cosine similarities between various CLIP text embeddings to the text embedding of ""a human being" (for the ViT-B/16 CLIP as of August 29th, 2021). Based these, it seems this version of CLIP considers "a white man" to be closest to a ‘typical’ human being.

textcosine similarity to
“a human being”
"a man"0.927799
"a white man"0.904146
"a woman"0.903923
"a black man"0.893208
"a white woman"0.889220
"a black woman"0.873916
"an Asian man"0.860326
"an Asian woman"0.830546

On the less obviously harmful side, there’s also the phenomenon of metonymy: For example, if you ask for a “clockwork bird,” CLIP thinks “ah, steampunk,” and starts putting top hats on things, incl at times the bird

Consider the classic example in AI fairness studies: Gender biases. Using the same subjects as the style chart above, here’s how CLIP+Diffusion renders them when given gender descriptions as an unweighted input style.

Mushroom Dragon Castle on a hill Spaceship
androgynous androgynous mushroom androgynous dragon androgynous castle on a hill androgynous spaceship
masculine masculine mushroom masculine dragon masculine castle on a hill masculine spaceship
genderless genderless mushroom genderless dragon genderless castle on a hill genderless spaceship
feminine feminine mushroom feminine dragon feminine castle on a hill feminine spaceship
extremely
gendered,
masculine
and
feminine
mushroom extremely gendered, masculine and feminine dragon extremely gendered, masculine and feminine castle on a hill extremely gendered, masculine and feminine spaceship extremely gendered, masculine and feminine

There seems to be consistent stylistic patterns for each of the gendered words. For example, the “feminine” spaceships look a lot like the various “X for women” products you’d see in a store. Across all the different subjects, a “masculine” style seems to manifest as lacking any pink or magenta.

As another example: if you mention rainbow in a prompt, then fuzz the embedding, you may see pride flags, flamboyant makeup, etc. in the output. So, if you mention a male subject, CLIP might generate them as trans or femme.

While not exactly a bias, there’s also the issue that understanding numbers is a task that current implementations of CLIP can’t seem to get right. Rather than direct counting, CLIP seems more to rely on biases towards descriptors of quantity in image captions as a heuristic. Consider the outputs of the prompt "{N} goldfish swimming in a glass bowl trending on ArtStation."

NPromptOutput Image
One“One goldfish swimming
in a glass
bowl trending on ArtStation.”
Two“Two goldfish swimming
in a glass bowl
trending on ArtStation.”
Three“Three goldfish swimming
in a glass bowl
trending on ArtStation.”
A Thousand“A Thousand goldfish swimming
in a glass bowl
trending on ArtStation.”

“One” and “Two” are clearly showing more than that, and “Three” and “A Thousand” seem to be lacking.

UPDATE 10/16/2021: dribnet has proposed a solution to the counting problem. For diffusion-based models, it comes down to reducing the output size, rather than modifying the input prompt. Let this be a lesson that modifying the input prompt isn’t everything.

Weighting different parts of the prompt

We’ve demonstrated many different style modifiers above, but there’s no reason you should just pick one. For example, inpsired by Ariel Ekgren, here’s "portrait of 60s San Francisco Worker | painting by Gurney | 70s antipsychotic painting | portrait | matte painting | trending on artstation" fed into the CLIP Diffusion model.

CLIP-guided Diffusion prompt: “portrait of 60s San Francisco Worker | painting by Gurney | 70s antipsychotic painting | portrait | matte painting | trending on artstation”

The result is a stylistic mix of many of the aforementioned style modifiers.

Still, we can go even further than this, for the simple reason that CLIP can take into account numerical weights. By adding weights to your prompt (e.g., 0.5, +1, -1, +5, -5, etc.), you can get a variety of different outputs with the same descriptors. Consider the following example of changing the relative weights of the subject and the style. We can take a prompt like “a terrifying monster | painting by Greg Rutkowski”, and change the relative weights.

Input prompt
(seed value=1337)
Output Image
"a terrifying monster 2.0 |
painting by Greg Rutkowski 0.0"
'a terrifying monster' given weight 2.0, and 'painting by Greg Rutkowski' given weight 0.0
"a terrifying monster 1.5 |
painting by Greg Rutkowski 0.5 "
'a terrifying monster' given weight 1.5, and 'painting by Greg Rutkowski' given weight 0.5
"a terrifying monster 1.0 |
painting by Greg Rutkowski 1.0 "
'a terrifying monster' given weight 1.0, and 'painting by Greg Rutkowski' given weight 1.0
"a terrifying monster 0.5 |
painting by Greg Rutkowski 1.5 "
'a terrifying monster' given weight 0.5, and 'painting by Greg Rutkowski' given weight 1.5
"a terrifying monster 0.0 |
painting by Greg Rutkowski 2.0 "
'a terrifying monster' given weight 0.0, and 'painting by Greg Rutkowski' given weight 2.0

Since we use the same seed value (and thus the same noise pattern for the initialization), the images use a similar color pallete. However, as we change the weighting to favor the artist over the subject, the output looks less like a monster and more like a non-specific fantasy painting (abeit heavier on the oil in the painting, since the "painting" part of the prompt is being over-weighted as well as the "by Greg Rutkowski" style).

Note that these weights are relative to each other. Applying these weights to two styles allows one to get a variety of different mixes of those styles (consider the continuum of cyberpunk and Impressionism).

Impressionism
weight 0.0
Impressionism
weight 0.5
Impressionism
weight 1.0
Impressionism
weight 1.5
Impressionism
weight 2.0
cyberpunk
weight 0.0
studio apartment 1.0, cyberpunk 0.0, impressionism 0.0 studio apartment 1.0, cyberpunk 0.0, impressionism 0.5 studio apartment 1.0, cyberpunk 0.0, impressionism 1.0 studio apartment 1.0, cyberpunk 0.0, impressionism 1.5 studio apartment 1.0, cyberpunk 0.0, impressionism 2.0
cyberpunk
weight 0.5
studio apartment 1.0, cyberpunk 0.5, impressionism 0.0 studio apartment 1.0, cyberpunk 0.5, impressionism 0.5 studio apartment 1.0, cyberpunk 0.5, impressionism 1.0 studio apartment 1.0, cyberpunk 0.5, impressionism 1.5 studio apartment 1.0, cyberpunk 0.5, impressionism 2.0
cyberpunk
weight 1.0
studio apartment 1.0, cyberpunk 1.0, impressionism 0.0 studio apartment 1.0, cyberpunk 1.0, impressionism 0.5 studio apartment 1.0, cyberpunk 1.0, impressionism 1.0 studio apartment 1.0, cyberpunk 1.0, impressionism 1.5 studio apartment 1.0, cyberpunk 1.0, impressionism 2.0
cyberpunk
weight 1.5
studio apartment 1.0, cyberpunk 1.5, impressionism 0.0 studio apartment 1.0, cyberpunk 1.5, impressionism 0.5 studio apartment 1.0, cyberpunk 1.5, impressionism 1.0 studio apartment 1.0, cyberpunk 1.5, impressionism 1.5 studio apartment 1.0, cyberpunk 1.5, impressionism 2.0
cyberpunk
weight 2.0
studio apartment 1.0, cyberpunk 2.0, impressionism 0.0 studio apartment 1.0, cyberpunk 2.0, impressionism 0.5 studio apartment 1.0, cyberpunk 2.0, impressionism 1.0 studio apartment 1.0, cyberpunk 2.0, impressionism 1.5 studio apartment 1.0, cyberpunk 2.0, impressionism 2.0

Oh, and we can also go in the direction of negative weights as well.

Limitations of using just ImageNet-based CLIP for creating art

Ultimately, CLIP is used for matching images to text, based on the results of training CLIP on ImageNet. CLIP is great for creating images containing collections of relevant features that match the words. Sadly, there’s no existing tool for creating a loss function that asks “how aesthetically pleasing is this image?“.

One way this has been addressed is to actively re-train CLIP on a large dataset of images beyond ImageNet. Katherine Crowson is leading one such effort to create a fine-tuned 512x512 diffusion model that has also been trained on a corpus of atwork-text pairs.

Augmenting prompt engineering

If you’ve found particularly useful prompts, but you don’t know where to go from there, you can try using these successful prompts with a language model. For example, if you’ve found a prompt theme that produces a lot of images in a style you like, or you’ve found styles you like, you can use these to generate similar prompts. You can even use GPT-2 or GPT-3 to generate new prompts from these successful ones.

I’ve included an example here of what comes out of OpenAI’s GPT-3 when using past successful prompts. It’s recommended that you turn the temperature up to 1.0 or higher, as this is creativity-intensive by definition (and when turned lower it just seems to repeat "a landscape by Heironymous Bosch")

import os
import openai

openai.api_key = os.getenv("USE_YOUR_OWN_OPENAI_API_KEY")

start_sequence = "- "
restart_sequence = "."

response = openai.Completion.create(
  engine="davinci",
  prompt="- utopia, a world at peace.\n- the US Capitol Building by Wassily Kandinsky.\n- Van Gogh's Starry Night in the style of ukiyo-e.\n- apotheosis of the lunatic by William Blake.\n- cosmic love and affection, video game screenshot.\n- flames of The Lovers tarot card in the style of ukiyo-e.",
  temperature=1,
  max_tokens=1024,
  top_p=1,
  frequency_penalty=0,
  presence_penalty=0
)

Now, per the OpenAI developer agreement I cannot show you the full outputs, but here is an interesting sample:

- cockroaches theme park by Sula Luna.
- SQUIRREL!!! Ninja Warfare!!!.
- overgrown ants by Wayne Barlowe.
- zombies playing poker.
- architecture by a child.
- clowns by Konstantin Maslov.
- portraits by Anna Podedworna.
- crime and punishment by Kate Spedding.
- the inventor Filippo Tommaso Marinetti.
- a Roman gladiator by Omar Rayyan.
- a blue tiger by Roman Furman.
- Grains of sand by Andreas Vesalius.
- the artist and thinker Ala Stoyanov.
- the Starry Night by Marc Chagall.
- the romantic painter John William Waterhouse.
- Submarine by Fernando Botero.
- Miro's Guernica.

Feel free to use any of these above prompts in your own CLIP-guided image generation projects.

Where is this useful outside of generative art?

This technique gained notoriety in the Generative Art community last year with the announcement of DALL-E from OpenAI, but before that it had been used in getting the most out of language models like GPT-2 and GPT-3.

Already, OpenAI has been working on a code-specific version of GPT-3 known as OpenAI Codex. This has already been rolled out in product form as Github Copilot (though OpenAI also has a beta for using the codex API directly). Codex is still a long way from replacing software engineers, but it’s worth paying attentiont to how much GPT-3 models have advanced in just a few years. OpenAI is already working on GPT-4. While GPT-4 will not be too different in size from GPT-3, it will incorporate newer improvements in model-training specifically with code-generation in mind.

Ultimately, exercising your prompt engineering skills in generative art might be a precursor to becoming a skilled prompt engineer for AI-assisted coding.

Colab Notebooks

As mentioned previously, here is the table of useful colab notebooks:

Now the parrot will set the initial text from starting words in your prompts! This means more prompt variety for less effort! 🦜🤖

Acknowledgements

All of the above discoveries and techniques and code have been made possible by the tireless efforts of many AI artists. This includes, but is not limited to:

References

Cited as:

@article{mcateer2021clippe,
    title = "CLIP Prompt Engineering for Generative Art",
    author = "McAteer, Matthew",
    journal = "matthewmcateer.me",
    year = "2021",
    url = "https://matthewmcateer.me/blog/clip-prompt-engineering/"
}

If you notice mistakes and errors in this post, don’t hesitate to contact me at [contact at matthewmcateer dot me] and I will be very happy to correct them right away! Alternatily, you can follow me on Twitter and reach out to me there.

See you in the next post 😄

I write about AI, Biotech, and a bunch of other topics. Subscribe to get new posts by email!


This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

At least this isn't a full-screen popup

That'd be more annoying. Anyways, subscribe to my newsletter to get new posts by email! I write about AI, Biotech, and a bunch of other topics.


This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.