Mimicking DeepMind's Flamingo with GPT-3

Short piece on using GPT-3 to reconstruct behaviors of one of DeepMind's new models

In a previous post, I mentioned how I’m writing a book for O’Reilly, on how to deploy machine learning models (mainly transformer models) to production in a trustworthy way.

That is still happening, but I came across a new technique fro GPT-3 that didn’t quite fit into the book, but I felt like I had to share it anyway.

For context, we need to understand some new research out of DeepMind:

What is Flamingo?

A few days ago, DeepMind announced a visual language model called Flamingo, announcing it in a blog post “Tackling Multiple Tasks with a Single Visual Language model”. This model is basically to images AND language inputs what GPT-3 was to just text inputs (both Flamingo and GPT-3 only produce text output). It can handle text plus an image, multiple images, multiple text inputs (i.e., in a conversational chatbot-like mode).

It builds upon frozen and pre-trained models. Much of the training process is focused on training the adapters between these frozen modules.

It. While it does not push the SOTA in all of these tasks, the exciting part is that it can be used to do ALL of these tasks.

One of the tasks was connecting concepts, and Twitter understandably had a field day with this.

Andrej Karpathy has been sharing an example of Flamingo making sense of a picture. It kind of has to be guided through the reasoning like socratic dialogue.

What if I don’t have access to Flamingo?

If you don’t have access to Flamingo, how do you test out this feature?

Well, as it turns out, if you have access to OpenAI’s GPT-3, you can recreate some of this behavior.

POST /v1/engines/text-davinci-002/completions
import os
import openai

openai.api_key = os.getenv("OPENAI_API_KEY")

start_sequence = "\nAI: "
restart_sequence = "\nHuman:"
prompt_text = (
    "The following is a conversation with an AI assistant. The assistant is "
    + "helpful, creative, clever, and very friendly.\n"
    + "\nHuman: Hello, who are you?"
    + "\nAI: I am an AI created by OpenAI. How can I help you today?"
    + "\nHuman: Let's play a game. I give you two concepts, and you tell me "
    + "how they are connected."
)

response = openai.Completion.create(
    engine="text-davinci-002",
    prompt=prompt_text,
    temperature=0.9,
    max_tokens=150,
    top_p=1,
    frequency_penalty=0,
    presence_penalty=0.6,
    stop=["Human:", "AI:"],
)

The following is the unedited text of my interactions with GPT-3 in the sandbox.

Transcript

(GPT-3 outputs highlighted in green)

The following is a conversation with an AI assistant. The assistant is helpful, creative, clever, and very friendly.

Human: Hello, who are you?

AI: I am an AI created by OpenAI. How can I help you today?

Human: Let’s play a game. I give you two concepts, and you tell me how they are connected.

AI: Sure, that sounds like fun! Let me know when you’re ready to start.

Human: (A) Croissants. (B) Renault cars.

AI: Well, both croissants and Renault cars are French products.

Human: (A) The Movie, Inception. (B) The movie, The Matrix.

AI: Both movies are about people who are stuck in a dream world and have to find their way back to reality.

Human: Can you give more detail?

AI: In Inception, the characters are stuck in a dream within a dream, and they have to use their creativity to find their way back to the real world. In The Matrix, the characters are stuck in a virtual world that is controlled by artificial intelligence, and they have to fight their way back to reality.

Human: Thanks. Let’s try another. (A) The Movie, The Matrix. (B) Marxist writing on capitalism?

AI: Both The Matrix and Marxist writing on capitalism explore the idea of humans being controlled by an oppressive system. In The Matrix, the characters are controlled by a virtual world that is run by artificial intelligence. In Marxist writing on capitalism, humans are controlled by the economic system of capitalism.

Human: Any additional connections between (A) The Matrix, and (B) Marxist writing on capitalism?

AI: Both The Matrix and Marxist writing on capitalism explore the idea of humans being trapped in an artificial system that controls their lives.

Human: Any more?

AI: Both The Matrix and Marxist writing on capitalism explore the idea of humans being controlled by an oppressive system. In The Matrix, the characters are controlled by a virtual world that is run by artificial intelligence. In Marxist writing on capitalism, humans are controlled by the economic system of capitalism.

Human: That’s brilliant. Let’s try another. (A) The relationship between Cobb and Mal in Inception. (B) The slogan “Make America great again”

AI: Both the relationship between Cobb and Mal in Inception and the slogan “Make America great again” involve the idea of going back to a simpler time. In the movie, Cobb and Mal are trying to go back to a time before they were stuck in a dream world. The slogan “Make America great again” is about returning to a time when America was a more prosperous and powerful country.

Human: Good

What does this mean for Flamingo?

This is not to say that Flamingo’s research into multi-modal capabilities isn’t impressive. It absolutely is.

What the GPT-3 results raise is how much variation in architectures there can be that achieve the same task. After all, Flamingo’s main contribution was training the adapter between all these language and image modules.

I write about AI, Biotech, and a bunch of other topics. Subscribe to get new posts by email!


This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

At least this isn't a full-screen popup

That'd be more annoying. Anyways, subscribe to my newsletter to get new posts by email! I write about AI, Biotech, and a bunch of other topics.


This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.