New AI research tool turns ideas into art

Imagine creating a digital painting without ever picking up a paintbrush or instantly creating picture book illustrations to go with the words. Today we present an exploratory artificial intelligence (AI) research concept called Make-A-Scene that enables people to bring their visions to life.

Make-A-Scene allows people to create images with text prompts and free-form sketches. Previous image-generating AI systems typically used text descriptions as input, but the results could be difficult to predict. For example, typing “a painting of a zebra riding a bicycle” may not reflect exactly what you had in mind. The bike could be sideways or the zebra could be too big or too small.

With Make-A-Scene, that’s no longer the case. It shows how people can use both text and simple drawings to more specifically convey their visions using a variety of elements.

Make-A-Scene captures the scene layout to allow for nuanced sketches as input. It can also generate its own layout with plain text prompts if the creator chooses. The model focuses on learning key aspects of the images that are more likely to be important to the creator, such as objects or animals.

Fostering creativity for artists and non-artists alike

As part of our research and development process, we have shared access to our Make-A-Scene demo with AI artists such as Sofia Crespo, Scott Eaton, Alexander Reben and Refik Anadol.

Crespo, a generative artist focused on the intersection of nature and technology, used Make-A-Scene to create new hybrid creatures. She found that the free-form drawing capabilities allowed her to quickly get started with new ideas.

“Sometimes, as a visual artist, you just want to be able to create a base composition by hand to draw a story for the eye to follow, and that’s exactly what makes it possible.” — Sofia Crespo, AI Artist

A GIF of the output generated by Make-A-Scene through artist Sofia Crespo's text prompt

Make-A-Scene isn’t just for artists – we think it could help everyone express themselves better. Andy Boyatzis, program manager at Meta, used Make-A-Scene to create art with his two- and four-year-old children. They used playful drawings to bring their ideas and imagination to life.

“​​If they wanted to draw something, I would just say, ‘What if…?’ and that led them to create wild things, like a blue giraffe and a rainbow airplane. It just shows the limitlessness of what they could come up with.” — Andy Boyatzis, Program Manager, Meta

An image of the output generated by Make-A-Scene via the text prompt

Building the next generation of creative AI tools

It is not enough that an AI system only generates content. To realize the potential of AI to boost creative expression, humans should be able to shape and control the content generated by a system. It should be intuitive and easy to use so that people can use whatever means of expression best suits them, whether it be voice, text, gestures, eye movements, or even sketches, to bring their vision to life.

Through projects like Make-A-Scene, we continue to explore how AI can augment creative expression. We are making progress in this area, but this is just the beginning. We will continue to push the boundaries of what is possible by using this new class of generative creative tools to develop methods for more expressive messaging in 2D, mixed reality and virtual worlds.

Leave a Comment

Your email address will not be published.