How to Break an Algorithm’s Brain 🧠

I'll admit it—when I first heard whispers of visual AI becoming a powerhouse, I wasn't exactly jumping for joy. In fact, my internal monologue was more along the lines of, "Can't AI be used to fight climate change instead? Or at least help with the endless admin of modern life?" I'm still convinced that blindly following an algorithm is a terrible idea and that AI will have massive effects on how we work. But I'm now allowing myself to have an open mind about its creative potential, and what I'm finding is an utter surprise: I'm actually having a blast with it.

So, I did what any self-respecting nerd would do: I signed up for a university course on AI and design. And as a result, I've decided to turn this page into a playground for my experiments. As the course progresses, I'll update this with notes, observations, and some truly bizarre reference images that I've created in varying degrees of collaboration with AI.

To make sense of the AI madness, it helps to break it down. When we talk about visual AI, we're not just talking about one big robot artist. There are several different types, each with its own special talents. The one you've probably heard of is text-to-image generation, which turns a written prompt into a brand new picture. Then there's image-to-image, where the AI takes a drawing you've made and "remixes" it into a new style. There are even tools for generative video, which can create an animation from a single image or line of text. It's a bit like having a magic wand for different creative tasks—and we are only just getting started.

Adobe

The first software that we started experimenting with was Adobe Firefly. Thank God for that free trial week because Adobe products are never cheap, even with a student discount. While ChatGPT is a powerful language model that generates text, Adobe Firefly is a family of creative AI models that focus on visuals. It's built to be a creative co-pilot that can produce high-quality images and text effects directly from your written prompts. You can use it as a standalone tool on the Firefly website to create new visuals from scratch or to explore features like Generative Fill and Generative Recolour, making it easy to experiment without needing any prior expertise.

What makes Firefly so powerful is its deep integration into the Adobe applications you might already use (in that case kudos to you because every time I don’t need that software for a while, I completely forget all the commands). This means you can use its generative capabilities directly within your existing workflow. For example, in Adobe Photoshop, you can use its Generative Fill feature to seamlessly add or remove objects from an image with a simple text prompt.

Working with it is like having a well trained but alien assistant who has seen pictures of everything and has been told what they are, but has no real concept of what a horse is let alone a human. So from afar things look correct but then you zoom in and the person has seven fingers, the house has windows and stucco in the most random places and the elephant that you asked for with the description: standing on one leg, has one leg raised instead. It is both marvellous and infuriating to work with. This tool is so powerful at coming up with polished appearing pictures within seconds but has absolutely no clue about the real world.

One time I asked for a black woman on a spaceship and it gave me a white woman with a black shirt instead… bordering on racism if I didn’t believe it to just be utterly ignorant. The assumption, if I don’t specify further usually is thin, white people as if that is just what people always are. Now that we have the grumpy rant out of the way let me return to the actual exploration. Since it would not follow my detailed prompts accurately, I decided to go in the opposite direction instead, to play with the fantastical and to see what kind of images it would come up with when I intentionally confused it. Here are some of the results.

Space related art using text to image ai

Early moon cake pattern Firefly
Moon cake pattern further developed with Firefly
Why is she putting mascara on her nose? with Firefly. Ai fail
I don’t know what these guys are eating but I hope you are having something better
Finally managed to get a black, female astronaut
Don’t be fooled, this pattern doesn’t actually repeat correctly.
An odd strength: Armageddon images
Also exploding candy does look oddly delicious…
I managed to utterly confuse ai with this leopard Pegasus request