Generative AI Investigations
2023
By Neil Blevins
Created On: May 8th 2022
Updated On: July 25th 2024
Back in 2022 and 2023, I did some investigations into using Generative
AI for my art making process. While I am interested in the
possibilities of the technology for the future, I feel the current
technology isn't being pushed in the correct direction, and so have
discontinued my exploration until a number of major issues have been
sorted through (such as the use of copyrighten imagery in the
datasets). Below is a set of articles, tests and tutorials I made
during that year long period that I feel still have value, exploring
some of the ways AI could potentially be used to enhance the artist
workflow. My hope is to see the tech change course in the coming years
either voluntarily or through regulations, at which point I'll give it
another try.
Lesson 1:
AI
Artwork, The Road Ahead
If you've been following social media these last few months, you've
probably come across a ton of artwork generated by
Artificial Intelligence. AI can take two different images and mix them
in surprising ways, it can
shade a line drawing, or it can even generate artwork by providing it a
line of descriptive text. Tons of artists are starting to play with
this sort of software, including myself, and so I felt now was a good
time to go over some of what these sorts of applications can do, and
have a brief discussion of where this may all be leading us.
Note, this is not a lesson on how to use a particular piece of
AI software, this is about my experiments with them, giving you some
idea of
the sorts of things you can achieve with the techniques.
You have two choices with this lesson, watch me discuss the issue in
the video below, or read the full text.
Lesson 2:
How To Use AI In Your
Photobashing Process
AI (Artificial Intelligence) artwork has been a huge subject of
experimentation and debate these last few months. Whether this is a fad
or the future, who knows. But if we are going to use AI, my goal
is have the AI assist me rather than replace me, I want to find that
balance between my own artistic ideas / style and that of the
algorithm. And while AI continues to evolve, and so workflows will as
well, I wanted to share with you my current attempts at finding that
balance.
As a visual
artist, I've been looking at AI tools, and trying to decide how I could
use their strengths without reducing my contribution to a few well
chosen keywords. So this tutorial will discuss the methods I've been
using to
incorporate
AI artwork into some of my paintings.
Note: This isn't a tutorial showing you how to use any specific piece
of software, this is speaking about the process at a higher level, and
can be applied to pretty much any software you'd like.
You have two choices with this lesson, watch me discuss the issue in
the video below, or read the full text.
Lesson 3:
Layer
Breakdown: The Final Push
As many of you know, I've been playing a lot recently with AI generated
artwork, especially creating robot designs. However, one thing I've
been unable to achieve in midjourney (the AI I've been currently using)
is a 3/4 view of the robot designs. All of them end up being frontal
view only. This is likely because the images the AI trained on either
haven't been 3/4 view, or maybe they have but that particular attribute
hasn't been labeled, and so the AI hasn't learned what a 3/4
perspective is. While I'm sure this will be fixed in the future, I've
been playing with giving the robots depth by projecting the artwork
onto simple 3d geometry. So this tutorial discusses this technique
which I used to make my painting "The Final Push".
You have two choices with this lesson, watch me discuss the issue in
the video below, or read the full text.
Lesson 4:
Keyword Or
Placebo? Testing
Midjourney Prompts
With the current crop of AI Art Generators, your two major
inputs are image prompts and text prompts. This tutorial is going to be
focusing on text prompts in Midjourney, and asks the question do all of
the keywords people tend to add to their prompt really affect the final
image? Or are they actually not contributing at all, and maybe even
confusing the AI?
This tutorial shows some experiments I did with the Midjourney AI
trying to figure out what are the most useful and least useful
keywords, and I hope by presenting them it gives you a better idea of
the best way to turn your ideas into images.
If you want to download the giant tables in the video that show the
different keyword comparisons, they are here and here.
Lesson 5:
The 10 Things I've Learned
Comparing Midjourney And DALL-E
So I recently got access to the AI Art Generator known as
DALL-E 2, and after some initial playing, my next big question was how
it compared and contrasted to the other major AI Art Generator that
I've been using, Midjourney. So I ran both systems through a battery of
tests, and have found some interesting results. In short, each as you'd
expect have their own advantages and disadvantages. So if you've been
playing with Midjourney and are wondering how it compares to DALL-E,
I've compiled my 10 things I've learned comparing the two software.
If you want to download the giant tables in the video that show the
different keyword comparisons, they are here
and here.
Lesson 6:
What Would Concept Artists
Want From An AI Tool?
The main focus I've seen so far in the world of Generative AI
Artwork is how to replace the artist. But what about tools that help an
artist to
work faster? If the AI field could be anything a concept artist wanted
it to be, what would that look like? This discussion will explore some
of the things I personally would like to see from an AI tool that helps
our work, as opposed to disrupting it entirely.
Features I'd Want:
- Copyright Free Database with Opt In
- Training an AI on your own Work
- Generate Mood Boards
- Make a Sketch Photoreal
- Deeper Image to Image Control
- Make Variations
- Make Photobash Elements
- Alpha Channels
- Lighting Direction And Time Of Day
- Change the Weather
- Change the Material On An Object
- Tileable Textures
- Upresing Lowres Images And Textures
- Making Orthos / Change Camera Position
- 2D to 3D
- Text To 3D and Other 3D Features
Below is a video I made on the subject in September 2022, which is
still relevant, but a lot has changed in the last 8 months, so until I
make a brand new video, the text version below the video is far more up
to date.
Automating Your Art Process
When the software 3dsmax 2 came out in the late 90s, it
introduced its first scripting language, and I used it to write tools.
Some people though I did this because I liked programming, and it may
shock you to know I don't particularly like programming, and am not a
very good programmer. But the reason I made those tools was because
something I dislike even more is doing boring repetitive tasks, and
writing tools gave me a way to remove the "boring" parts from my
artistic workflow. I didn't want to place every single tree by hand, I
wanted to define what the forest looks like, and then the software
fills in the details, which I can then later tweak. And if someone
hasn't already written that software, then I was going to.
So that said, I am not new to the idea of letting software assist me in
handling the parts of my art process that I don't find fun. As my
friend Apurva put it, "Machine Assisted Human Expression". And AI has a
lot of potential to help in that area.
But AI has a lot of potential harm as well, as artists we fear being
replaced. This is bad in 2 ways.
First off, obviously, the monetary issue. A lot of people's careers are
based on producing artwork for all sorts of products. Yes, there will
always need to be some human artists, but if the number of jobs for a
human artist decreases, a lot of people are going to be unable to pay
their rent, unable to afford college for their kids, etc. Yes, artists
will need to adapt their skillset, but if there's just less work to do,
no amount of adaptation is going to replace that lost income.
The second issue is the larger philosophical one. If a computer can
make artwork that looks good, why should I bother making art? I
certainly agree that the thing we humans bring to the table is the why,
it's the idea behind the art, it's the emotion, it's the message. And
those skills are stuff that can only be replaced by an AI if the AI
becomes sentient, which I think is still a very long ways away. But if
an AI can make artwork that's good enough, if it replaces meaning with
using the right combination of color and contrast to give the audience
a dopamine hit, if quality is replaced with literally infinite
quantity, if the audience no longer cares about meaning or expression
just as long as the rods and cones in their eyes are excited, then
we're going to see very few artists feel the need to produce anything,
at least digitally. I've seen many people say "AI isn't making creative
work, it's just remixing everything that's already been done." The
thought that the audience cares a lot about originality is a wonderful
fiction we tell ourselves, but at least in the film and videogame
industry, a lot of work is about making something similar to what has
already been done before. And these AI art generators seem really good
at doing that.
But lets put these issues aside right now and imagine a world where
these companies producing AIs wanted to use the algorithms to enhance
human art instead of replace it. What would those pieces of software
look like? If you could design your own personal AI tool, what would
that tool do? If you could steer the direction of this technology, what
direction would you steer it? Some people have said AI Art Generation
is no different than the introduction of Photoshop to the industry. But
right now 99% of the artists contribution to an AI image isn't that
different from doing a google image search in a large dynamically
created database. What if the AI software really was more like
photoshop, a tool that gave the artist far more control over the images
than just writing a line of text. This is what I hope to approach in
this talk.
So let's go big
picture. What parts of the art making process would I
want the AI to help with? And what features would we want in our own
personal AI?
1) Copyright Free Dataset With
Opt In
So one of the biggest issues with AI right now is the
datasets are based on copyrighted work. In fact, in many cases, the
work of the artist themselves is part of the dataset, and without their
permission. This is unacceptable, and beyond the fear of losing their
jobs, probably the biggest beef artists have with current AI
technology. So first artists would like to have their tools use a
dataset that contains exclusively public domain work, or artwork from
other artists if those artists have given their permission for their
work to be used in this fashion. I'm sure many artists would be
interested in Opting in to have their artwork trained on if asked, but
because they weren't asked, and in most cases can't even Opt out,
people are angry and rightfully so. Some newer AIs are starting to get
tested and released that claim they use only public domain and licensed
work (such as Adobe Firefly), so hopefully they live up to their claims.
2) Training An AI On Your Own Work
If we can get datasets that have no copyrighted work in it, we could
use them as a base to which we add a few of our own pieces of artwork
to train our own AI models, which can be
kept locally and private. Basically the ability to create your own
personalized digital apprentice that mimics your work. Right now there
is software that does this, but because the larger underlying dataset
contains copyrighted material, your images may fine tune the outputs,
but a lot of the final look of the images is still based on the
copyrighted work. But if we get a clean base dataset, and then artists
can privately use their own images to fine tune the AI, I can see many
advantages to having an AI that can mimic your personal style. For
example, say I create 10 paintings of a particular alien planet, I
could use the AI to create the rest of the planet in the same style, a
task far too time consuming for a single artist to do themselves
manually.
3) Generate Mood Boards
This task is already covered for the most part with the current AI
tools that do Text to Image Generation. When making a mood board or
gathering reference for either personal use or you're an
art director and want to make a design brief for one of your artists,
you frequently go onto google images to find a bunch pictures that give
the general mood of what you're looking for in the final piece of
concept art. It's a no brainer that if you can't find what you're
looking for on
Google Images, you can ask an AI to generate some art for the same
purpose, not to be used directly in the concept art, but to be
reference.
![](concept_artist_wishlist_01.jpg)
4) Make A Sketch Photoreal
Whether it's a good thing or not, a lot of clients want their concept
art to look realistic as opposed to sketchy. And getting something to
look realistic takes a lot of time, sometimes 10-20 hours of work,
while the rough painting can be
made quite quick, and for many people is the most fun part of the
process.
While I know some concept artists and illustrators love to add all the
little details, for many they'd love a way to do the rough sketch and
then have the AI fill in the rest. An example of software that already
does this is Nvidia's Canvas, where you paint a simple sketch of
different types of landscape elements like sky, trees and mountain and
then it makes the results look real. But even pumping an image into
Stable Diffusion as an image prompt can turn a 3 min sketch into
something pretty realistic with the right settings.
![](concept_artist_wishlist_02.jpg)
5) Deeper Image To Image Controls
A common technique to direct an AI is you give it an image and a
prompt, and it creates a new image based on the text prompt, but that
also contains elements of the image. While this cool, it would be great
to have more granular control over this process. For example, allow
multiple image references, each one directs the AI in a specific way.
For example, image 1 affects the composition of the resulting image.
Image 2 affects the media style. Image 3 affects the color palette.
Image 4 affects the character's pose, etc. That way you can control
different aspects of the generated image by providing different
concrete examples.
6) Make Variations
After coming up with an initial design, it's quite common to make
variations in order to explore different details or shapes. Then you
can pick the best. Both midjourney AI
and Dall-e have methods for making variations of an
existing image, however in general I've found Dall-e seems good at
making similar
variations, Midjourney seems good at making very different variations.
Stable Diffusion has a nice slider that goes from 0 to 100%, 0 is
barely related to your image, 100% is exactly like your input image,
and all the values in between gives you a ton of control over how
different you want your image to look from the original.
![](concept_artist_wishlist_03.jpg)
7) Making Photobash Elements
Most concept artists use some combination of photo elements, 3d and
hand painting in Photoshop or another paint app to arrive at their
final concept, a technique I like calling the digital collage. Adding
AI elements in the same way we can add photo elements to a concept is
another great way to incorporate AI into your images. This is another
usage of the Text To Image method, but this time part of the results
are used directly in your final image.
![](concept_artist_wishlist_04.jpg)
8) Alpha Channels
Speaking of Creating Elements for photobashing, having an option to
create your element with an alpha channel would be super helpful. There
is software out there that can create an alpha from an image on a blank
background, but it would be even easier to be able to give a text
prompt "Mountain on a transparent background" and it gives you an
automatic alpha channel. This would be especially helpful for things
like trees, if I take a photo of a tree which I'll use in the
background of a concept painting, extracting it from the background is
a huge pain because of all the leaves. Far better to have the AI
generate the tree for you on a transparent background.
![](concept_artist_wishlist_05.jpg)
9) Lighting Direction And Time Of
Day
Another problem with photobash elements is they often have the wrong
lighting in them for whatever final painting you're trying to create.
So the ability to take an element and specify the lighting direction
would be so helpful to create elements that work in your painting. In a
similar vein, being able to give the time of day and lighting direction
in a fully AI Generated painting would also be so helpful.
![](concept_artist_wishlist_06.jpg)
10) Change The Weather
Also helpful would be the ability to change the weather
in an element or painting. So for example, if I've painted a desert
scene, and then the client says they'd like winter instead, a button
click to add snow to the image you already painted would be great!
11) Change The Material On An
Object
In a 3d model, I can take an object and change its
material from metal to wood in a couple of clicks. But wouldn't it be
helpful to do the same thing with a 2d image? If the client wants that
robot, but instead of chrome they want rusty metal, a button click to
change the material without losing the overall shape or details.
12) Tileable
Textures
When making textures for a 3d object, a common practice
is making the texture seamless so you can repeat it on a surface. While
there are plenty of manual ways to do this, having the AI make it
seamless for you would be super helpful.
![](concept_artist_wishlist_07.jpg)
13) Upresing Lowres Images And Textures
It's quite common to get an image you want to use in a
painting or as a texture in a 3d object that has too low a resolution.
Or did you ever start a painting at 1k resolution and then find out it
needs to be 4k? So why not use AI to make it higher res? There's a
number of pieces of software that does this already such as Topaz
Gigapixel AI, but would be great to have it built in directly into your
main AI platform.
![](concept_artist_wishlist_08.jpg)
14) Making Orthos / Change Camera
Position
Another task that many people don't like is making orthographic
drawings of a finished concept. So another task an AI might be good at
is taking a single concept image and producing orthographic views,
side, top, bottom, etc of the design that the artist has made. It could
compare the design you did a front view of, look through its dataset of
similar designs, and guess what that object might look like from other
angles.
![](concept_artist_wishlist_09.jpg)
A couple of notes about this.
First off, the current AI art generators are not great at producing
images from a specific camera angle. So if you ask for a character face
from a 3/4 view, you're very likely to not get a 3/4 view. So to get an
AI to produce orthos, we'd need a more reliable way to specify camera
angles. This might actually be pretty simple to do if someone fed data
into an AI that included a lot of these keywords, from what I've seen,
a lot of the current software have not prioritized these sorts of
images.
And second, it is important to mention that traditionally, a lot of
entry level concept art positions in the film and videogame industry
were all about making orthos. So even if its not the most fun job to do
sometimes, having it all automated with AI may drastically reduce the
number of entry level jobs in the field. And if people don't get those
jobs, they don't get training, and then there's no one to replace to
more senior level people when they decide to move on. Of course if AI
replaces most of us at every level of a company, then I suppose this
phenomena won't matter because there won't be senior jobs to move into.
15) 2D To 3D
If we have AI that can make orthos, then its likely we could have an AI
that could use the same technique to produce at least simple 3d models
from a 2d image. I did this manually for a test a few months back, but
a completely automated way would certainly speed up the process, and be
helpful at producing rough models to pass to the next department in
your pipeline.
![](concept_artist_wishlist_10.jpg)
16) Text To 3D and Other 3D
Features
And beyond that, having the ability to do Text to 3D, and to have a 3D
equivalent to almost every feature in this list would be helpful to
take care of the more boring parts of producing 3d models. Anyone who's
done UVing of a 3d model before, would it be great if that process were
automatic?
Conclusion
So in closing, these are some of the things I'd like to see from
an AI art assistant. In short, I'd love AI to help me do all the art
making parts that I find boring. Of course, everyone has different
elements of the art making process they find boring or exciting, by no
means are all artists the same. So what are some of your wishlist
items? The area of concept art is very broad, and the kind I do is a
small fraction of it, so I'd love to hear from concept artists working
in other fields. If you could have an AI that helped speed up your
workflow instead of replace it, what would you wish for?
Lesson 7:
Making Variations Of A Design
Using AI
In my last video, "What Would A Concept Artist Want From An
AI Tool?", I discussed the need for a feature that lets you make real
easy variations of one of your own pre-existing designs. Well a couple
of hours after I posted the video, the Ai "Stable Diffusion" added this
exact feature. So in this update video, I show off my tests using the
new Initial Image and Variations feature in Stable Diffusion's
DreamStudio.
Lesson 8:
Using AI To Make A Tileable
Texture
You may remember a month or so ago I did a video called
"What would concept artists want from an AI tool", and one feature I
didn't think about at the time was tileable textures, especially if
you're using 3d as part of your concepting process. But since that
video's release, AIs like DALL-E and midjourney have added the ability
to do this, and I can certainly say making a texture tileable was a
boring job that I have no issue seeing automated away. So here's an
example of the technique using Dall-e.