Generative AI Investigations 2023
By Neil Blevins
Created On: May 8th 2022
Updated On: July 25th 2024

Back in 2022 and 2023, I did some investigations into using Generative AI for my art making process. While I am interested in the possibilities of the technology for the future, I feel the current technology isn't being pushed in the correct direction, and so have discontinued my exploration until a number of major issues have been sorted through (such as the use of copyrighten imagery in the datasets). Below is a set of articles, tests and tutorials I made during that year long period that I feel still have value, exploring some of the ways AI could potentially be used to enhance the artist workflow. My hope is to see the tech change course in the coming years either voluntarily or through regulations, at which point I'll give it another try.

Lesson 1:
AI Artwork, The Road Ahead

If you've been following social media these last few months, you've probably come across a ton of artwork generated by Artificial Intelligence. AI can take two different images and mix them in surprising ways, it can shade a line drawing, or it can even generate artwork by providing it a line of descriptive text. Tons of artists are starting to play with this sort of software, including myself, and so I felt now was a good time to go over some of what these sorts of applications can do, and have a brief discussion of where this may all be leading us.

Note, this is not a lesson on how to use a particular piece of AI software, this is about my experiments with them, giving you some idea of the sorts of things you can achieve with the techniques.

You have two choices with this lesson, watch me discuss the issue in the video below, or read the full text.



Lesson 2:
How To Use AI In Your Photobashing Process

AI (Artificial Intelligence) artwork has been a huge subject of experimentation and debate these last few months. Whether this is a fad or the future, who knows. But if we are going to use AI, my goal is have the AI assist me rather than replace me, I want to find that balance between my own artistic ideas / style and that of the algorithm. And while AI continues to evolve, and so workflows will as well, I wanted to share with you my current attempts at finding that balance.

As a visual artist, I've been looking at AI tools, and trying to decide how I could use their strengths without reducing my contribution to a few well chosen keywords. So this tutorial will discuss the methods I've been using to incorporate AI artwork into some of my paintings.

Note: This isn't a tutorial showing you how to use any specific piece of software, this is speaking about the process at a higher level, and can be applied to pretty much any software you'd like.

You have two choices with this lesson, watch me discuss the issue in the video below, or read the full text.



Lesson 3:
Layer Breakdown: The Final Push

As many of you know, I've been playing a lot recently with AI generated artwork, especially creating robot designs. However, one thing I've been unable to achieve in midjourney (the AI I've been currently using) is a 3/4 view of the robot designs. All of them end up being frontal view only. This is likely because the images the AI trained on either haven't been 3/4 view, or maybe they have but that particular attribute hasn't been labeled, and so the AI hasn't learned what a 3/4 perspective is. While I'm sure this will be fixed in the future, I've been playing with giving the robots depth by projecting the artwork onto simple 3d geometry. So this tutorial discusses this technique which I used to make my painting "The Final Push".

You have two choices with this lesson, watch me discuss the issue in the video below, or read the full text.



Lesson 4:
Keyword Or Placebo? Testing Midjourney Prompts

With the current crop of AI Art Generators, your two major inputs are image prompts and text prompts. This tutorial is going to be focusing on text prompts in Midjourney, and asks the question do all of the keywords people tend to add to their prompt really affect the final image? Or are they actually not contributing at all, and maybe even confusing the AI?

This tutorial shows some experiments I did with the Midjourney AI trying to figure out what are the most useful and least useful keywords, and I hope by presenting them it gives you a better idea of the best way to turn your ideas into images.



If you want to download the giant tables in the video that show the different keyword comparisons, they are here and here.

Lesson 5:
The 10 Things I've Learned Comparing Midjourney And DALL-E

So I recently got access to the AI Art Generator known as DALL-E 2, and after some initial playing, my next big question was how it compared and contrasted to the other major AI Art Generator that I've been using, Midjourney. So I ran both systems through a battery of tests, and have found some interesting results. In short, each as you'd expect have their own advantages and disadvantages. So if you've been playing with Midjourney and are wondering how it compares to DALL-E, I've compiled my 10 things I've learned comparing the two software.



If you want to download the giant tables in the video that show the different keyword comparisons, they are here and here.

Lesson 6:
What Would Concept Artists Want From An AI Tool?

The main focus I've seen so far in the world of Generative AI Artwork is how to replace the artist. But what about tools that help an artist to work faster? If the AI field could be anything a concept artist wanted it to be, what would that look like? This discussion will explore some of the things I personally would like to see from an AI tool that helps our work, as opposed to disrupting it entirely.

Features I'd Want:

  1. Copyright Free Database with Opt In
  2. Training an AI on your own Work
  3. Generate Mood Boards
  4. Make a Sketch Photoreal
  5. Deeper Image to Image Control
  6. Make Variations
  7. Make Photobash Elements
  8. Alpha Channels
  9. Lighting Direction And Time Of Day
  10. Change the Weather
  11. Change the Material On An Object
  12. Tileable Textures
  13. Upresing Lowres Images And Textures
  14. Making Orthos / Change Camera Position
  15. 2D to 3D
  16. Text To 3D and Other 3D Features

Below is a video I made on the subject in September 2022, which is still relevant, but a lot has changed in the last 8 months, so until I make a brand new video, the text version below the video is far more up to date.



Automating Your Art Process

When the software 3dsmax 2 came out in the late 90s, it introduced its first scripting language, and I used it to write tools. Some people though I did this because I liked programming, and it may shock you to know I don't particularly like programming, and am not a very good programmer. But the reason I made those tools was because something I dislike even more is doing boring repetitive tasks, and writing tools gave me a way to remove the "boring" parts from my artistic workflow. I didn't want to place every single tree by hand, I wanted to define what the forest looks like, and then the software fills in the details, which I can then later tweak. And if someone hasn't already written that software, then I was going to.

So that said, I am not new to the idea of letting software assist me in handling the parts of my art process that I don't find fun. As my friend Apurva put it, "Machine Assisted Human Expression". And AI has a lot of potential to help in that area.

But AI has a lot of potential harm as well, as artists we fear being replaced. This is bad in 2 ways.

First off, obviously, the monetary issue. A lot of people's careers are based on producing artwork for all sorts of products. Yes, there will always need to be some human artists, but if the number of jobs for a human artist decreases, a lot of people are going to be unable to pay their rent, unable to afford college for their kids, etc. Yes, artists will need to adapt their skillset, but if there's just less work to do, no amount of adaptation is going to replace that lost income.

The second issue is the larger philosophical one. If a computer can make artwork that looks good, why should I bother making art? I certainly agree that the thing we humans bring to the table is the why, it's the idea behind the art, it's the emotion, it's the message. And those skills are stuff that can only be replaced by an AI if the AI becomes sentient, which I think is still a very long ways away. But if an AI can make artwork that's good enough, if it replaces meaning with using the right combination of color and contrast to give the audience a dopamine hit, if quality is replaced with literally infinite quantity, if the audience no longer cares about meaning or expression just as long as the rods and cones in their eyes are excited, then we're going to see very few artists feel the need to produce anything, at least digitally. I've seen many people say "AI isn't making creative work, it's just remixing everything that's already been done." The thought that the audience cares a lot about originality is a wonderful fiction we tell ourselves, but at least in the film and videogame industry, a lot of work is about making something similar to what has already been done before. And these AI art generators seem really good at doing that.

But lets put these issues aside right now and imagine a world where these companies producing AIs wanted to use the algorithms to enhance human art instead of replace it. What would those pieces of software look like? If you could design your own personal AI tool, what would that tool do? If you could steer the direction of this technology, what direction would you steer it? Some people have said AI Art Generation is no different than the introduction of Photoshop to the industry. But right now 99% of the artists contribution to an AI image isn't that different from doing a google image search in a large dynamically created database. What if the AI software really was more like photoshop, a tool that gave the artist far more control over the images than just writing a line of text. This is what I hope to approach in this talk.

So let's go big picture. What parts of the art making process would I want the AI to help with? And what features would we want in our own personal AI?

1) Copyright Free Dataset With Opt In

So one of the biggest issues with AI right now is the datasets are based on copyrighted work. In fact, in many cases, the work of the artist themselves is part of the dataset, and without their permission. This is unacceptable, and beyond the fear of losing their jobs, probably the biggest beef artists have with current AI technology. So first artists would like to have their tools use a dataset that contains exclusively public domain work, or artwork from other artists if those artists have given their permission for their work to be used in this fashion. I'm sure many artists would be interested in Opting in to have their artwork trained on if asked, but because they weren't asked, and in most cases can't even Opt out, people are angry and rightfully so. Some newer AIs are starting to get tested and released that claim they use only public domain and licensed work (such as Adobe Firefly), so hopefully they live up to their claims.

2) Training An AI On Your Own Work

If we can get datasets that have no copyrighted work in it, we could use them as a base to which we add a few of our own pieces of artwork to train our own AI models, which can be kept locally and private. Basically the ability to create your own personalized digital apprentice that mimics your work. Right now there is software that does this, but because the larger underlying dataset contains copyrighted material, your images may fine tune the outputs, but a lot of the final look of the images is still based on the copyrighted work. But if we get a clean base dataset, and then artists can privately use their own images to fine tune the AI, I can see many advantages to having an AI that can mimic your personal style. For example, say I create 10 paintings of a particular alien planet, I could use the AI to create the rest of the planet in the same style, a task far too time consuming for a single artist to do themselves manually.

3) Generate Mood Boards

This task is already covered for the most part with the current AI tools that do Text to Image Generation. When making a mood board or gathering reference for either personal use or you're an art director and want to make a design brief for one of your artists, you frequently go onto google images to find a bunch pictures that give the general mood of what you're looking for in the final piece of concept art. It's a no brainer that if you can't find what you're looking for on Google Images, you can ask an AI to generate some art for the same purpose, not to be used directly in the concept art, but to be reference.



4) Make A Sketch Photoreal

Whether it's a good thing or not, a lot of clients want their concept art to look realistic as opposed to sketchy. And getting something to look realistic takes a lot of time, sometimes 10-20 hours of work, while the rough painting can be made quite quick, and for many people is the most fun part of the process.

While I know some concept artists and illustrators love to add all the little details, for many they'd love a way to do the rough sketch and then have the AI fill in the rest. An example of software that already does this is Nvidia's Canvas, where you paint a simple sketch of different types of landscape elements like sky, trees and mountain and then it makes the results look real. But even pumping an image into Stable Diffusion as an image prompt can turn a 3 min sketch into something pretty realistic with the right settings.



5) Deeper Image To Image Controls

A common technique to direct an AI is you give it an image and a prompt, and it creates a new image based on the text prompt, but that also contains elements of the image. While this cool, it would be great to have more granular control over this process. For example, allow multiple image references, each one directs the AI in a specific way. For example, image 1 affects the composition of the resulting image. Image 2 affects the media style. Image 3 affects the color palette. Image 4 affects the character's pose, etc. That way you can control different aspects of the generated image by providing different concrete examples.

6) Make Variations

After coming up with an initial design, it's quite common to make variations in order to explore different details or shapes. Then you can pick the best. Both midjourney AI and Dall-e have methods for making variations of an existing image, however in general I've found Dall-e seems good at making similar variations, Midjourney seems good at making very different variations. Stable Diffusion has a nice slider that goes from 0 to 100%, 0 is barely related to your image, 100% is exactly like your input image, and all the values in between gives you a ton of control over how different you want your image to look from the original.



7) Making Photobash Elements

Most concept artists use some combination of photo elements, 3d and hand painting in Photoshop or another paint app to arrive at their final concept, a technique I like calling the digital collage. Adding AI elements in the same way we can add photo elements to a concept is another great way to incorporate AI into your images. This is another usage of the Text To Image method, but this time part of the results are used directly in your final image.



8) Alpha Channels

Speaking of Creating Elements for photobashing, having an option to create your element with an alpha channel would be super helpful. There is software out there that can create an alpha from an image on a blank background, but it would be even easier to be able to give a text prompt "Mountain on a transparent background" and it gives you an automatic alpha channel. This would be especially helpful for things like trees, if I take a photo of a tree which I'll use in the background of a concept painting, extracting it from the background is a huge pain because of all the leaves. Far better to have the AI generate the tree for you on a transparent background.



9) Lighting Direction And Time Of Day

Another problem with photobash elements is they often have the wrong lighting in them for whatever final painting you're trying to create. So the ability to take an element and specify the lighting direction would be so helpful to create elements that work in your painting. In a similar vein, being able to give the time of day and lighting direction in a fully AI Generated painting would also be so helpful.



10) Change The Weather

Also helpful would be the ability to change the weather in an element or painting. So for example, if I've painted a desert scene, and then the client says they'd like winter instead, a button click to add snow to the image you already painted would be great!

11) Change The Material On An Object

In a 3d model, I can take an object and change its material from metal to wood in a couple of clicks. But wouldn't it be helpful to do the same thing with a 2d image? If the client wants that robot, but instead of chrome they want rusty metal, a button click to change the material without losing the overall shape or details.

12) Tileable Textures

When making textures for a 3d object, a common practice is making the texture seamless so you can repeat it on a surface. While there are plenty of manual ways to do this, having the AI make it seamless for you would be super helpful.



13) Upresing Lowres Images And Textures

It's quite common to get an image you want to use in a painting or as a texture in a 3d object that has too low a resolution. Or did you ever start a painting at 1k resolution and then find out it needs to be 4k? So why not use AI to make it higher res? There's a number of pieces of software that does this already such as Topaz Gigapixel AI, but would be great to have it built in directly into your main AI platform.



14) Making Orthos / Change Camera Position

Another task that many people don't like is making orthographic drawings of a finished concept. So another task an AI might be good at is taking a single concept image and producing orthographic views, side, top, bottom, etc of the design that the artist has made. It could compare the design you did a front view of, look through its dataset of similar designs, and guess what that object might look like from other angles.



A couple of notes about this.

First off, the current AI art generators are not great at producing images from a specific camera angle. So if you ask for a character face from a 3/4 view, you're very likely to not get a 3/4 view. So to get an AI to produce orthos, we'd need a more reliable way to specify camera angles. This might actually be pretty simple to do if someone fed data into an AI that included a lot of these keywords, from what I've seen, a lot of the current software have not prioritized these sorts of images.

And second, it is important to mention that traditionally, a lot of entry level concept art positions in the film and videogame industry were all about making orthos. So even if its not the most fun job to do sometimes, having it all automated with AI may drastically reduce the number of entry level jobs in the field. And if people don't get those jobs, they don't get training, and then there's no one to replace to more senior level people when they decide to move on. Of course if AI replaces most of us at every level of a company, then I suppose this phenomena won't matter because there won't be senior jobs to move into.

15) 2D To 3D

If we have AI that can make orthos, then its likely we could have an AI that could use the same technique to produce at least simple 3d models from a 2d image. I did this manually for a test a few months back, but a completely automated way would certainly speed up the process, and be helpful at producing rough models to pass to the next department in your pipeline.



16) Text To 3D and Other 3D Features

And beyond that, having the ability to do Text to 3D, and to have a 3D equivalent to almost every feature in this list would be helpful to take care of the more boring parts of producing 3d models. Anyone who's done UVing of a 3d model before, would it be great if that process were automatic?

Conclusion

So in closing, these are some of the things I'd like to see from an AI art assistant. In short, I'd love AI to help me do all the art making parts that I find boring. Of course, everyone has different elements of the art making process they find boring or exciting, by no means are all artists the same. So what are some of your wishlist items? The area of concept art is very broad, and the kind I do is a small fraction of it, so I'd love to hear from concept artists working in other fields. If you could have an AI that helped speed up your workflow instead of replace it, what would you wish for?

Lesson 7:
Making Variations Of A Design Using AI

In my last video, "What Would A Concept Artist Want From An AI Tool?", I discussed the need for a feature that lets you make real easy variations of one of your own pre-existing designs. Well a couple of hours after I posted the video, the Ai "Stable Diffusion" added this exact feature. So in this update video, I show off my tests using the new Initial Image and Variations feature in Stable Diffusion's DreamStudio.



Lesson 8:
Using AI To Make A Tileable Texture

You may remember a month or so ago I did a video called "What would concept artists want from an AI tool", and one feature I didn't think about at the time was tileable textures, especially if you're using 3d as part of your concepting process. But since that video's release, AIs like DALL-E and midjourney have added the ability to do this, and I can certainly say making a texture tileable was a boring job that I have no issue seeing automated away. So here's an example of the technique using Dall-e.



This site is ©2024 by Neil Blevins, All rights are reserved.
NeilBlevins.com TwitterMastodonBlueskyInstagramCaraBloggerFacebookLinkedInArtStationKickstarterGumroadYouTubeIMDB