AI Upscaling: Can AI find what’s not there?

Can AI make things up? Well the simple answer is yes!

Artificial intelligence can be generative which means based on the information it is trained on and is presented at the moment, it is able to create something brand new!

One class of AI that is interesting here is AI that can take some content (such as a photo) and make it better. Think like Google’s Magic Eraser function on their phones.

Using artificial intelligence Google is able to erase unwanted things from your photos. When you think about it, this is an algorithm, actually generating pixles and predicating what is supposed to be there.

Another type of image prediction is image upscaling. What do you do if you have a blurry or low-resolution image? If you want to make it bigger, you will need to predict what pixels would be in between the existing ones. This is a task perfect for AI, as it can find patterns between training sets and predict highly effectively and efficiently.

Creating a Model

Creating a generative simple generative model is a relatively achievable task. With a sufficient amount of data such as a dataset from kaggle and a simple generative network (such as a GAN or Variational Autoencoder) there are good grounds for success.

Here are the two methods I took to upscaling images.:

GAN

The first method was using a Generative Adversarial Network (or GAN), this model is based on Nassau. GANs work by splitting a generational task into a generator and discriminator. This is to say that the model learns by having two iterating but competing models which makes eachother better at their task.

In this case the discriminators job is to identify between pictures that are the original size and pictures that are upscaled. This means that the upscaling will be done by a generator.

This code is able to create a discriminator based on a convolutional neural network that down samples its input and generator and can use random values and transpose / upscale into a new image. Putting this into a training algorithm means that over many iterations (epochs) the model is able to upscale images consistently.

To test this out I used a dataset with 64x64 pixel images of cats which provided a wide variety of circumstances for the model to learn from. These 64x64 images could then be turned into 256x256 images, which is very promising.

VAE

Variational autoencoders work in another way using an encoder and decoder, new outputs can be formed.

This is because VAE works by mapping data to latent space (a lower dimensional representation of input data) which a decoder tries to generate based on a point in latent space that the decoder picks.

In the context of upscaling this would look like an encoder which would be of lower resolution and a decoder which would generate a higher resolution version of the data provided.

However, with this task there were some challenges I needed to face, resulting in a VAE that wasn’t able to properly upscale images. Working on VAE upscaler is something I am working on as of the time of writing.

What is the Impact

While general upscaling tools exist to remove blurriness from pictures, upscaling becomes a tool with an impact when you consider an important data gap. If the data you have needs to be of greater resolution but it is difficult to reach that higher resolution through imaging alone, an upscaler might be the path to success.

This is the dilemma which makes AI upscaling relevant in medecine. Specifically in 3D-Bioprinting, where it is important that printed tissue look just like the original counterparts.

The only realistic way to find out what actual tissue looks like is through tissue-imaging. Technology like MRIs, Xrays and CT scan allow researchers a rare glimpse of tissue, but not at a scale that impact 3D-printers. Since the 3D-bioprinters require data that is at a scale more detailed than the imaging technology available, AI upscaling technology (such as a more advanced version of this proof-of-concept) can be used to accurately predict the data required for printers.

This becomes especially true when considering the personal tissue printing. In this case while it is unfeasible to get a personalized scan of patient tissue as data for a 3D-bioprint, using generalized information to train an upscaling model, can be a plausibble idea to overcoming a physical limitation of today’s technology.

Where This is Going

Image upscaling is just one of several examples of how artificial intelligence can be used to overcome roadblocks.

Although this project is an example of general upscaling technology, it is possible to use preexisting to make this more specialized. In the future I am planning on using imaging data available (such as xrays) to make a model that can upscale, imaging data.

Ultimately artificial intelligence especially generative AI is a clear path to a bright and healthier future.