Tool around in a real-time generated AI version of ‘GTAV’

Last month, you may have seen that a group of researchers created a machine learning system that could transform the presentation of Grand Theft Auto V into something that looks almost photorealistic. It turns out, at about the same time, another group of AI enthusiasts were working on something even more impressive involving Rockstar’s open world title. On Friday, YouTuber Harrison Kinsley shared a video showing off GAN Theft Auto, a neural network that can generate a playable stretch of Grand Theft Auto V’s game world on its own.

Kinsley and collaborator Daniel Kukieła made GAN Theft Auto with GameGAN, which last year recreated Pac-Man by watching another AI play through the game. GameGAN, as the name suggests, is a generative adversarial network. Every GAN consists of two competing neural networks: a generator and a discriminator. The generator is trained on a sample dataset and then told to produce content based on what it saw. The discriminator, meanwhile, will compare the output of the generator with the original dataset, in the process “coaching” its counterpart to output content that is closer and closer to the source material.

“Every pixel you see here is generated from a neural network while I play,” Kinsley says in the video. “The neural network is the entire game. There are no rules written here by us or the [RAGE] engine.”

Training a GAN is a very GPU intensive task. NVIDIA loaned Kinsley a DGX Station A100 computer to make the project a reality. The system comes with four of the company’s A100 GPUs and a 64-core AMD server CPU. Kinsley and Kukieła used all of that computing power to run 12 rules-based AI simultaneously. Those programs would drive the same stretch of highway, collecting the data the neural network needed to start generating its own game world. The two also developed a supersampling AI to clean up the output of the neural network so it wouldn’t look so pixelated.

As you can see from the video, the network models a surprising number of systems from the game. As the car moves, so does the shadow underneath it and the reflection of the sun on its back windshield. The mountains in the distance even get closer as well. That’s not something Kinsley necessarily expected the AI would do when he and Kukieła first started training the AI.

There’s also a dream-like quality to the gameplay and that’s thanks in part to the fact the neural network doesn’t replicate every aspect of GTA V perfectly. For one, collisions give it trouble. Kinsley says there was one instance where he saw an oncoming police cruiser split into two just as it was about to crash with his car.

If you want to try GAN Theft Auto for yourself, Kinsley and Kukieła have uploaded the project to GitHub. They say most computers should be able to run the demo.