Digital Ventures

Back to blog

Virtual or Reality: Researchers train AI to drive with help of GTA game



“How real!” I always said when I played computer game when I was young.

At that time, graphic processing competition was very high (still it is today). Back then, I heard lots of new terms to explain how one’s graphic was more superior than others. One game might said it processed data with more polygons than other games in just one shot. Another game might said it had effects that were more real. That didn’t make sense to me at all. To me, a game is a game. Only having fun playing it is good enough. Why bother caring about how real it is.

Today, as we see, games are more real. Sometimes, we cannot tell whether this is a game or a reality. Then, it rings the bell in my head. A game has to be real because it works as a “matrix” or acting like a “virtual reality” that teleports the players into another world. The more real it is, the more players would forget all about the reality they live in. Virtual reality also enables players to share the feeling and goals of the characters in the game and feel like what happens in the game really happens in life. That makes the game more fun.

Let’s take a look at AAA games, like Uncharted 4, Mass Effect and Watchdog. The three games allow us to take different role. In one game, we are a treasure hunter, like Indiana Jones. Another game lets us play the role of the captain of a spaceship on a duty in a faraway galaxy. In the other game, you will play the hacker role who has a mission of overthrowing a secret agency. All the three games look visually real. Together with good game playing technique, the game can be impressive. These games are more talked about in our country and do not promote violence like Grand Theft Auto.

In Grand Theft Auto game, we will take the role of, as you may know, a criminal. The final goal is to realize the ethical practice but along the way we may have to hurt or kill many people. We do all these in the “Open World” designed by the game without any specific order. We may choose to take a walk in the city without caring about the goal for a few hours. Many people choose to walk around seeing things for hours, eventually.

That’s the case because the game offers really beautiful scenery and visually real. It makes us feel as if we were really in the setting. How real? Well, it is as real as training AI to understand the real world.

Mark Schmidt, a researcher from British Columbia University, researched and made AI to understand the real world more. AI can understand that this is a window, a human being, a building, so and so. The problem he found is that he has to use enormous real pictures (e.g. real road) to improve AI performance. In fact, there are plenty of pictures of roads in the Internet. He, however, thinks that what would happen if he used pictures from a virtual reality game, such as the Grand Theft Auto. He wanted to see if AI will work better.


The result is that AI which trains itself from the game can work as well as, or in other cases, even better than AI which learns things from the real world.

Mark Schmidt said, “Video game graphic today is so good that we can use it to train AI using raw data and get similar result to training it using real world data.”

Real life-like visual and graphics in game can be very useful. When considering that most of the information Mark has is obtained from Citiscapes or CamVid, which have a good collection of roads in Europe only, games like the Grand Theft Auto with setting in the US (e.g. New York, Miami, San Francisco and Las Vegas) are completing the missing jigsaws. For example AI being trained in the real world in Europe where roads are narrow will not work well in the US where the roads are wider.


Another thing that visual graphics from games help AI researchers a lot is that when using pictures from the real world, researchers have to tag each object in the picture individually and manually (to help AI learn that this part has specific name). In a game, tagging names has already been integrated since the development phase (as game developers have already name those objects such as power poles and buildings. In this sense, such graphics can help shorten time for researchers and training.

Researchers believe that AI that can really learn about the world will be useful to building of driver-less cars, machinery that can work in harmony with human beings or the intelligent eye-glasses (that know what you are looking at and then display related information), etc. What researchers expect to see is that game developers will cooperate and allow them to use the information they have for researches.

Now, there is another reason for me to understand why game has to be real.