AI in Virtual Reality
*This article was published in contribution to the new frontier of Intelligent Reality (IR). Click here to learn more about the 2022 IEEE 2nd International Conference on Intelligent Reality (ICIR).
One thing that both artificial intelligence and virtual reality technology have in common is that they have been around for longer than people think. While early attempts look little like today’s solutions, both have existed in recognizable form for more than 70 years. AI technology, specifically software-based artificial intelligence, was first created in 1951 at the University of Manchester, in the UK. Two programs were created, one to play checkers and the other to play chess. If we broaden our definition of artificial intelligence, we find history is littered with examples of “automata”, mechanical figures that were capable of simple decision making. Some examples date back thousands of years.
Similarly, virtual reality can be argued to be almost 200 years old, if not older. In 1838, Charles Wheatstone invented the stereoscopic viewer, which created the illusion of a 3D scene from pairs of photographs. In 1929, Edward Link built a flight simulator that was used to train pilots prior to World War II. In the 1950s, Morton Heilig developed the Sensorama, an arcade-style cabinet that immersed the viewer in a stereoscopic display with sound, fans to generate wind, and a vibrating chair to simulate motion. In 1965, Ivan Sutherland described the first modern virtual reality system, including the use of a computer to generate the imagery for the system. But, like Artificial Intelligence, if we broaden our definition, we can find examples throughout history that meet the basic criteria of virtual reality: to artificially create a sense of immersion.
In the past decade, artificial intelligence, machine learning, virtual reality and augmented reality have all seen rapid development. Computer science breakthroughs like deep learning, and massive collections of training data have yielded impressive results. Outside the realm of AI technology and data science, breakthroughs in optics, display technology, microprocessors and algorithms have all played a role in creating ever-better virtual reality and augmented reality experiences. Consumers can interact with many different types of virtual environment, with or without a VR headset, from video games to virtual tours, to social and educational platforms.
But is there a role for artificial intelligence to play in virtual reality? Do these technologies work well together?
Common uses for AI in virtual reality today
As digital transformation gains momentum, driving technology adoption around the globe, both virtual reality and augmented reality have a role to play. A good virtual reality experience will seamlessly integrate hardware (such as a VR headset), software, and an immersive user experience. The same applies to augmented reality too. Both types of immersive experience make use of artificial intelligence in multiple ways. But in terms of application areas, adoption and usage patterns are quite different. Virtual reality with a headset is more mature than headset-based augmented reality due to the broad availability of inexpensive hardware. However, mobile phone-based augmented reality is far more common than VR.
Similarly, when discussing common uses for AI in virtual reality today, you can find some common threads and some areas of stark difference. To understand how artificial intelligence can evolve in partnership with an application area, it’s useful to look at examples outside the world of virtual reality first.
One of the industries with the longest history of implementing practical artificial intelligence is the video game industry. In video games, AI algorithms were initially used to solve the age-old problem faced by almost all games: the need for an opponent. Simplistic expert systems gave way to more sophisticated artificial intelligence, but the goal was always to provide a player with believable and challenging opponents. But video games now use artificial intelligence in many different ways, such as world building, path finding, data mining and player experience modeling. One particularly interesting application is the use of AI, machine learning and deep learning for content generation. Early work focused on creating environmental elements, music and other components. More recent work has explored the creation and design of entire game levels. Clearly, there is considerable overlap between the games industry and virtual reality. As more video games become available in virtual reality, the tools and techniques used in mainstream games are increasingly being used in VR.
Other examples of AI algorithms at work include Natural Language Processing (NLP). Multiple industries, including virtual shopping, are exploring the use of voice recognition in both virtual reality and augmented reality applications. This type of interaction is heavily reliant on artificial intelligence and machine learning.
A broadly used example of artificial intelligence that can be found in both VR technology and AR technology is computer vision. Using artificial intelligence, one or more camera inputs can be used to provide motion tracking data. This allows the virtual reality or augmented reality experience to seamlessly blend with the user’s perspective.
One emerging area of AI research is the creation of 3D scenes from single photographs. This technology is still in its infancy, but it could dramatically reduce the cost of creating an immersive experience. Another exciting area is the idea of augmented intelligence—using artificial intelligence to augment human intelligence and make humans more productive. When coupled with virtual reality, or augmented reality, augmented intelligence could deliver spectacular results. For example, imagine designing a product in VR with feedback from an AI that can access detailed knowledge of production methods and limitations. The product designer could avoid potential problems and create optimal designs far more efficiently.
However, there are limitations on the use of AI in virtual reality and augmented reality. Any VR or AR headset has to make tradeoffs between power consumption, processing power, size, weight, heat and user comfort. AI often requires significant amounts of processing power, which in turn requires more electrical power, adding weight to the headset and generating more heat. Work is being done to develop more energy efficient chipsets for AI, as well as ways to implement AI on low power chipsets. But as new and more advanced applications of AI are constantly emerging, the need to balance power and functionality is likely to be an ongoing challenge for the foreseeable future.
Enhancing virtual reality with AI
There are many ways to enhance virtual reality with AI. One of the most obvious is using artificial intelligence and machine learning to create intelligent, interactive avatars. These are virtual assistants that can understand natural conversation and respond with detailed, and highly customized, assistance. Almost any type of VR experience requires some kind of introduction and user training. Instead of providing this outside of the VR application, why not provide VR training as a VR experience? This is precisely how many video games work. They don’t provide an instruction manual for the player, they throw them straight into an immersive experience and explain the game through actual gameplay. And of course, we don’t need to stop at training for a specific application, we can use the same tools and approach to provide almost any type of training. In VR applications, it is possible to personify training material as an interactive character. When coupled with AI applications, such as natural language processing, sentiment analysis, path finding and others, it’s easy to see how immersive technology could revolutionize training.
Virtual Reality appears to have many useful applications, but training is arguably one of the most important. Training is a perennial problem in many industries. It is often slow, expensive and inefficient. Industries such as manufacturing are facing a rapidly aging workforce. As older workers retire, critical knowledge and experience is being lost. Thanks to Digital Transformation and automation, many menial jobs are disappearing. The new jobs being created increasingly require more advanced skills, which again means we need better training. It seems likely that a VR device, or an AR device, will be a key component of the future of work.
Another technological component that will impact AI is 5G. 5G communication promises high bandwidth, low latency and massive connectivity. This enables new developments in edge and cloud computing that could reduce the data processing demands placed on VR and AR headsets, allowing them to become smaller, lighter and have longer battery life while simultaneously becoming more powerful. When connected to such resources, along with IoT sensors, the possibilities for an immersive, intelligent reality become almost limitless.
Pros and cons of AI in virtual reality
The benefits of using AI in virtual reality and extended reality are clear. They include better user interaction, better user experiences, the possibility to augment human intelligence and human senses, and the ability to drive down the cost of creating an immersive virtual world. Almost any type of VR simulation can be improved through the use of AI technology. Extended reality in all of its forms can create an immersive learning environment, but when coupled with AI, it can become truly compelling. A clear opportunity for AI and extended reality is training. VR training is one application that can help transition human resources to new job opportunities. The same is true of AR training, which has the added advantage of taking place in the real world. Imagine being able to pre-train for any job simply by opening a VR app, putting on a VR device and jumping in.
Of course, AI is not without its challenges. This article previously discussed the problems of processing power and power consumption. But AI has other issues, some of them non-technical in nature. One of the biggest is bias. Any AI tool has the potential to show bias based on the big data sets used to train it. For example, speech recognition tends to work significantly better for some regional accents. There is a very strong correlation between the typical engineer working on speech recognition (male, English speaking, with an American, Indian or Chinese accent), and the accents that get the most accurate results. Accents that are outside of this group often perform poorly, making the user experience very frustrating for those users. It is important to carefully review any AI system for this type of unintentional exclusion or negative impact.
Future developments of AI for virtual reality applications
In recent years, the term “metaverse” has become increasingly popular. For many people, the metaverse is defined by sci-fi movies like “Ready, Player One”, where users can use immersive technology to enter an almost unlimited virtual world filled with new and novel experiences. Even with the latest VR technology, AR technology and immersive technology in general, we are still a long way from the metaverse being a reality. But it is also true that digital transformation is accelerating, AI applications are expanding, and immersive technology as a whole is developing rapidly.
Companies from Google, to Meta (formerly known as Facebook), to Microsoft, Nvidia, countless startups and many others are all driving innovation in artificial intelligence, machine learning, augmented reality, mixed reality, virtual reality and related technologies. While we’ve come a long way since the early days of the VR headset and the simplest ai algorithm, there is still much work to be done. The future of immersive technology, artificial intelligence and intelligent reality looks bright indeed.
2022 IEEE 2nd International Conference on Intelligent Reality (ICIR)
Want to learn more about Intelligent Reality? Why not register to attend IEEE's International Conference on Intelligent Reality (ICIR). The IEEE International Conference on Intelligent Reality aims at identifying the challenges and opportunities inherent in deploying intelligent tools and interactive disruptive technologies into immersive environments. It provides a forum for leading researchers, industry professionals, and standards experts to share their research findings and ideas.