Artificial Intelligence and Augmented Reality
*This article was published in contribution to the new frontier of Intelligent Reality (IR). Click here to learn more about the 2022 IEEE 2nd International Conference on Intelligent Reality (ICIR).
It would be hard to find a technology more breathlessly hyped than Artificial Intelligence, but Augmented Reality is probably a close second. Both technologies share some interesting traits. In addition to being wildly popular additions to any good sci-fi novel, Artificial Intelligence and Augmented Reality have also been a topic of research for decades. AI has literally millions of citations on Google Scholar. AR has more than 400,000. Both have had some successes, but both have had their share of disappointments and setbacks. Artificial Intelligence and Augmented Reality are also both inherently complex, relying on a variety of cutting edge technologies.
For Artificial Intelligence, a principal bottleneck has been processing capability. The rise of incredibly powerful GPUs and processors optimized for AI has helped drive advances in deep learning and other new types of AI algorithm that have shown considerable promise. Applications range from serious data analysis, drug discovery and design automation, to deep fakes and whimsical AI art generators. While no true Artificial General Intelligence has been developed, narrowly focused Artificial Intelligence is clearly gaining ground.
In contrast, while Augmented Reality faces some constraints from processing capability, the far greater challenges come from the laws of Physics. For example, the quest for compact, wide field-of-view display capability is pushing the boundaries of optical technology, leading to advances in nanometer waveguide manufacturing and other technologies. An all-day wearable Augmented Reality device also has to contend with non-trivial issues of power consumption, battery life, heat, miniaturization and ergonomics. Current generation Augmented Reality devices fall into two categories: wearable and smartphone-based, and many experts believe that the future of Augmented Reality lies in a wearable device, such as an AR headset or contact lens. Within the wearable category, there are reasonably functional high end devices costing several thousand dollars or more, and lower end devices that generally offer lower performance at a lower price. However, like AI, Augmented Reality still has a way to go before it delivers on its true potential.
In this article, we will explore how and if these two futuristic technologies can work together, and what benefits such a pairing might unlock.
How artificial intelligence and augmented reality work together
It’s useful to consider augmented reality, mixed reality and virtual reality as related fields within the broader field of Extended Reality. For example, the line between Augmented Reality and Mixed Reality is so blurred that the terms can be used almost interchangeably in most contexts. Blurring the differences still further is the fact that some of the technology already developed for virtual reality can be applied to augmented reality, and vice-versa. Arguably the first available product that incorporated inside-out tracking, where the user’s position is tracked using cameras without the use of external markers or sensors, was Microsoft’s Hololens. This AR technology was later adapted and offered as part of Windows VR. Conversely, VR technology is often used to prototype an AR application in the early stages of development. More recently, some VR headsets have begun offering “passthrough” modes where the user sees a live camera feed in their headset, providing what is effectively an Augmented Reality experience.
So what happens when AI and AR technologies work together? And what technology is needed for artificial intelligence and augmented reality to work together?
The simple answer is that they already do work together, but like everything else in the fields of Artificial Intelligence and Augmented Reality, the full answer is more complex. There are already many benefits of artificial intelligence and augmented reality working together, but there are more to come. One area of growing interest is the concept of intelligence augmentation – more on that later!
Computer Vision and Machine Learning are already used in positional tracking. Imaging sensors mounted on AR headsets feed data into computer vision and machine learning algorithms in order to track the user’s position and orientation in real time. Positional tracking is a critical step in any Augmented Reality experience. Unless the AR app knows exactly where you are and how you are positioned in actual reality (i.e. the real world) it cannot calculate where to place virtual objects around you in a manner that creates a convincing and immersive mixed reality.
AI technology is also used for speech recognition in Augmented Reality experiences. Voice-based interaction can be a very natural and intuitive, hands-free user experience. Artificial Intelligence has been applied to voice recognition with considerable success, as smart home devices like Amazon’s Alexa ecosystem, and virtual assistants like Apple’s Siri, clearly demonstrate.
Another useful form of AI technology is object recognition. Whether being used to enable entertaining AR filters that replace or modify a person’s face, or being used to recognize critical components within a complex machine, object recognition has enormous potential in augmented reality. Object recognition is also useful in Virtual Reality where it can help users immersed in VR avoid the real world obstacles they are surrounded by that are hidden by the headset they are wearing.
At a fundamental level, humans are designed to process enormous amounts of visual data. The true promise of Augmented Reality is to harness that ability and combine it with Artificial Intelligence’s ability to process raw data. Working together, Augmented Reality and Artificial Intelligence can dramatically augment human intelligence in many different ways. In an increasingly complex world, augmented intelligence will rapidly become essential.
Applications of artificial intelligence and augmented reality working together
As already discussed, AI and AR are already working together in the real world to deliver immersive mixed reality experiences. Imagine a training scenario for a manufacturing company. The company wants to improve the quality and efficacy of its training. They could consider creating a virtual reality experience. This has the advantage of isolating the trainee from the real world, but it also requires that every important detail of the training process be created in 3D. In contrast, an augmented reality experience will take place in the real world. AR technology, combined with artificial intelligence and machine learning, can deliver effective training in the environment the trainee will actually work in. If computer vision technology in the form of object recognition is applied to the training scenario, the system can recognize real world objects and use them within the training scenario. Trainees can be guided to press specific buttons on a machine, use a specific tool, or place objects in a particular bin. This becomes a truly mixed reality experience, where actual reality and virtual objects blend seamlessly. This type of highly immersive training is an excellent example of the promise of augmented intelligence.
An increasingly popular use case for artificial intelligence and machine learning is to drive AI chatbots. In a well crafted chatbot, the neural network behind the AI can deliver an engaging and useful user experience. Blending these types of AI applications with augmented reality can have numerous benefits. In the hypothetical training scenario already discussed, an intelligent chatbot could take the role of a training mentor, providing additional guidance and feedback to the trainee. Challenges with AI chatbots that are caused by issues with training data, such as using inappropriate language or spreading misinformation, may be addressed through the use of synthetic data, an area of active AI research.
Of course, there are also an enormous number of potential personal applications for combined AI and AR technology. These range from serious to pure entertainment. Imagine learning a language with an AI-driven avatar as your coach, or being able to visualize your personal finances and discuss them with an AI advisor. Applications already exist to provide virtual access to rare museum objects. Combining such applications with an AI-driven tutor could be a powerful tool for learning and engagement. Social applications like SnapChat already combine augmented reality and artificial intelligence to deliver fun and engaging filters – it’s fun and easy to add an outrageous mustache to your cat , or replace your head with a talking potato. Google Maps has even trialed an "AR guidance" mode. And of course, more immersive video games with intelligent AI-driven characters will always be popular. While entertainment applications may not create augmented intelligence, they certainly help get the general public used to the idea.
Smartphone-based Augmented Reality is currently by far the most used form of AR, and it is how most people gain their first AR experience. Most of the available applications are for entertainment purposes. Despite this, many of them, such as SnapChat, incorporate surprisingly sophisticated AI.
However, more serious use of AR is on the rise. Industries using augmented reality include a broad range of manufacturing companies, from aerospace to automobile manufacturing, and an increasing number of service industries, from elevator maintenance to healthcare. Other growing areas include construction, oil and gas, retail and the military. Almost all are exploring the ways that AR and AI can work together to provide humans with augmented intelligence.
Differences between artificial intelligence and augmented intelligence
Artificial Intelligence and Augmented Reality are fundamentally different technologies, but as explored in this article, working together they can deliver intelligence augmentation that has the potential to magnify human effort and significantly enhance human intelligence. In some ways, the major differences between artificial intelligence and augmented intelligence come down to the challenges they face.
Artificial intelligence and machine learning face fundamental issues around what constitutes intelligence and how human intelligence actually works–questions we currently have few answers to. AI solutions to date have shown discouraging levels of bias, and the creation of training data sets for AI and deep learning often relies on unethical and exploitative work practices. There are also privacy issues around the big data gathered by AI systems and many other ethical considerations. Traditionally, proponents of AI have often ignored these issues, but the growing use and commoditization of AI, coupled with rising public awareness, is starting to drive change. In the future, it seems increasingly likely that any AI system will be held accountable for its actions.
Augmented Reality also faces ethical issues, ranging from data privacy to complex issues of digital rights in mixed reality environments. It is also important to consider equity and access. If an organization is going to adopt AR, will it be available and accessible to all who need to use it? Beyond issues of equity and access, usability concerns and delivering a good user experience are also significant challenges in AR. Last, but not least, there are many engineering challenges that need to be solved, from miniaturization and power consumption, to compact optical systems and high quality display technologies.
When it comes to best practices for artificial intelligence, the issues surrounding training data have to be addressed. Assuming that has been addressed, best practices for augmented intelligence should be more focused on the user experience and ensuring an optimal outcome for the users of augmented intelligence systems.
The prerequisites for artificial intelligence versus augmented intelligence are not that different. Both require a clear understanding of the proposed use case, and a carefully crafted plan of attack that ensures the end result is actually useful. Careful consideration of ethical considerations should also form a foundational part of any such plan.
Human augmented AI has enormous potential, far beyond prior concepts of intelligent automation. But any such intelligence must be deployed thoughtfully and carefully.
Advantages of using augmented reality and artificial intelligence together
As this article has explored, AI and AR technology can accomplish many things together that they cannot achieve alone. Despite being quite different technologies, the areas of overlap between them provide a rich array of opportunities, especially in the area of augmented intelligence.
We live in a world of big data, and the volume of data is growing exponentially. Human intelligence can be impressive, but it has finite limits. Numerous studies have shown that VR technology can deliver powerful learning and training experiences via a VR headset. While there is less data for Augmented Reality and Mixed Reality, early indicators are that they are just as powerful, if not more so, than Virtual Reality. But expanding our learning environment is just one potential use for augmented intelligence.
Artificial Intelligence, Machine Learning, and especially deep learning, have been advancing rapidly. Deployments of AI that previously required a powerful computer are now available as a mobile app on a smartphone. The opportunities to combine these advancements with AR technology for intelligence augmentation are almost endless. Reality will never be the same again!
2022 IEEE 2nd International Conference on Intelligent Reality (ICIR)
Want to learn more about Intelligent Reality? Why not register to attend IEEE's International Conference on Intelligent Reality (ICIR). The IEEE International Conference on Intelligent Reality aims at identifying the challenges and opportunities inherent in deploying intelligent tools and interactive disruptive technologies into immersive environments. It provides a forum for leading researchers, industry professionals, and standards experts to share their research findings and ideas.