Our experiences are no longer simply real or imagined. Today, technology can change our experience of the world and introduce us to new worlds altogether. Augmented Reality (AR) and Virtual Reality (VR) are the most common forms of Extended Reality (XR), but there’s also Mixed Reality (MR). What does that mean?
It turns out, Mixed Reality can mean different things to different people. But, after reading this article, you’ll have a better understanding of what they’re saying – no matter who they are.
MIXED REALITY AND THE VIRTUALITY CONTINUUM
Extended Reality can be understood as spectrums in terms of the display technology and/or in terms of the nature and behavior of the virtual elements as well as how we interact with them. In both cases, unassisted vision lands on one end (no virtual elements) and Virtual Reality exists on the other end
In the display spectrum, Augmented Reality would land in the middle, as it is a combination of the world as it appears unaided and the world altered and augmented with virtual elements.
Another approach encourages looking at how users interact with an experience rather than merely how they view it. Virtual Reality environments are immersive and often interactive experiences like games and simulations that exist in their own virtual worlds, while Augmented Reality applications rely on the physical world to add value. Some criteria for whether an experience is classified as augmented reality, mixed reality, or virtual reality include the ability of virtual elements to react with one another as well.
Image Source
This “virtuality continuum” introduced by Paul Milgram and Fumio Kishino in 1994 is the touchstone of Mixed Reality. When companies or individuals use the term “Mixed Reality” they usually have a version of the virtuality continuum in mind, whether they realize it or not. However, these terms were introduced almost thirty years ago and the ways that these technologies have manifested since then leaves room for debate and discussion.
One Term, a Dozen Definitions
In the emerging technology space, different companies define Mixed Reality slightly differently, and there is no single universally accepted definition. While they don’t all explicitly cite the virtuality continuum, they do all address MR as incorporating both virtual elements and interactions in a physically-grounded environment. For example, ROSE uses the following definition of Mixed Reality:
“Mixed Reality (MR) allows real and digital elements to interact with one another and the user like they would in the real world. Mixed reality maintains a connection to the real world, similar to Augmented Reality, and therefore is not considered fully immersive. In a Mixed Reality environment, 3D content will react to the user the same way it would in the real world. You must have an MR device, like a headset or glasses, to view an MR experience making it less accessible than Augmented Reality.”
However, Knowing how ROSE defines uses a term doesn’t always help if you’re talking with someone from Microsoft, Meta, Varjo, or any other number of Extended Reality companies. For example, Microsoft provides the following definition of Mixed Reality: “Mixed reality is a blend of physical and digital worlds, unlocking natural and intuitive 3D human, computer, and environmental interactions.”
Meta defines Mixed Reality in more abstract terms, discussing what the experiences should feel like for users. While it’s not as quotable, it gets at the core values that consistently make up our shared understanding of Mixed Reality: A computer-assisted view of the physical world designed around a human user. With that in mind, let’s look at some examples of Mixed Reality and “Mixed Reality-like” hardware and experiences.
MIXED REALITY HARDWARE
There are dedicated Mixed Reality devices. However, these days, most Virtual Reality devices are capable of experiences that arguably qualify as Mixed Reality. Similarly, some experiences available on Augmented Reality glasses and even mobile devices may constitute Mixed Reality-like experiences.
Mixed Reality Devices
Dedicated Mixed Reality devices, like Magic Leap and Microsoft’s Hololens, were designed specifically for MR experiences and are the best (and least contested) examples of the technology.
These headsets feature transparent lenses allowing a view of the physical environment that is augmented with a holographic display. These headsets also include advanced depth sensors, cameras, and software allowing “scene understanding” for interactive virtual elements to exist in or even originate from the user’s physical environment.
However, these displays are bulky and expensive to produce. The software behind them also requires a lot more computing power than other forms of extended reality. As a result, they are almost exclusively limited to enterprise use cases. In response, other forms of hardware have a different approach to Mixed Reality-like experiences.
Some coming devices, like the Lynx-R1, offer VR and AR with MR operating as a scale between these views.
Virtual Reality Devices
Most modern Virtual Reality devices are also “Mixed Reality” devices thanks to a technique called “passthrough.” This technique augments a live camera feed of the user’s surroundings instead of using a translucent or transparent display like AR and MR
Virtual Reality devices don’t have transparent lenses that allow a user to see their physical environment directly. Instead, VR devices have a growing number of increasingly robust cameras. In addition to tracking, these cameras can reconstruct the user’s view within the VR displays and augment it to create a Mixed Reality-like experience on VR hardware.
Companies like Varjo and Meta use the term “Mixed Reality” to describe experiences enabled via passthrough. Meta’s Quest line is a great example of how this technology is developing over time. Passthrough on the Quest 2 is black-and-white, grainy, and not very useful for most experiences. Passthrough on the Quest Pro is higher quality, color, and far more interactive.
Varjo’s Reality Cloud even allows a sort of environment transfer that allows one user’s physical environment to be recreated in real time and rendered as a remote user’s virtual environment. This is another example of an experience that blurs the lines between Mixed Reality and virtual reality in ways that were likely not anticipated by Milgram and Kishino.
Experiences enabled via passthrough are MR in terms of their interactivity with the user and with the user’s environment, even if they still aren’t as fully-featured as experiences on dedicated Mixed Reality devices. However, because the display of the user’s physical environment is digitally rendered, they aren’t “pure” Mixed Reality.
Augmented Reality Glasses and Mobile Devices
Augmented Reality glasses have a transparent display so, even though the virtual aspect is handled differently by the hardware, they have a similar starting point to dedicated Mixed Reality devices. However, the experiences that these devices can offer are more limited.
This is largely because of computational constraints. Most AR glasses still use a small computing puck or a mobile phone so that they can maintain their small and mobile form factor. Even some mobile devices like smartphones can deliver Mixed Reality-like experiences using an approach similar to passthrough on VR headsets.
The device limitations prevent the full-featured environmental awareness and interactivity that makes MR so impactful. As a result, most Extended Reality applications on mobile devices and AR glasses consist of virtual elements placed into the environment by the user that remain largely non-responsive to the user and to the environment.
Ever-Changing Technologies
While developments like passthrough make Mixed Reality-like experiences more viable on VR headsets, developments in connectivity and computing help to bring these experiences to AR glasses. Shifts like cloud and edge computing are making it easier for smaller devices to do more work by moving computing to remote servers.
Changes in hardware and design also make XR experiences on mobile devices more powerful. A few years ago, simple AR on most mobile devices was impossible because of the lack of cameras and depth sensors. Between mobile devices designed with these experiences in mind and developments in software, this is rapidly changing.
MIXED REALITY EXPERIENCES
Dedicated Mixed Reality headsets remain priced outside of availability for most consumers and most applications developed for these headsets accordingly fit into enterprise or academic use cases. However, Mixed Reality as it is offered through passthrough on VR headsets has opened the door more widely to MR consumer experiences.
Medical Education
GigXR was launched in 2019 specifically to take over XR content and projects from Pearson. Since then, the company has expanded the volume and interactivity of the content that it offers – often through partnerships with imaging and technology companies.
Insight is a series of mixed reality medical education experiences created by GigXR and ANIMA RES, a 3D medical illustration company. The program requires at least one Microsoft HoloLens headset to run, allowing a student or instructor to manipulate virtually reconstructed organ systems in real-time. Additional viewers can join on headsets or on 2D platforms.
Design and Training
Campfire uses its own in-house headset to view 3D models in a user’s environment that can be viewed and annotated collaboratively in real-time regardless of whether the users are together or remote. Like GigXR, not all participants need to have a headset. In fact, users without a headset can still interact with the model on desktop or mobile devices – just not in MR. The device is used for product design, as well as for training and education use cases.
Entertainment
“I Expect You to Die” is a Virtual Reality game series from developer Schell Games. However, with the “Home Sweet Home” installment of the series, a player’s den becomes a mini escape room, thanks to Mixed Reality.
The free-to-play game runs on either the more rudimentary passthrough of the Meta Quest 2 or the more powerful MR display of the Quest Pro. Through clever tech and clever story writing, the experience incorporates elements of the player’s home environment into the plot.
TOMORROW’S TECHNOLOGY, TODAY’S WORLDS
There is a debate about which experiences and devices really qualify as “Mixed Reality.” Many people see this tension as unnecessary, arguing that most average users don’t use these terms anyway. While the term is valuable to specialists today, it is interesting to wonder what will happen to it as technological advances bridge the gap between AR and MR.
Sign Up For Updates.