fbpx

What Is Mixed Reality?

Our experiences are no longer simply real or imagined. Today, technology can change our experience of the world and introduce us to new worlds altogether. Augmented Reality (AR) and Virtual Reality (VR) are the most common forms of Extended Reality (XR), but there’s also Mixed Reality (MR). What does that mean? It turns out, Mixed Reality can mean different things to different people. But, after reading this article, you’ll have a better understanding of what they’re saying – no matter who they are.

MIXED REALITY AND THE VIRTUALITY CONTINUUM

Extended Reality can be understood as spectrums in terms of the display technology and/or in terms of the nature and behavior of the virtual elements as well as how we interact with them. In both cases, unassisted vision lands on one end (no virtual elements) and Virtual Reality exists on the other end In the display spectrum, Augmented Reality would land in the middle, as it is a combination of the world as it appears unaided and the world altered and augmented with virtual elements. Another approach encourages looking at how users interact with an experience rather than merely how they view it. Virtual Reality environments are immersive and often interactive experiences like games and simulations that exist in their own virtual worlds, while Augmented Reality applications rely on the physical world to add value. Some criteria for whether an experience is classified as augmented reality, mixed reality, or virtual reality include the ability of virtual elements to react with one another as well. Virtuality_Continuum Image Provided by CreatXR https://creatxr.com/the-virtuality-spectrum-understanding-ar-mr-vr-and-xr/
Image Source
This “virtuality continuum” introduced by Paul Milgram and Fumio Kishino in 1994 is the touchstone of Mixed Reality. When companies or individuals use the term “Mixed Reality” they usually have a version of the virtuality continuum in mind, whether they realize it or not. However, these terms were introduced almost thirty years ago and the ways that these technologies have manifested since then leaves room for debate and discussion.

One Term, a Dozen Definitions

In the emerging technology space, different companies define Mixed Reality slightly differently, and there is no single universally accepted definition. While they don’t all explicitly cite the virtuality continuum, they do all address MR as incorporating both virtual elements and interactions in a physically-grounded environment. For example, ROSE uses the following definition of Mixed Reality: “Mixed Reality (MR) allows real and digital elements to interact with one another and the user like they would in the real world. Mixed reality maintains a connection to the real world, similar to Augmented Reality, and therefore is not considered fully immersive. In a Mixed Reality environment, 3D content will react to the user the same way it would in the real world. You must have an MR device, like a headset or glasses, to view an MR experience making it less accessible than Augmented Reality.” However, Knowing how ROSE defines uses a term doesn’t always help if you’re talking with someone from Microsoft, Meta, Varjo, or any other number of Extended Reality companies. For example, Microsoft provides the following definition of Mixed Reality: “Mixed reality is a blend of physical and digital worlds, unlocking natural and intuitive 3D human, computer, and environmental interactions.” Meta defines Mixed Reality in more abstract terms, discussing what the experiences should feel like for users. While it’s not as quotable, it gets at the core values that consistently make up our shared understanding of Mixed Reality: A computer-assisted view of the physical world designed around a human user. With that in mind, let’s look at some examples of Mixed Reality and “Mixed Reality-like” hardware and experiences.  

MIXED REALITY HARDWARE

There are dedicated Mixed Reality devices. However, these days, most Virtual Reality devices are capable of experiences that arguably qualify as Mixed Reality. Similarly, some experiences available on Augmented Reality glasses and even mobile devices may constitute Mixed Reality-like experiences.

Mixed Reality Devices

Dedicated Mixed Reality devices, like Magic Leap and Microsoft’s Hololens, were designed specifically for MR experiences and are the best (and least contested) examples of the technology. 
Magic Leap 2 Image provided by Magic Leap https://www.magicleap.com/magic-leap-2
These headsets feature transparent lenses allowing a view of the physical environment that is augmented with a holographic display. These headsets also include advanced depth sensors, cameras, and software allowing “scene understanding” for interactive virtual elements to exist in or even originate from the user’s physical environment. However, these displays are bulky and expensive to produce. The software behind them also requires a lot more computing power than other forms of extended reality. As a result, they are almost exclusively limited to enterprise use cases. In response, other forms of hardware have a different approach to Mixed Reality-like experiences. Lynx-R1 Image provided by Lynx https://www.lynx-r.com/products/lynx-r1-headset Some coming devices, like the Lynx-R1, offer VR and AR with MR operating as a scale between these views.

Virtual Reality Devices

Most modern Virtual Reality devices are also “Mixed Reality” devices thanks to a technique called “passthrough.” This technique augments a live camera feed of the user’s surroundings instead of using a translucent or transparent display like AR and MR Virtual Reality devices don’t have transparent lenses that allow a user to see their physical environment directly. Instead, VR devices have a growing number of increasingly robust cameras. In addition to tracking, these cameras can reconstruct the user’s view within the VR displays and augment it to create a Mixed Reality-like experience on VR hardware. Companies like Varjo and Meta use the term “Mixed Reality” to describe experiences enabled via passthrough. Meta’s Quest line is a great example of how this technology is developing over time. Passthrough on the Quest 2 is black-and-white, grainy, and not very useful for most experiences. Passthrough on the Quest Pro is higher quality, color, and far more interactive. Image (c) Varjo https://varjo.com/solutions/design-and-engineering/ Varjo’s Reality Cloud even allows a sort of environment transfer that allows one user’s physical environment to be recreated in real time and rendered as a remote user’s virtual environment. This is another example of an experience that blurs the lines between Mixed Reality and virtual reality in ways that were likely not anticipated by Milgram and Kishino. Experiences enabled via passthrough are MR in terms of their interactivity with the user and with the user’s environment, even if they still aren’t as fully-featured as experiences on dedicated Mixed Reality devices. However, because the display of the user’s physical environment is digitally rendered, they aren’t “pure” Mixed Reality.

Augmented Reality Glasses and Mobile Devices

Nreal-Air Photo provided by Jon Jaehnig Augmented Reality glasses have a transparent display so, even though the virtual aspect is handled differently by the hardware, they have a similar starting point to dedicated Mixed Reality devices. However, the experiences that these devices can offer are more limited. This is largely because of computational constraints. Most AR glasses still use a small computing puck or a mobile phone so that they can maintain their small and mobile form factor. Even some mobile devices like smartphones can deliver Mixed Reality-like experiences using an approach similar to passthrough on VR headsets.  The device limitations prevent the full-featured environmental awareness and interactivity that makes MR so impactful. As a result, most Extended Reality applications on mobile devices and AR glasses consist of virtual elements placed into the environment by the user that remain largely non-responsive to the user and to the environment.

Ever-Changing Technologies

While developments like passthrough make Mixed Reality-like experiences more viable on VR headsets, developments in connectivity and computing help to bring these experiences to AR glasses. Shifts like cloud and edge computing are making it easier for smaller devices to do more work by moving computing to remote servers. Changes in hardware and design also make XR experiences on mobile devices more powerful. A few years ago, simple AR on most mobile devices was impossible because of the lack of cameras and depth sensors. Between mobile devices designed with these experiences in mind and developments in software, this is rapidly changing.  

MIXED REALITY EXPERIENCES

Dedicated Mixed Reality headsets remain priced outside of availability for most consumers and most applications developed for these headsets accordingly fit into enterprise or academic use cases. However, Mixed Reality as it is offered through passthrough on VR headsets has opened the door more widely to MR consumer experiences.

Medical Education

GigXR-and-ANIMA-RES-Insight-Kidney-module Image provided by ANIMA RES https://animares.com/ https://animares.com/ GigXR was launched in 2019 specifically to take over XR content and projects from Pearson. Since then, the company has expanded the volume and interactivity of the content that it offers – often through partnerships with imaging and technology companies. Insight is a series of mixed reality medical education experiences created by GigXR and ANIMA RES, a 3D medical illustration company. The program requires at least one Microsoft HoloLens headset to run, allowing a student or instructor to manipulate virtually reconstructed organ systems in real-time. Additional viewers can join on headsets or on 2D platforms.

Design and Training

group-teleport-dollhouse Image provided by Campfire https://www.campfire3d.com/ Campfire uses its own in-house headset to view 3D models in a user’s environment that can be viewed and annotated collaboratively in real-time regardless of whether the users are together or remote. Like GigXR, not all participants need to have a headset. In fact, users without a headset can still interact with the model on desktop or mobile devices – just not in MR. The device is used for product design, as well as for training and education use cases.

Entertainment

I_Expect_You_to_Die Screenshot provided by Schell Games https://schellgames.com/portfolio/home-sweet-home “I Expect You to Die” is a Virtual Reality game series from developer Schell Games. However, with the “Home Sweet Home” installment of the series, a player’s den becomes a mini escape room, thanks to Mixed Reality. The free-to-play game runs on either the more rudimentary passthrough of the Meta Quest 2 or the more powerful MR display of the Quest Pro. Through clever tech and clever story writing, the experience incorporates elements of the player’s home environment into the plot.

TOMORROW’S TECHNOLOGY, TODAY’S WORLDS

There is a debate about which experiences and devices really qualify as “Mixed Reality.” Many people see this tension as unnecessary, arguing that most average users don’t use these terms anyway. While the term is valuable to specialists today, it is interesting to wonder what will happen to it as technological advances bridge the gap between AR and MR.

What Is Virtual Reality?

With Virtual Reality’s increasing time in the spotlight, most people probably have some surface-level understanding of what it is. However, different types of VR experiences and how they are being used in business, education, and entertainment are unclear for many. While many people still view VR as a technology of the future, it is already being used successfully across industries.  

What Is VR?

Virtual reality is a computer display technology that places the user within a completely digital world. That world can be something that has never existed in physical space or something that could never exist in physical space. This approach is common in video games and some social and cultural use cases, but it isn’t the only approach. Less imaginative virtual worlds are often used for remote work, education, or social use cases that give participants an immersive way to access an experience without being overwhelmed by complicated controls. “Digital Twins” – exact virtual replicas of physical spaces – are also used for workplace training, tourism, architecture, engineering, and construction, and other uses. By combining a real location with VR artistry, a physical place can be reimagined throughout its history – or its future. This approach is common in gaming, entertainment, and design planning. Students can visit a virtual version of Imperial Rome. Gamers can play adventures set in historic time periods or visions of the future. Designers can envision site construction or renovation.

A Brief History of Virtual Reality

Head-worn VR goes back to the late 1960s, but the “modern era” starts with the Oculus DK1 in 2013. The device was large, expensive, and had limited use cases but won big with developers, researchers, and yes, some early adopters.  Later that year, the Oculus Rift came out to become one of the first accessible Virtual Reality headsets. Two years later, Facebook (now Meta) purchased Oculus. In another two years, HTC released VIVE, with VIVE later to become its VR division with all headsets bearing the name. That same year, Sony launched the original PlayStation VR 1, Varjo was founded( though its VR 1 wouldn’t come out until 2019), and, Pico released “Goblin” – the first stand-alone VR headset. Products have been discontinued, names have changed, other companies have come and gone, and other companies are out there with their own products. But this is the basic timeline.  

Do I Need a Headset?

A virtual reality headset is required for the greatest sense of immersion in a Virtual Reality experience and some experiences only exist within VR headsets. However, a number of VR experiences also work on desktop or even on mobile devices. Some users find these devices more comfortable as well as more accessible than dedicated VR headsets. If you do decide on using a headset for full immersion, you should know that there are a large variety of Virtual Reality headsets currently on the market. Different headsets offer different abilities and limitations, and come in at different price points.

Tethered VR Headsets

Tethered VR headsets are wired to either a powerful PC or a game console. These headsets are capable of more robust experiences, but also cost more money, require expensive computers, and might be unsuitable for some applications because of the wired connection. The headset serves as a display as well as a tracker of the head, hands and controllers, and in some cases where a user is looking. Because most of the heavy computational lifting is done on the connected console, these headsets are able to offer more full-featured experiences and better graphics. The best example in gaming is currently Sony’s PlayStation VR 2. Gamers with this system have graphics nearly unparalleled in the consumer space, as well as advanced software like eye tracking. However, the headset costs $40 more than the only gaming console with which it is compatible – the PlayStation 5, which costs over $500. An image of the VIVE Pro Eye headset, from release https://www.vive.com/us/newsroom/2019-12-05/ One of the best examples in industry is the HTC VIVE Pro 2. VIVE is known for their unparalleled tracking options, and comprehensive headset design including impressive onboard audio hardware. However, the $800 headset relies on a whole network of additional hardware including a highly-capable computer, external tracking units, and an optional wireless adapter.

Stand-Alone VR Headsets

Stand-alone VR headsets are designed to work completely on their own, although most can plug into a computer to access more content. These devices tend to be more affordable and easier to set up, wear, use, and ship. However, experiences for these devices are often less fully featured and less graphically impressive. Because the device serves as the display as well as the computer, these headsets are more likely to have inhibitive memory limitations and shorter battery life. Most stand-alone headsets come in special editions with expanded memory and are compatible with adapters like external battery packs. However, they still don’t stand up to tethered headsets for some uses. Meta Quest 2 with KKCOBVR battery pack in front of nested controllers on white background. The Meta Quest 2, starting at $400, is currently the headset to have for lightweight VR gaming and many social and remote work applications. While the headset does have limited memory and graphics capabilities, it’s more than enough for remote collaboration at work and it has a growing ecosystem of accessible games and fitness applications as well as productivity apps.

3DoF Headsets

Almost all modern headsets operate in six degrees of freedom (6DoF). Older headsets had more limited sensors that didn’t support modern controllers or tracking inputs and offered only “3DoF” – a user can look around within a 3D virtual environment but can’t move freely within it. These headsets have waned in popularity as the ability to offer more complex VR experiences has expanded. However, these headsets were sufficient for some use cases including viewing 3D images and videos. Further, they are less expensive and easier to use. As a result, they have held on in some enterprise and entertainment settings. The Pico G2 4k with accompanying remote As of this writing, the most recent 3DoF headset is the Pico G2 4K. However, Pico recently announced an upcoming third generation of their 3DoF enterprise offering demonstrating the continued utility of this often overlooked model.

Specialized Headsets

Some headsets come with even more sensors for use in assessment, diagnostics or academia, or are specifically calibrated for certain specialist applications. Finnish manufacturer Varjo is probably the leader in both categories. The $7,100 Varjo XR-3 Focal Edition is a modified version of one of the company’s existing enterprise headsets with an adjusted focal plane to optimize for near-to user interactions. The headset is specifically designed for simulation use cases including flight training. The $25,000 Galea combines the Varjo Aero headset with neurological sensors developed by OpenBCI. The VR headset will be capable of measuring a wearer’s gaze, heart rate, skin response, brain activity, and more. The headset should be in the hands of early-access groups this summer for use in human studies and advanced software computer development.

VR Controllers and Adapters

In addition to specialized headsets, specialized controllers can increase the sense of immersion in a VR experience for entertainment, simulations, or athletic training.  A trainer uses VR accessories, including a gun adapter, for increased immersion. Image from KAT VR blog https://www.kat-vr.com/blogs/news/kat-walk-mini-s-vr-arcade-training-treadmill-coming-in-july-2021 Gamers and law enforcement or defense personnel have access to attachments that turn the controllers into a reliably tracked simulation gun. Casual gamers or professional athletes working on their form can turn their controllers into golf clubs or pickleball paddles. Surgeons can use replicas of medical instruments to realistically practice complex procedures.  

Virtual Reality Industries and Use Cases

Some use cases of virtual reality like training, onboarding, and remote collaboration, could benefit literally any industry. However, there are some industries that have particularly adopted VR and that VR is particularly well suited for.

AEC and Design

During a previous technological shift, the Architecture, Engineering, and Construction (AEC) industry showed us how useful 2D computer visualization could be with their massive adoption of Computer Assisted Design (CAD). With the power of virtual reality, these files take on a whole new dimension. Automotive and product design follows a similar pattern. Entire models can be designed in VR (or an existing CAD model can be turned into a virtual environment). This makes it easier than ever for designers to get a feeling of what a building, vehicle, or product could be before so much as 3D printing a mockup.
Image (c) Varjo https://varjo.com/solutions/design-and-engineering/
(c) Varjo
One company, Treble Technologies, even uses 3D models to replicate the acoustics of a space. Through this technology, designers don’t just know what a building will look like before it’s built, they know what it will sound like.

Fashion

Physical fashion designers can benefit from VR in all of the ways that other kinds of designers can – through remote collaboration and immersive modeling. Enterprising companies are also using extended reality technologies to show fashions to retailers and wearers without the cost of physical fashion shows. However, with virtual spaces and virtual representations of users comes the need for virtual fashions. Some fashion companies, like House of Blueberry exist solely in VR and create exclusively digital fashions – even teaming up with physical fashion brands to do it.

Food, Beverage, and CPG

Most common food, beverage, and consumer packaged goods (CPG) XR activations happen in AR on the mobile phone because mobile phones are much more widely used than VR headsets (for now). However, a growing number of these experiences are increasingly immersive worlds, like CocaCola’s recent Dreamworld activation, or a more recent activation that created a personalized “metaversion” avatar from a series of interactive prompts. The author's uniquely generated "metaversion" - one of Coca-Cola's interactive XR activations.

Retail

With the emergence of virtual real estate, a growing number of retail companies are establishing presences in virtual worlds. While these may or may not sell physical goods, they are a good way of spreading brand representation to an emerging medium in a potentially impactful way. An author's screenshot of PINKO's virtual storefront by Emperia, featuring shelves of physical bags represented by 3D models. Companies don’t necessarily need to buy virtual land either. Emperia works with companies to create virtual showrooms that integrate with a retailer’s existing online presence and online payment strategies.  

Future of the Industry

VR software is continuing its history of becoming more user-friendly and more visually impressive, while the hardware becomes smaller and more affordable. As big names like Sony and Meta increasingly produce better content on more accessible devices, adoption is continuing to grow. Meanwhile, advancements in how content is created for immersive worlds on consoles, apps, and the web make it easier than ever for new and aspiring developers to leave their mark on the virtual world.

What Is Extended Reality?

Extended Reality (XR) is one of the many “-R” abbreviations used in the immersive technology space these days. With so many similar terms floating around, it’s easy to get confused. Fortunately, “XR” is a sort of umbrella term that probably includes any other “-R” term out there.  

What Is XR?

The difficulty with the definitions comes from the “X.” Depending on who you ask, it might not stand for anything. Some people use it as a placeholder, like a variable in a math problem. Some even pronounce “XR” as “X Reality.” Others use XR not to mean “any reality” but to mean “all reality” for example to discuss immersive technology generally rather than one at a time. People in this camp are more likely to say “XR” as “Extended Reality.” People have their preferences between the two “XR” uses but both can be handy in different situations depending on what you’re talking about.  A lot of companies getting into immersive activations want to do it because they’re flashy. They might know that they want to do something with immersive technology but might not know whether they want to use AR, VR, or MR. Here the first use, X Reality, can be fitting because they’ll only use one form of immersive technology but they don’t know which one. A lot of academics, journalists, and technologists use “XR” as “Extended Reality” because they’re not just talking about AR or MR or VR – they’re talking about all of these technologies at once. This use is particularly helpful when talking about solutions like Varjo Reality Cloud which operates more like AR for an on-site user and more like VR for remote users. So, what are the differences between the other R terms? Why might it or might it not be important to specify how they are being grouped?  

The “-R” Abbreviations in Immersive Technology

VR, AR, MR – in all of those familiar abbreviations the “R” stands for “reality” and that’s true for “XR” as well. But, with XR being an umbrella term, it’s easier to understand if you also have a firm grasp on the other Rs as well.

AR – Augmented Reality

Augmented reality places virtual elements into a user’s view of their physical surroundings using a camera and either a transparent lens or a live-view of a camera feed often through a mobile phone. Most modern virtual reality headsets have a similar function called “passthrough” but this particular technology is still largely experimental except on professional-grade devices. The virtual elements in augmented reality activations aren’t usually responsive – they add value to the user’s surroundings, or the user’s surroundings add impact to the virtual elements. For example, in the AR lookbook that ROSE developed with KHAITE, users could see models walking in their actual surroundings or view virtual representations of items in their own homes.

MR – Mixed Reality

Mixed reality is similar to augmented reality in that it all starts with the user’s environment. However, the virtual elements in an MR experience are much more intelligent and interactive. They may interact believably with one another or with the environment. They may also collect and display information on the environment from connected devices or onboard sensors. Mixed reality requires a lot more computing power both to drive the interactive virtual elements and to display them in a meaningful way. As a result, most mixed reality experiences are made available exclusively on dedicated mixed reality devices like Magic Leap or Microsoft’s HoloLens. GigXR’s Insight series with ANIMA RES uses a HoloLens headset to display detailed and interactive anatomy models in a healthcare and education solution. If more than one person has a Hololens they can both join that session, or one presenter with a headset can stream or record a session to remote users without access to headsets.

VR – Virtual Reality

Virtual Reality is entirely virtual. The user’s natural field of view is entirely replaced by computer-rendered settings and elements, potentially including other users represented as avatars. That doesn’t mean that everything in a VR experience has to be built from the ground up. For example, products like Microsoft Mesh can place a live volumetric capture of an individual within a virtually constructed environment. Similarly, some VR experiences take place within 3D images or videos. VR is popular in gaming and social applications but is also used for remote collaboration, design, and training. In fact, 3lb XR and 3lb Games design enterprise training simulations and other solutions as well as games, cross-pollinating one another to make intuitive and immersive enterprise solutions as well as fun and challenging entertainment experiences.

The Acronym of Possibilities

Whether XR means one unspecified form of immersive technology or all forms of immersive technology together depends on who uses it and in what context. It’s also one of those terms that end users of the technology don’t really use at all – it’s primarily used at a relatively higher level of discourse. With this knowledge, you’ll probably be able to tease out what someone means when they say “XR” and if you don’t it’s okay to ask them to clarify. This is an emerging technology with an emerging dictionary of terms and everyone being on the same page is more important than appearing to understand nuanced specialist terminology.

Why is AR So Appealing to Marketers?

Augmented Reality is often touted as a user-friendly and efficient way to bring brands to consumers. However, due to the shock value of the still nascent technology and the engagement of a well-designed experience, AR can also be a great way to bring consumers into your brand.  

Augmented Experiences – Not Augmented Ads

Augmented Reality (AR) technology uses digital elements superimposed over a user’s live camera feed. Because most modern smartphones can run most AR experiences, just about everyone has access to AR content. That’s a powerful tool for companies looking to grow their brands. “Currently, for brands using AR to sell goods, it is quite common to use the technology to digitally place real-world items in the user’s environment. While this is a good application of the technology, brands would be mistaken to stop just there. With a bit more imagination AR can be used to create an experience that has a much more emotional impact on the user.” Because AR relies on the view of the user’s physical surroundings, including the objects, people, and settings that are meaningful to them, AR experiences are inherently personal in a way that no other medium is. This is helpful because a strong brand isn’t just about “stuff” either. Bridging physical and digital experiences can help to convey values that aren’t just material. Often, the most successful branded activations aren’t about selling things at all. Rather, experiential AR is about communicating with people on a personal level by letting them explore the world around them through the window of augmented reality.  

Experience Something New

Because AR is an emerging technology, we can limit ourselves by thinking about it as strictly a way to experience futuristic applications. AR can also allow users to put themselves in the past or experience another place as it is today. For brands that have long histories or a far reach, this can be a surprisingly impactful way to engage your community.

Step Into a Memory

Martin was an iconic sitcom that ran from 1992 to 1997. The show opened to cast members posing and dancing infront of solid color backgrounds and their names in the show’s memorable font. This unique and memorable piece of television history was begging to get an AR twist. In 2022, the surviving cast members reunited on BET+ to celebrate the 30th anniversary of the show’s premiere. At the event, visitors had access to a screen where they could dance and pose to have the magic of AR place them into the show’s familiar opening. They could then keep and share the clips, or edit them together to make their own show openings. Fans of the series were already excited to be at the reunion, which was its own piece of Martin history. However, the AR experience allowed them to be more than viewers of an event. They were able to participate in the show’s history in a unique and memorable way.

Visit Miami Without Leaving Home

Priceless” is a promotional initiative for MasterCard holders, giving them access to membership perks including online experiences. These experiences are increasingly taking place in Augmented Reality. MasterCard recently worked with ROSE and 8th Wall to create an AR tour of artwork in Miami’s Design District. In this window to another world experience, the user’s phone became their ticket to a guided tour of the renowned art installations.  Touchscreen navigation even allowed Priceless members to move around the artwork to see them from any angle in their 360-degree virtual view just as they would if touring the Design District in person. A plane ticket to the same experience in person would have been a hefty gift from MasterCard and a hefty commitment from card holders, but the AR experience was achievable for both.  

See Yourself Differently

In the Martin example, the background was all that was augmented and the people stayed the same. However, AR filters and lenses – the joy of modern social media – can help viewers see themselves in new ways as well.   Using AR for social media marketing is also a good business strategy. Social media users use the platform to share their lives with their friends as well as to share in the lives of their friends. A well-designed AR experience can bring viewers into your brand but viewers are also more likely to share their experiences with their own followings.

Enter the Maddenverse

Clothing company Steve Madden already has a strong conventional social media strategy, which encourages customers to tag the company in social media posts that feature themselves wearing Steve Madden apparel. The company can then feature these customers’ user-generated content on its own social media platforms and website, both of which provide purchase options. In 2021, the company decided to get more immersive in their social media campaigns and launched “the Maddenverse.” For one activation, the company worked with ROSE to produce an AR filter for Instagram that turned user selfies into avatars of Steve Madden models. Users were again encouraged to share the images and tag the company’s profile. Like the Martin experience, this Maddenverse activation didn’t cost any money for users or make any money for the company. That wasn’t the point. Rather, the experience allowed fans to express their brand support in a new and fun way, growing their loyalty to the brand while also encouraging them to put the brand in front of their own social media followings. In just one week, almost 18,000 people used the filter to create personalized AR images of themselves in the Maddenverse. The users sharing those images resulted in a total 675,000 impressions in the first week. This illustrates the kind of scale that using AR for social media marketing can achieve when users are encouraged to share their creations with others.  

Give Your Audience Whatever They Want

Customers aren’t just customers anymore. They can be your audience, but they can also be creators working in a sort of partnership as casual ambassadors for your brand. This has huge potential, but it will only work if you cultivate a meaningful relationship with them. Experiential AR and using AR on social media can help to remind your audience why they’re passionate about your brand and it can allow them to express that passion to others. But it  may mean rethinking what you want to give your community and what your community wants from your brand, other than just a purchasable product.

How AR Impacts Shopping in the Fashion and Retail Industry

Extended Reality can replace a lot of things, like most in-person work meetings, or product design. But, it will never replace the runway or the changing room. Right? Buying clothing can be a hands-on process that can feel very intimate. And, for some situations, that’s not likely to change. But, consumer fashion is increasingly being helped by augmented reality, from virtual clothing try-ons to virtual fashion shows.  

The Role of AR in Retail

Augmented reality has a growing role in the fashion industry, from clothing design to completely virtual clothing. But, for the average shopper buying a physical garment, what good is AR? AR (displaying virtual elements in a user’s physical environment through the use of smart devices like phones) allows the shopper to get a fair understanding of a garment without having direct access to that garment. Throughout the rest of this article, we’ll talk about things like fashion shows viewable from anywhere, or trying on a clothing item before it’s even in the store. AR technology means viewers don’t have to travel to a fashion show to see the latest looks – they can do it from home.  They can see what a clothing item would look like on them without going to the store, seeing if it’s in stock, and trying it on. These factors, and many others, reduce cost for manufacturers and even result in more satisfied customers less likely to return items they buy. It’s true, augmented reality might not let you see exactly how different light would dance off of a reflective bauble, or let you feel the material on your skin. At least, not yet. Still, it’s never been easier to get to know a garment without having it in your hands.  

Virtual Fashion Shows

Fashion shows are one of the industry’s standard methods of introducing the world to their new products. These are the first opportunities for people outside of the designer’s studio to see what a garment looks like on a person, how it flows and moves. However, conventional fashion shows are typically restricted to people in the fashion industry. Even if the average person could find the time and money to travel to one, they probably wouldn’t be allowed in the door. AR helps to bring fashion shows to the people. By replacing the catwalk with a capture studio, experienced designers can make high-quality virtual versions of fashion models. Apply a little techno-wizardry, and these models do their walks wherever a viewer points their mobile device. If the viewer is only interested in a few looks in the collection, they can view those fashions without sitting through the whole show.

See it in Action

Bloomingdale’s  created an AR fashion show for its 150th anniversary. ROSE created it using designs exclusively available for the celebration. The fashion show was visible in Bloomingdale’s stores, or in the homes of over 400,000 shoppers who received an AR-activated catalog. You can still view the experience by scanning a QR code on Bloomingdale’s website. Bloomingdale’s reported a 38 percent increase in shopper engagement and a 22 percent increase in conversions. A lot of those conversions were thanks to a click-to-buy feature that allows shoppers to purchase looks from within the experience just by tapping their favorite fashions.  A similar activation by ROSE and KHAITE led to a 400 percent increase in sales. Further, users browsed an average of 16 looks spending over four minutes in the experience. An experience for Selfridges that was only available in-store caught an average 51 seconds dwell time. Compare that to the amount of time that shoppers spend looking at mannequins.

Becoming the Model

Watching a fashion show can be fun. But, what about being the model? ROSE worked with Steve Madden for an initiative in “the Maddenverse.” This time, it was an Instagram effect that turned users into a stylized Steve Madden virtual avatar. In this case, the idea wasn’t to realistically represent clothing that the viewer could actually buy but rather to give them a fun opportunity to engage with the brand itself. But that’s a topic for another day.  

Virtual Try-Ons and Samples

Seeing an outfit on a model is nice, but when people buy an item, they’re going to care more about how that item looks on them. This can be trickier with augmented reality, but it’s possible. Creating a virtual version of a garment can allow a prospective buyer to see how it works with them. They can see how it matches other items in their collection, their skin color and makeup choices, their hairstyles, and even places where they anticipate wearing the garment like their home or their favorite restaurant. AR tryon in eCommerce may present another “hurdle” – it’s an extra screen tap – but many shoppers find it fun. Further, they’re statistically less likely to return items that they purchase after engaging with them in AR. In fact, ROSE worked with Adidas to create a virtual model of one of their shoes that buyers could “unbox” on social media before the physical shoe shipped. Creating a virtual clothing item can be a lot of work. But, increasingly, that work is already done. Designers and manufacturers frequently start with a digital model because it’s easier to see potential changes than with physical prototypes. In some cases, these design models can be adapted for virtual try-on cases.  

The Future of Virtual Try-On

We’ve discussed some things AR clothing doesn’t do too well. For example, faithfully replicating reflective surfaces, or the way that a fabric moves – or even how a garment will actually fit. Fortunately, all of these are aspects of the technology that are improving year-over-year. While you’ll (probably) never be able to feel an AR garment before you have the physical version, effects like how a piece of clothing reflects light in your environment are improving. In terms of fashion try-on, it was largely pioneered for jewelry in particular but we’ll likely see the approach extended to other materials as well. If you’ve tried virtual tryon before, you have probably had one or two unsatisfactory experiences. Either the jacket that you’re trying on doesn’t move at all, or those earrings that you’re trying on make it look like there’s an earthquake. These technologies are improving too, but there are two big hangups. These effects are powered by physics engines. Different platforms can use different physics engines, so it can be hard to get a quality experience without optimizing for each app and website that you want to publish on. Further, the more advanced the models and effects are, the heavier the experience is. To reach customers where they are, you’ve got to publish on the apps and devices they use. Sometimes that can mean making compromises. Fortunately, this is also being solved by the move toward cloud and edge computing that takes some of that burden off of a user’s device. After all, movie-quality virtual clothing couldn’t be real-time rendered too well within Snapchat. And even that tech is tricky to pull off realistically. That’s one of the reasons Hollywood loves costumed and masked characters so much. That kind of work is closer to VR than AR.  

The Final Frontier of “Virtual Clothing”?

Right now, we’re talking about virtual clothing as a way to drive purchases of physical clothing. However, in recent years, there has been growth in the idea of virtual clothing that stays virtual. That can mean clothing for avatars or virtual clothing that appears in photographs and videos on social media. Whether or not you might ever be interested in buying and selling digital clothing, breakthroughs in this field will likely help to improve the technology as it appears in other use cases as well.

Giving the Gift of AR this Holiday Season

Some people can be hard to shop for. There are two great ways to get around this problem.  The first is to give them something completely unique and personal to them. The second is to give them something that you already know that they love but give it to them in a unique way. Augmented Reality can be an exciting and unexpected way to explore either of these approaches.  

Isn’t VR cooler?

There’s a lot of hype around virtual reality right now – and with good reason. However, virtual reality (in addition to requiring more robust hardware) means that everything is digital. That means that everything has to be created. Items, landscapes, maybe even representations of other users. That takes a lot of time, effort, and money. Augmented reality primarily uses a person’s physical surroundings, with a couple of changes brought to you by creative technologists. That means that a single item, character, or special effect can create a completely unique experience without needing to reinvent the wheel – and everything else – on a computer.  

AR: The Gift That Keeps on Giving

What’s more than all of that, AR draws on the viewer’s connection to their physical environment. It uses computer magic to bring a little something extra to the way that they experience the places, items, and even people that they already love. That brings us back to using AR to solve tricky problems on your holiday gift list. 

Give Something Truly Unique

Everything experienced in AR is completely unique to the viewer because what is going on in the camera feed is going to be different every time. No matter how special the experience is, the physical setting where the user chooses to launch that experience makes it even more personal and meaningful. ROSE created a virtual model of the real-life Edmund Pettus Bridge for an educational AR experience that viewers could visit from anywhere in the world. Some chose to go through the experience wherever it was convenient or practical for them. But, users can also choose to place the experience in an area that has emotional significance to them. A complete experience may be difficult to give as a gift. It is possible for you to create a one-of-a-kind AR item. That could be an object or character that only exists in the digital world. It could also be a 3D model of a physical object with a special significance to the friend or loved one to whom you present it. The great thing about digital objects is that they don’t have to exist in one format. While you might choose a special experience for the initial gifting, consider giving the file of the object itself as part of the gift. That way, the receiver can take their digital object or character with them into other virtual worlds and digital experiences.

Give Something Physical – but Augment it

Some augmented reality experiences originate in the digital world and project out into the environment, like the digital objects that we were just talking about. Other augmented reality experiences start with a physical object that computer magic only enhances. In this way, you can give a “normal” gift that stands out a lot more. Patrón’s digital wrapping project took a bottle and some care to create a magical holiday gift Gifters created a personalized virtual wrapping for a Patrón bottle, including photographs, text messages, and other AR customizations. As a result, the end gift wasn’t “just a bottle of liquor,” it was a meaningful and personal one-of-a-kind experience – through the magic of AR.

Get Really Creative

Some AR gifts combine everything that we’ve talked about: a digitally-enabled personal experience, a virtual object, and a physical object with augmented value.  The adidas DEERUPT sneaker launch involved a physical box that appeared empty. Inside of that box was a grid that served as a target for a social media-friendly AR version of the shoe. This allowed fans to enjoy a product “unboxing” before the shoe was physically available. Giving a gift like this allowed a special early opening of a product naturally followed by the object itself. It’s not every day that a company does something like a virtual unboxing. However, you can apply this idea to your own gifts. Give someone a marker that launches an AR experience, even a simple one, while the “real” gift is something much bigger. That could be an item that hasn’t arrived yet, a trip someplace special, anything that you can think of. You can also use AR to let your friend or family member choose their own gift. Fashion brand KHAITE partnered with ROSE to bring models and fashions into a user’s home using augmented reality. Users got to see a personalized fashion show in their own chosen environment – and then had the option to buy the fashions that they viewed.  

Think Outside the Box

This article has provided a few ideas and a few examples. But, no article could capture all of the possibilities that AR presents for gift giving. In part, that’s because AR allows us to think outside of the box – or any other physical constraints. So, let your imagination run wild.  Freely available AR object and experience building platforms are proliferating but still require a certain amount of skill. So, this article has included links to sites that you can use to have an expert help you create a digital item or experience of your own. You can also keep an eye out for ready-made experiences from brands who are increasingly using AR in creative ways.