In the XR world, there’s an up-and-coming medium in town, one that is now gaining traction with technical innovations and an expanding library of titles: Mixed Reality (MR).
MR is an immersive technology that falls between augmented reality (which overlays graphics onto your physical surroundings) and virtual reality (which completely immerses you in a digital environment). In MR, users can view their immediate environments, see their friends and interact with virtual objects in real-time. Mixed reality utilizes passthrough, “a headset function that allows users to ‘see through’ the displays and get a view of their physical surroundings while wearing the immersive device,” according to VIVE.com. MR relies on cameras, sensors and spatial computing to map one’s physical environment.
Apple Vision Pro, Meta Quest 3 and 3S, Magic Leap 2, Samsung Galaxy XR, HTC VIVE Focus Vision and Microsoft HoloLens 2 are among the XR headsets that are MR-ready. Mixed reality is a home-entertainment medium, an enterprise and training resource and an important element in location-based entertainment (LBE), enabling scalable, immersive experiences that blend physical and digital elements.
One high-profile MR release is Star Wars: Beyond Victory – A Mixed Reality Playset, a VR/MR experience developed by Lucasfilm’s Industrial Light & Magic (ILM), published by Disney Electronic Content and released for Meta Quest headsets (Quest 3 and 3S). In the game, the player assumes the role of aspiring podracer Volo Bolus. The Arcade mode, which is in MR, transforms the player’s physical space into a holotable-style racing arena where high-speed podraces are controlled from a top-down view. The Walking Dead: Saints & Sinners – MR Edition (Skydance Interactive), LEGO Bricktales MR (Thunderful Publishing) and Ghostbusters: Rise of the Ghost Lord (Sony Pictures Virtual Reality) are among titles with mixed reality modes.
“MR has contextual understanding of the environment, so the digital elements appear to interact with the physical ones,” Tuong Huy Nguyen, Director Analyst at Gartner, Inc, notes. “[This] makes experiences more dynamic and immersive and creates a seamless digital-physical experience.”
“Mixed reality matters because it connects the digital and physical worlds in a way that makes sense for how we live,” Ben Grossmann, CEO and Co-Founder of Magnopus, explains. “For a long time, tech has forced us to choose – look at a screen or look at the room around us. Mixed reality bridges that gap. Digital information and objects are aware of each other and can co-exist, whether they exist visually inside our physical spaces or whether they’re just aware of each other and working together.”
Grossmann continues, “As wearables get better and more accessible, mixed reality is starting to reach more people. We’ve moved past the gimmick phase; companies and creators are now using it to solve real-world design problems, build games and create entertainment that wasn’t possible a few years ago.” Founded in 2013, Magnopus is a developer of immersive experiences based in Los Angeles and St. Albans, U.K. The firm has worked on projects such as Mission: ISS: VR Experience. “Whether we’re building for AR, VR, web or mobile, our goal has been to connect the physical and digital worlds and make sure it’s a great experience across all of them.”
Magnopus has worked with Meta for more than a decade on experimental projects. Grossmann notes, “We’re often brought in to test and build experiences for their upcoming hardware and features while those core features are still in development, like our early work on hand tracking for Elixir – a virtual reality experience designed to place magic in the hands of the audience. When Meta launched the Presence Platform [a suite of machine perception and AI capabilities to enable MR developers] for the Meta Quest, we were invited to showcase the capabilities to the developer community.”
Cryptic Cabinet is a mixed-reality escape room experience for Meta Quest headsets, built by Magnopus. “A core challenge with MR is that every user’s physical environment is unique. You can’t design a one-size-fits-all digital world when one person is playing in a small office, and another is in a spacious living room,” Grossmann notes. “We created the MR escape room Cryptic Cabinet to showcase how developers can leverage Meta’s Presence Platform features – like the Scene API, Passthrough and Shared Spatial Anchors – to build immersive experiences tailored to any room.”
In terms of spatial computing or the general XR ecosystem, mixed reality is a “super important piece of the puzzle,” according to Mathieu Castelli, Chief Creative Officer of Stockholm-based Resolution Games. “By integrating MR capabilities into several of our existing [VR] games – including Demeo, Angry Birds VR: Isle of Pigs, Spatial Ops and others – MR has greatly enriched player experiences, often resulting in longer play sessions.” Resolution Games has been around since 2015 and has a dozen MR titles.
“Apple Vision Pro has been a great medium for really introducing the idea of MR – or spatial computing as Apple calls it – as a more ‘normal’ medium because everything translates so seamlessly,” Castelli explains. “We have had the pleasure of working closely with the Apple Vision Pro to launch several exclusive games like Game Room – one of the first spatial games designed/ developed natively for Apple Vision Pro – as well as Retrocade, Gears & Goo, Spatial Ops: Campaign Edition and Pickle Pro.
Game Room takes advantage of visionOS, which features a brand-new 3D user interface and input system controlled entirely by a user’s eyes, hands and voice, ideal for tabletop gameplay and using your hands to pick up game pieces, cards and more.
Thomas Van Bouwel, an XR game developer based in Brussels, created the pioneering MR titles Cubism and Laser Dance, and has been a leader in integrating passthrough and hand tracking in MR. “Purely from an accessibility and onboarding standpoint, mixed reality can add a lot of value to apps/games as users aren’t disconnected from their environment and can still see their hands/ controllers,” Van Bouwel explains. “This is where my first game, Cubism, benefits greatly from MR. It’s not a requirement for the game, but as it aims to be a good experience for first-time VR users, the mixed reality lowers the barrier of entry a bit for new folks to try the game. Beyond that, once you combine the ability to see the world around you with the ability for the headset to understand that environment and the player’s body within it, a lot of interesting new things become possible.”
Van Bouwel was inspired to explore mixed reality at Meta Connect, an annual developer conference. He recalls, “Meta showcased a first video of The World Beyond, a short experience where the walls of your living room disappear to reveal a whole environment beyond it. Seeing that made it click for me that mixed reality could be much more than a passthrough background to some VR games [like Cubism] and could enable new types of immersive gameplay that really made use of a player’s space.” Van Bouwel notes, “At release, Cubism was a simple VR game played with controllers, but as new technologies like hand tracking and passthrough mixed reality started becoming available, the game gradually evolved into a game primarily played in mixed reality.”
Laser Dance is a game that turns almost any room into a laser obstacle course. A level in Laser Dance consists of a sequence of laser patterns, which adapt to your room’s size and layout. Players move between two buttons on opposite ends of the room, and dance, dodge and crawl their way to the finish line with each button press. Explains Van Bouwel, “Laser Dance came out of the desire to explore what type of gameplay could only be possible in MR. The game is enabled by several technologies that have only become available in the last few years. Color passthrough is the most obvious one, but more essential still is the ability to 3D scan the room you’re playing in, the ability to track the player’s upper body, and the ability to use a live depth map of the room to occlude virtual objects behind furniture and the player’s body.”
Puzzling Places is a 3D jigsaw puzzle VR game with a MR mode developed by Realities.io. It allows players to assemble miniatures of beautiful real-world locations from around the Earth and is ideal for multitaskers. “There are some users who can only play the game if there is mixed reality. If you are a parent with a young child or pets around or caring for the elderly or the sick, fully blocking out the environment around you [with VR] is not really an option. But with mixed reality, the user can play the puzzle while keeping an eye on whatever else they need to pay attention to. Since Puzzling Places is a slow-paced game, where the user decides the cognitive load, they can easily play the game in mixed reality while still paying attention to something else,” says Shahriar Shahrabi, Creative Lead at Berlin-based Realities.io Inc., which publishes Puzzling Places.
“Another factor is the matter of loneliness. Mixed reality allows people to sit next to each other, each doing their own thing while still feeling a connection,” Shahrabi comments. “The biggest wow factor of MR, as far as the game is concerned, is local multiplayer. Using the platform’s spatial features, we sync the multiplayer experience so that the pieces overlap in the real world for both players. This is a real sci-fi moment for first-timers. It feels natural to build a 3D model floating in the air, pass models around to each other, and forget about the headset or the interface.” He adds, “Last but not least, MR allows you to sip on a drink while playing. Who doesn’t want that?”
“In my opinion, what makes mixed reality so cool is that it stops isolating people. Let’s be honest: VR is like putting a bucket on your head – you are completely alone,” says Marcus Ernst, Creative Project Manager at MACK One (Europa-Park in Rust, Germany is part of the Mack Group). “Which is okay, if it fits the purpose, if the story you want to tell requires people to feel like they are in a totally different location. But this comes with a price: isolation. MR fixes this. It lets you see your hands, your friends and the real surroundings, but injects digital magic right into it – like a digital layer you put on top of it.”
YULLBE is a free-roaming VR/MR experience center at Europa-Park in Rust, Germany. “YULLBE introduced an ‘XR-Maze’ in the last edition of the horror experience Traumatica, where guests could walk through in mixed reality and follow two different storylines,” Ernst explains. “With the XR-Maze, we combined a real physical labyrinth with MR. We had the real physical environment, but we could add whatever digital kind of horror we wanted to the experiences. This allowed us to create something entirely new, since we are no longer limited to what a human actor can or cannot do in their horror character.”
Ernst says, “MR is the bridge between your physical body and the digital world. We need high-quality passthrough so people do not walk into walls, but we manipulate it. We do not just show the room; we make the real walls rot or let the ceiling collapse. It is a total reality hack, if done right. With devices like the Apple Vision Pro, we can see where the quality is heading when it comes to the passthrough. The industry finally understood that the real value lies in blending worlds, not replacing them.”
Nguyen comments, “The biggest challenge [for MR is developing] a comprehensive ecosystem. This includes a broad set of applications, services, devices, standards and protocols to ensure ease of use.” He adds, “New entertainment content will help drive awareness of MR, but it will take at least six more years before MR reaches early majority adoption (16%).”
Castelli says, “By combining the real with the imagined, mixed reality offers the best of both worlds. While there will always be experiences tailor-made for VR and AR, mixed reality provides an optimal experience for those keen to bring the magic of games into their homes and beyond. It’s also great for those who prefer to stay aware of their surroundings – and there are plenty of other great reasons to give it a try.”
