- What Is Extended Reality (XR)?
- How XR Technologies Are Being Used Across Industries
- The Benefits of XR
- The Future of XR
Extended reality (XR) has been seeping into modern life for decades. From virtual reality (VR) arcade games in the early ’90s to the millions of augmented reality (AR) lenses available on Snap AR, the evolution of XR is gaining steam. According to the research group Statista, the global XR market size, forecast at $31 billion in 2021, is expected to swell to $300 billion by 2024.
The growing enthusiasm for XR is no doubt fueled, in part, by Mark Zuckerberg’s recent announcement of Facebook’s name change to Meta and unveiling the company’s plans for the metaverse, a 3D evolution of the Internet where people can meet in shared virtual places to work, exercise, perform, and socialize.
Applications for XR, already well-established in entertainment and gaming, are quickly expanding to new industries. Car companies are using XR for design creation, where designers can create 3D concepts of new vehicle lines and then collaboratively review the designs. Health-care organizations such as the Mayo Clinic are applying the technology to create interactive preoperative plans, assist with heart surgeries, and train surgical teams. Construction companies are outfitting installation crews with AR glasses to improve the speed and quality of framing and wiring installations. And production studios are using XR to create fantastic, three-dimensionally lifelike worlds like those found in Disney’s The Mandalorian.
“We’ve already gone beyond two-dimensional technology and into three-dimensional technology,” says Nick Roseth, chief explorer at XPLR Design. “Whether we’re talking about visualization or three-dimensional audio, like what Apple’s doing in that space, we’re creating much more realistic and immersive experiences that go beyond just looking at our phone or a tablet or a computer.”
Here’s a look at what XR is, how it’s being used across industries today, and what the future could hold for the evolving technology.
What Is Extended Reality?
Extended reality (XR) is the umbrella term for a spectrum of immersive and interactive technologies. It encompasses augmented reality (AR), mixed reality (MR), and virtual reality (VR). You can access XR through mobile devices, VR headsets and glasses, and other technologies.
VR lets you immerse yourself in a digital world, experiencing the true depth and visual richness of digital objects. With traditional human-computer interfaces, you interact indirectly through a 2D view into a 3D world. But with VR, you are immersed in the world of that object and can experience it as if it were real.
AR lets you bring virtual objects to the real world, in an interactive and spatially consistent way with real objects. Unlike virtual reality, where you enter the world of the object, augmented reality brings the object to you, in your world.
MR describes the continuum between the virtual and the physical worlds. On one end of the spectrum are VR experiences; on the other end are AR experiences. Adding real-world elements, such as pass-through video, to a VR experience will push that experience toward physical reality. In contrast, adding virtual objects that replace or disguise the real world in an AR experience will push that experience toward digital reality. Imagine a hologram that can be placed in the physical world as though it were present.
How XR Technologies Are Being Used Across Industries
In November 2021, leaders from enterprise companies such as Boeing, Mayo Clinic, McKinstry, and Bristol Myers Squibb gathered at the Augmented World Expo (AWE) in Santa Clara, CA, to discuss current and projected uses for XR. A term heard repeatedly at the conference was transformation. This new kind of transformation implies a more radical departure from the era of Rolodexes, drafting tables, and paper—a wholesale organizational shift where 3D immersive technologies that blend virtual and physical worlds are changing the way teams collaborate, design and develop products, and serve their customers.
Using XR in the Architecture, Engineering, and Construction Industry
The architecture, engineering, and construction (AEC) industry is a leader in adopting XR: It’s being used to plan building projects, review and refine mockups, improve coordination and quality control on jobsites, and manage buildings across their lifecycles.
1. Project Planning
By overlaying cutaway sections of 3D intelligent models onto real-world sites, firms such as Gilbane Building Company, McKinstry Construction, and Gensler are using XR to test speculative scenarios before making costly decisions. Just as a prospective buyer at IKEA can use an AR app to place digitally rendered 3D furniture in her living room to check if it’s correctly sized and matches the decor, firms can use third-party apps to determine if design plans are congruent with actual site conditions.
“If it’s an empty parking lot, you can actually project the 2D plan in the physical space of that empty lot or visualize the 3D model using an augmented reality app,” says Viveka Devadas, technical account specialist at Autodesk. “That’s a lifesaver because you can identify potential clashes in the early planning and conceptual phases.”
Bringing designs from 3D modeling programs such as Autodesk Revit into VR at early stages can give architects a more visceral feel for a space before expensive or irrevocable decisions are made. “In architecture, you don’t get the feeling of the actual space you’re building,” says Hans Kellner, senior manager and principal engineer for Autodesk’s Emerging XR Technology division. “XR has really helped with that. Now, an architect can go into an immersive environment in VR and realize, ‘This volume is a lot larger than I expected,’ or, ‘It actually feels cold because it’s too large.’”
2. Design Collaboration and Review
Design collaboration no longer needs to be tethered to a physical office or local server. Assuming they have access to an XR-capable device, collaborators from nearly anywhere in the world, represented as avatars, can co-locate inside the same 3D model to conduct design reviews.
“This technology saves so much time and energy,” Devadas says. “Traditionally, we put everything on the wall, and all of us were stuck in a conference room making mockups, then going back to the computer and making changes. Now, you can make design decisions on the spot, in virtual space, and collaborate with multidisciplinary teams.”
Viewing a 3D virtual space from disparate locations is not the only advantage of XR. The configuration of a building’s skeleton—its framing, plumbing, air valves, and electrical conduits—can benefit from design-review packages such as Autodesk Navisworks that let teams navigate inside 3D models in real time and flag potential system clashes or measurement discrepancies.
3. Jobsite Coordination
Companies are using simulated VR environments during preconstruction to improve the coordination of people, equipment, and materials on jobsites. Can a crane access building supplies and move them to the correct floor without interference? What’s the best route for a supply vehicle to navigate a busy city center and approach a site? These questions are best answered before materials and equipment are procured and put in place, Kellner says, and VR provides a cost-effective means to do so.
“Here in the Twin Cities, Cargill is one of our biggest companies,” Roseth says. “During COVID, they still had to do their work but couldn’t do it with a bunch of people on the jobsite. So they were finding every [Microsoft] HoloLens they could to continue remote work and stream that information back to people at their homes. Whereas before you’d say, ‘All right, let’s 10 of us meet at this place, and we’ll do an inspection or a walkthrough or some design planning.’”
4. Real-Time Installation Guidance
At later stages of a build, AR helps installation crews see 1:1 building information modeling (BIM) data where and when it’s needed, says Dace Campbell, director of product management in construction at McKinstry. When McKinstry partnered with Emerald Development Group and Bouten Construction Company on a shared university health sciences building in Spokane, WA, crews used HoloLens headsets loaded with cloud-based Spectar software to install pipe hangers. Rather than consulting 2D plans at ground level, workers consulted holographic visual overlays of hanger bed connection points at the ceiling level.
Because installers had the designs directly in their fields of vision and didn’t have to repeatedly go up and down on scissor lifts, they worked faster, saving an estimated 3.1 minutes per hanger, or roughly 40% of the labor time. Projected across a portfolio of similar projects, the application could save McKinstry $1.4 million per year.
“XR aligns, I would argue, to a lean construction philosophy—if it is defined as providing the right information and the right materials to the right people in the right place at the right time,” Campbell says.
“Augmented reality is what I would call the ultimate stud finder. It gives people that kind of a superpower to see what’s behind the walls.”—Dace Campbell, Director of Product Management in Construction, McKinstry
5. Safety and Competency Training
One important application for XR in AEC is training employees, first responders, and building occupants. “There are technical folks and nontechnical folks on a design team,” Devadas says. “The people who actually go to a construction site might be nontechnical and have no idea where a beam is supposed to go. Then, you also have folks like firefighters, who focus on preventing safety hazards in buildings. For these people, training in VR is very important because it prepares them for events before they actually happen and helps avoid hazardous situations, which is lifesaving.”
Applying information from a 2D schematic drawing or an instruction manual to a real-world situation requires an extra layer of spatial processing that can be cognitively taxing and increase the risk of mistakes. In a high-risk situation, such as evacuating an airplane or managing the reactor of a nuclear facility, there is little room for error.
“If you think about dumping a training manual of 200 pages on somebody’s desk that they’re unlikely to read in the first place or spending a day with them going through immersive experiences and showing them what they should do under certain conditions, the benefits are obvious,” Roseth says.
6. On-Site Inspections
AR can give inspectors and facility managers a kind of X-ray vision into a building’s inner mechanics. Is there sufficient clearance to access equipment panels? Is the building in compliance with codes and Americans With Disabilities Act regulations? Are fire protection systems properly installed? Have the right number of parts been put in the right places? Although quality-control checks can be performed using more traditional markups, AR provides a more discerning lens.
“Augmented reality is what I would call the ultimate stud finder,” Campbell says. “If there was work put in place that was covered up—and there’s lots of that in construction—the facility operator needs to know it’s there. They need to rely on the BIM data, whether that was originally put in BIM or described later, using reality capture technology. Either way, augmented reality gives people that kind of a superpower to see what’s behind the walls.”
7. Physical Asset Management
Building owners and facility managers are increasingly turning to digital twins to evaluate performance data about a building’s HVAC and MEP (mechanical, electrical, and plumbing) systems and assess maintenance needs and environmental impacts. Employing embedded sensors, artificial intelligence, machine learning, and Internet of Things technologies, digital twins provide information about an asset across its lifecycle, from design and construction to operation and maintenance—even its future use. In short, they are virtual replicas of physical structures, loaded with actionable data.
Historically, this data has appeared on dashboards, but that paradigm is shifting. Inside the AEC industry, consensus is growing that presenting information in XR, where it can be seen in context, can help teams grasp information more easily.
“Imagine that you have a fully virtual representation of the building,” says Nic Fonta, director and general manager of XR at Autodesk. “For each of the managed sensors, you have the information in the 3D space presented in context. You could have trends and insights; you could do simulations—all this is better understood, consumed, and used to inform future designs when you have the information in an immersive space.”
8. Client Bidding and Sales
Most people are familiar with virtual home tours offered by real-estate tech companies like Zillow. Similar to the street view in Google Maps, these tours offer potential buyers 360-degree walkthroughs of properties. By 2025, the VR market in the real-estate sector is projected to be $2.6 billion, according to a Goldman Sachs report (PDF, p. 7).
In many ways, the use case in AEC is analogous: Both build trust in the veracity of an unseen building. In the future, VR is likely to become even more interactive, helping clients understand subtle features of a space, such as its lighting patterns and material textures. “The goal is to provide the end client a good sense of what they’re buying into and what will be built in the future,” Fonta says.
Using XR in Product Design, Manufacturing, and Automotive
With a rich history of using XR in interior and exterior car design, the manufacturing sector is poised to expand adoption of the technology. Research from PwC projects the use of XR for services and product development to lead to a $360 billion increase in GDP by 2030.
Lukas Fäth, senior product manager for Autodesk’s VRED and automotive divisions, saw VR take hold in the industry about 10 years ago. He says the use of small-scale clay models and full-size automotive prototypes for automotive concept designs has, for some companies, given way to digital prototyping, which can be performed over multiple iterations at a comparatively low cost.
XR emerged as the next logical progression. From early-stage exterior styling reviews to more detailed considerations of light reflections, human-computer interactions, and interior functionality, XR has revolutionized the way cars and other manufactured products are built.
1. Team Collaboration Across the Globe
Today, large auto manufacturers are using MR headsets supported by VRED to conduct global design reviews in immersive, photorealistic environments. At the European Design Center of Kia Motors, designers use Varjo VR headsets to transition fluidly between real and virtual cars, updating digital prototypes in real time. Streamed to teams across continents, the resolution of these visual displays, down to the glint of the metallic paint, is remarkably lifelike, Fäth says.
2. Design Review and Digital Prototyping
The argument for designing in XR, rather than from a series of physical prototypes, is rooted chiefly in economics. Creating a synthetic clay model can cost up to $900,000, so replicating it digitally “is a huge value for manufacturers,” Fäth says. Inside VR, designers, engineers, and test participants can conduct driving simulations; examine a car’s ergonomics; and assess how a car’s dashboard, steering wheel, mirror array, and digital interface can best support driver’s comfort and safety.
3. Factory and Assembly Line Planning and Design
A prime example of using VR for factory design relates to spatial clearances for assembly line workers and maintenance crews. “You can send the actual repair person into VR and say, ‘Here are some of the repairs that you will have to do. Go in and try,’” Kellner says.
Something relatively simple, such as removing a cover of an air compressor, can be complicated by surrounding machinery and access routes. VR can flag these issues prior to construction, avoiding project delays or costly rework.
4. Product Assembly and Digital Factories
Efficiencies can be found further downstream as well. Boeing’s use of the HoloLens and BARK (Boeing Augmented Reality Kit) as part of its digital manufacturing program exemplifies how AR can shave time and costs from later production stages. In a session at AWE, Laura Bogusch, the general manager at a Boeing airplane manufacturing site in Salt Lake City, says that mechanics used the system along with QR code guides to install 150 miles of cabling per plane. After reporting first-pass quality compliance of 88% and a 20% reduction in time, the company will use the system to wire its new Air Force One fleet.
Further afield, aerospace companies such as Airbus and Boeing plan to create digital factories in the metaverse. In these shared virtual spaces, 3D digital twins of aircrafts and manufacturing systems are supported by data reservoirs that, according to a report from Reuters, “stitch together every piece of information about the aircraft … from airline requirements to millions of parts to thousands of pages of certification documents.” By running sophisticated performance tests, if these systems evolve as executives and engineers hope, they could significantly improve production speed and quality.
5. Training for Facility Managers, Machinists, and Maintenance Workers
A study by the Manufacturing Institute found that 78% of surveyed companies were “very or somewhat” concerned about the exodus of the sector’s aging workforce; as of 2017, nearly a quarter of these employees were 55 or older. Facility managers may become particularly hard to find, as 40% of them are projected to retire in the next eight years. This is one reason why the use case for XR as a training tool has emerged in manufacturing.
“I wouldn’t call facility operators a dying breed,” Campbell says. “But there are fewer of them by the day; they’re retiring out of the industry, and we’re bringing in new ones. So there’s a model to say, ‘Let’s make a remote expert out of that guy who’s been turning wrenches for 50 years.’ And younger, greener staff can go out to the jobsites or the functioning buildings while senior employees advise them about how to change a filter, what sequence to turn the valves, and so on.”
“There’s the whole assembly aspect, as well: training people to assemble the car and making sure the movements they perform are ergonomic so they won’t be harmed by doing them 50 times a day,” Fäth says.
6. Co-Design With Customers
Beyond workflow efficiency, digital models aim to improve the likeliness of customer satisfaction. Although many car companies start with a physical model, Fäth says, they will then replicate that model with a digital twin in MR that has a translucent layer of interactive content on the car’s surface and in its interior.
At early design phases, customers participate in Q&A sessions to help refine key features. “Eye gaze, where you’re looking, is extremely interesting at the early design phase,” Fäth says. Using heat maps to plot where customers are looking, and for how long, can provide useful insights about the user experience and how to maximize customer satisfaction during the design phase.
7. Marketing and Sales
VR can also be used in manufacturing and product design to woo prospective clients. That’s not simply because it’s “a shiny new technology,” Campbell says. “It’s really about giving the client confidence that you understand the scope of work, and you’re in good alignment—like, this is what we’re going to do and how we’re going to do it.”
Many high-end automotive brands already offer sophisticated digital experiences at their dealerships. “You’re sitting in a virtual car, and they can change all the configurations,” Fonta says, referring to color and material options. “You have the flexibility to choose the final car you would actually buy—not just say, ‘I like the orange one more than the black one,’ then see, on the final delivery, there’s too much orange in the car.”
Using XR in Media and Entertainment
XR is seeing some of its strongest momentum in media and entertainment. From recently released VR games such as Cities: VR for Oculus (Meta) Quest 2 to Snapchat and Instagram Story AR filters that add digital layers (such as light effects or a new head of hair) to your photos, the technology is becoming familiar to many.
Whether it’s adding photorealistic scenery to video games, engaging film audiences, or creating virtual exhibitions like those found at the National Gallery in London, the convergence of virtual and real worlds is happening now. Gaming, a market valued at $11.5 billion in 2019, is projected to grow by 30% each year from now through 2027. And AR in retail is on the rise. According to a research report by Deloitte and Snap (the parent company of Snapchat), by 2025, 75% of the population—and almost all smartphone users—will be using AR frequently (PDF, p. 4). Brands such as Nike and L’Oreal have leaned into the technology, with AR experiences letting prospective customers try on sneakers or experiment with makeup using the front-facing camera on their phone.
1. Virtual Production
Hilmar Koch, director of Autodesk’s Industry Futures group in media and entertainment, says one of the newest use cases for XR is a kind of TV and film staging called virtual production. Rather than placing actors in front of green screens, resulting in the counterfeit look of, say, a Seinfeld episode when Jerry and Elaine are riding in a taxi, actors can be placed before holographic LED volumes depicting imagined settings. “This is how Lucasfilm is doing their work, nowadays, on The Mandalorian,” Koch says.
According to a blog by the UK animation and content company Studio Giggle, settings created in this way are rendered dynamically using real-time game engines so that the backgrounds move in concert with the characters. The technique makes settings look less hokey and saves money by reducing the locations needed for filming, limiting postproduction fixes, and allowing crews to film without waiting for the perfect weather.
2. Moviemaking in the Metaverse
Technology futurist Theo Priestly, writing for Medium, speculates that Travis Scott’s April 2020 virtual concert inside Epic Games’s popular battle game Fortnite portends a glimpse into future moviemaking. During the concert, attended by 27.7 million players, an avatar of the rapper performed to players inside the video game amid a spectacle of flaming microphones, comet crashes, and neon lasers. As game engines become more powerful and rendering speeds improve, Priestly says real-time, interactive VR could be possible for feature-length films.
This might seem far off, but directors already work inside VR to inform their sense of scale and depth as they envision and create scenes. “I’ve seen it live and in color with the biggest names in Hollywood,” Koch says. “Literally, Michael Bay and Steven Spielberg types will make different choices—and more assertive, effective choices—when they have the ability to see in augmented or virtual reality.”
Fundamentally, what would be different about seeing a movie in the metaverse—as opposed to in a movie theater or on TV—is that you’d be a part of it, seeing a world from inside a volumetric environment, with the autonomy to move about, fly, experience it from multiple vantages, and follow characters or storylines that interest you.
“If you’re a Boba Fett or Mickey Mouse fan, you’ll start to see them in the park, but you could also see them on your phone and then, maybe, even in your home.”—Hilmar Koch, Director, Industry Futures, Media and Entertainment, Autodesk
3. Virtual Retail, Learning, Games, and Commerce
Although everyone seems to have their own definition of the metaverse, fundamentally, it amounts to a shared virtual space where people inhabit and interact with their surroundings—like Fornite or a virtual surgical theater or a proposed site plan that clients can see in 3D through a Varjo Aero, Microsoft HoloLens, or Meta Quest headset. As writer Bernard Marr writes in Forbes, “3D environments, avatars, and gamification—three fundamental aspects of the concept—all fit well with VR interfaces.”
If the metaverse becomes as pervasive as proponents believe it will—something like the “air we breathe,” as New York Times columnist Kevin Roose described its most radical vision on a recent episode of The Daily podcast—virtual spaces could bleed into all aspects of daily life, with personal identities as plugged into the metaverse as to physical reality. Marr suggests there are already hints of that progression in e-commerce websites where “customers can ‘try on’ jewelry and clothing using digital avatars” and massive open online courses that use XR to help students visualize historical events and quantum-physics concepts.
On the flip side, 3D virtual characters could become more conspicuous presences in the material world. “No one can predict the future, but I’d be willing to bet some amount of money that we’re going to be seeing friends, virtual characters, in Disneyland,” Koch says. “If you’re a Boba Fett or Mickey Mouse fan, you’ll start to see them in the park, but you could also see them on your phone and then, maybe, even in your home.”
The Benefits of XR
As XR continues to gain ground across industries, many companies are realizing the benefits, including:
Design reviews of 3D digital prototypes can cut costs and carbon impacts by reducing the need to create and ship synthetic product models.
AR headsets with guided instructions can help employees increase the pace of assembly on a construction site or production floor. Boeing and McKinstry offer excellent use cases.
An architect or designer’s sense of scale, dimensionality, crowdedness, light, acoustics, and materiality are constrained by 2D screens. Full immersion in volumetric space can improve understanding of key features. The same principle applies to clients.
Using AR and VR at early planning stages can ensure stakeholder alignment, improve coordination of people and equipment at jobsites, reduce travel, and limit the need for rework.
Surgeons, firefighters, soldiers, nuclear technicians, construction workers—anyone exposed to potentially dangerous situations—can benefit from safety training in an immersive, lifelike environment.
Digital twins of buildings, bridges, cars, and airplanes can offer engineers, owners, and facility and operations managers performance insights across an asset’s lifecycle. This can help improve maintenance and cut energy and water use.
The Future of XR
COVID-19 changed the world in many ways—most of them tragic, some of them oddly hopeful. XR has emerged from the confines of incubators and innovation labs and rebounded from commercial flops like Google Glass to find real-world applications in offices, design and production studios, surgical theaters, and manufacturing and construction sites. One reason for this, Roseth says, is that when COVID-19 hit, teams needed to move projects forward, even when they couldn’t share the same physical spaces. VR and AR provided a digital alternative, backed by data and enriched by the verisimilitude of 3D spaces.
For a long time, clunky and expensive headsets, siloed software applications, sluggish rendering speeds, and the limited bandwidth of wireless networks have stood in the way of XR adoption, according to Kellner. But as technology has advanced, companies across the supply chain (particularly those in AEC) are starting to see its potential as an end-to-end decision-making tool that can cut waste, improve coordination, and increase economic output.
There are still plenty of hurdles to pass, Fonta says. Headsets need to become cheaper and less constrained by battery life. Image rendering speeds need to improve. Siloed software platforms need the privacy controls and infrastructure support to integrate more seamlessly in the cloud. Interactions need to become more intuitive, supporting gesture and eye-gaze signals that orient users in space and aid in sketching, prototyping, and animation. Most of all, Fonta says, geospatial tags that sync real-world location data with building models need to be automated and achieve greater precision.
There are compelling signals the technology has reached a critical mass: the growth of 5G infrastructure, expanded hardware options, and tools developed by software companies like Niantic that make it easier for developers to connect XR experiences to specific locations.
“Once we’ve figured out some of the challenges of XR, you’ll be able to be on a construction site walking around with your super-lightweight smart glasses that have the power to last the entire day,” Fonta says. “You’ll be able to recognize that the design and the as-built models are not the same and update the BIM data at the source. The data will inform the design, and it will go full circle.”
One day, XR may be as easy to get in and out of as Zoom—except instead of seeing your colleagues in a grid of tiles, like the opening credits to The Brady Bunch, you’ll be there, wherever there is.
“Let’s say I’m designing a hospital, and I need a consultant,” Kellner says. “I can quickly ping somebody, and they can appear in my design, as if we were in a real hospital room. Maybe it’s an MRI device, and I am trying to change the ergonomics of it. Instead of having an image in pictures and models and trying to explain things, I can have an ergonomics specialist right there and show them, and it’s very clear.”