The blower was broken and I had to fix it. I could see the faulty part, highlighted in red, and how to repair it.
The blower, in this instance, was a massive chunk of machinery in a wastewater-treatment plant. And I was wearing a remarkable augmented reality (AR) headset from Microsoft called the HoloLens 2. It showed me, through a hologram over the physical hardware, what I needed to do.
It was a demonstration at Microsoft’s launch event at Mobile World Congress in Barcelona last month to show off the HoloLens 2, which was unveiled by CEO Satya Nadella at the annual conference for the wireless industry.
"This new medium shows just the beginning of what’s possible when you connect the digital world to the physical world to transform how we work, learn and play," Nadella said.
This is part of Microsoft’s new vision as a cloud-orientated company, a strategy involving what it calls intelligent cloud and intelligent edge. The "edge" is whatever device is at the periphery of the cloud and connected to servers offering a range of services.
"Computing is no longer confined to a device," Nadella said, arguing that a wide range of inputs will replace keyboards and mice, from touch to speech and gestures.
The HoloLens is a powerful example of this, as multiple demonstrations of its capabilities showed. The original model was launched four years ago and was used by space agency Nasa for its Mars Rover mission. Because of the eight-minute delay in communications from earth to the red planet, controllers needed to map, plan and execute the path the rover would take, former Nasa administrator Charles Bolden told me last year.
I had seen the tourist version of this during a trip to Seattle earlier. It depicted images of Mars so you could see the surface of the planet. But Nasa applied it for planning purposes, Bolden explained, using imagery captured by the rover’s cameras to rehearse its route and make sure it didn’t run into obstacles or get stuck in crevices.
This year’s model, which cost $3,500, is lighter, balances its weight better and has a clever mount at the back of your head. The entire computer that runs it is encased in this lightweight frame and gives you what Microsoft calls "mixed reality".
The HoloLens works by mapping your eyes and where you focus — you attune it when you first put it on, by looking at various points. The holograms are projected onto the screens in front of your eyes, on a carbon fibre face plate that can be lifted up if necessary.
Microsoft demonstrated numerous applications for which using the headset has been helpful in real-world scenarios. Instead of a technician reading a paper-based manual for assembling the complicated interior of a sleeper truck, the HoloLens showed a holographic overlay of where individual parts were placed.
And when a building was being constructed, architects and engineers could see where real-world problems might occur when conduits were moved and other pipes would not have fitted into the ceiling space.
Quoting Mark Weiser, the chief technologist at the renowned Xerox Parc research institute, Nadella said: "The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it."
The HoloLens 2 offers accurate tracking of your hands — and uses eye gestures, like looking down to scroll through pages of text, or voice recognition, when you give it verbal instructions — and even gives you haptic (touch) feedback when you interact with holograms. I was amazed when I experienced it. It really is a game-changer.
"For the first time, you’re going to experience what it feels like to touch a hologram," said Alex Kipman, a technical fellow in Microsoft’s Cloud and AI group and one of the original HoloLens inventors, "to interact with a hologram and to play with it, almost where you forget that this is a piece of digital content you’re looking at as opposed to it existing in the real world".
Arguably the most immediate use for most consumers, and the most compelling, will be in gaming. Epic Games CEO Tim Sweeney joined Kipman on stage at the Barcelona congress and said Epic’s Unreal game engine would include HoloLens functionality for creating AR and virtual reality content.
Along with the HoloLens 2, Microsoft unveiled an upgraded device that is infused with the same cloud intelligence — the Azure Kinect.
The first Kinect was a motion-tracking bar that rested on top of the Xbox gaming console and was used for tracking a player’s activity. Instead of using a handheld controller, you used your body and the console monitored your movements.
From games, it started to find other useful applications, including in operating theatres, where surgeons could use gestures to page through textbooks while in surgery — without having to physically touch anything and thereby contaminate themselves.
The Azure Kinect is a cloud-connected camera and sensor device that can track when patients might fall out of bed, as well as do a range of other useful things.
As Nadella said: "When you change the way you see the world, you change the world you see".