Editors Blog – Feeling our way in the metaverse

Now that Facebook has rebranded as Meta, to reflect Mark Zuckerberg’s ambitions to operate in the metaverse, companies are fast-tracking haptic and sensor technologies for VR, writes Caroline Hayes

The metaverse challenge is to see and feel what’s in front of you. Picture source: eternalcreative

It’s not just Facebook  – sorry Meta,  that is looking to exploit a virtual universe that is accessed by virtual reality (VR). Fortnite was there first as its gaming platform evolved and gamers danced together and attended concerts. Nvidia’s CEO, Jensen Huang referred to its version of the metaverse as the Omniverse during its latest technology conference, and brands such as Nike have already seen its potential for VR advertising, signing a deal with video game company, Roblox to create Nikeland.

In Meta, they are taking the metaverse seriously, with its own development team in Meta’s Reality Labs research unit. The group of researchers, engineers and designers are conducting R&D into augmented reality and virtual reality in the future, from medical applications to space travel.

For participants, not only is a 3D avatar required but for the fully immersive experience, haptic technology will deliver the touch and feel sensations needed. One vision proposed by Reality Labs is to work on a virtual 3D puzzle with a friend’s 3D avatar in which you are able to pick up puzzle pieces and feel it between your fingers as you hold it up to examine its markings to see where it might be placed.

VR has progressed but users can still only see a digital version of your own hands but added haptic touch is the goal of today’s research.

Reality Labs’ research director, Sean Keller, explained why his team is excited about haptic technology in the metaverse: “We use our hands to communicate with others, to learn about the world, and to take action within it. We can take advantage of a lifetime of motor learning if we can bring full hand presence into AR and VR,” he says. “People could touch, feel, and manipulate virtual objects just like real objects — all without having to learn a new way of interacting with the world.”

The team is working on developing soft, lightweight haptic gloves that address the interaction problem currently experience by AR / VR interaction. Part of the work is to help the computer to accurately understand and reflect the wearer’s hand movements. It also has to reproduce a range of complex, nuanced sensations for the wearer such as pressure, texture, and vibration to create the feeling of holding a virtual object. 

The gloves will need to pair with a VR headset and – eventually – work with AR glasses. This means learning how people perceive the sensations of touch, how they form their hands to do particular tasks and how to couple this to the user using soft robotics and instrumented tracking systems. 

“And we’re inventing entirely new soft materials and manufacturing technologies — it’s a clean break from the past,” says Keller.

Figure 1: An early haptic glove research prototype: Picture Reality Labs Research.

One of the areas of research is creating actuators which can be fitted all over a glove to relay the feel, intensity and force of an object. Today’s mechanical actuators create too much heat, making the glove uncomfortably hot and impractical. They are also large, rigid and consume a lot of energy.

“You categorically can’t set 1,000 little motors and wires on the hand,” says Tristan Trutna, hardware engineering director at Reality labs. “There’s simply too much mass and too much heat. If you need thousands of tangible forces in different locations at different distances, you either need pneumatics, hydraulics, or high-density electroactive actuators.”

R&D has lead to soft robotics, pneumatic actuators (which use air pressure to create force) and electroactive ones which change shape when in the presence of an electrical field.

The soft actuators would be controlled by a microfluidic processor which controls the air flow that moves the actuators and controls when and to what degree, the valves open and close.

All of this has to be lightweight so as not to drag now the wearer’s hand and must be fast, in the order of milliseconds rather than the seconds performance of today. The team is also working on fluidic logic circuits to eliminate electromechanical components and reduce the number of electromechanical valves needed to control the actuators.

A haptic glove is not made of actuators alone. Another blog post will look at the options for textiles or smart fabrics that might be used to accelerate sensory information in a lightweight, wearable, moveable material that can be cut and formed into VR equipment that fits – well, like a glove.