The end of mice and keyboards?

Researchers at Meta have developed a wristband that converts the user's hand gestures into commands to interact with a computer, including moving a pointer and even converting "in-the-air" handwriting into digital text. The technology could significantly increase the accessibility of modern personal devices for people with reduced mobility or muscle weakness and open up new ways to easily control electronic devices.

In an article published in the journal Nature, the Reality Labs team described the sEMG-RD (surface electromyography research device) device, which uses sensors to convert electrical motor nerve signals traveling from the wrist to the hand into digital commands to control connected devices. These signals actually represent brain impulses that tell the hand to perform the intended action, so they can be understood as conscious instructions. You can see how it works in the video below.

Meta started developing this technology some time ago. In 2021, they assembled a team led by Thomas Reardon, who joined Reality Labs in 2019 as Head of Neuromotor Interfaces. The team's goal was to develop a prototype device for gesture control based on electromyography. At the time, the goal was to use the technology to enhance interaction in augmented reality and enable simple functions such as simulating a single mouse click. Reardon also later led the research documented in the aforementioned article.

There have been several attempts to develop similar systems on the market, including a device from 2023 that used barometric pressure sensors to recognize ten different gestures, and the Mudra Band, which uses surface nerve conduction to control the Apple Watch with simple finger movements. sEMG-RD technology goes much further. It allows not only to control the pointer on the screen in one-dimensional mode (like a laser pointer), but also to navigate the user interface and select items with gestures such as pinching, scrolling, thumb tapping, and more. In addition, by simulating writing in the air, you can enter text at a speed of about 20.9 words per minute, which is currently comparable to the average smartphone typing speed (36 words per minute).

The user adjustment system does not require individual calibration before use, but allows additional adjustment for greater accuracy. It is the development team acquired large amounts of training data from many test participants and processed it through a neural network to accurately convert raw signals into digital commands regardless of the individual user.

The use of data from thousands of subjects enabled the development of generic decoding models that interpret user input with high accuracy for different people. This eliminates the need for individual setup of the sEMG-RD device, allowing immediate and widespread use of the device – similar to how people without disabilities use a computer mouse without special calibration.

The team believes that this technology could be upgraded to also detect the intended strength of the gesture, allowing for more precise control of devices such as cameras and joysticks. In addition, it could reduce the (already minimal) physical effort required to operate smartphones and other digital devices. Perhaps even more promising is the possibility of creating completely new ways of interacting by exploiting various muscle synergies that we do not yet know or for which we do not yet have names, or by introducing new signals that the bracelet can interpret.

Source: Scimex

AI translated