Microsoft’s HoloLens has an
impressive ability to quickly sense its surroundings, but limiting it to
displaying emails or game characters on them would show a lack of creativity.
New research shows that it works quite well as a visual prosthesis for the
vision impaired, not relaying actual visual data but guiding them in real time
with audio cues and instructions. The researchers, from Caltech and University
of Southern California, first argue that restoring vision is at present simply
not a realistic goal, but that replacing the perception portion of vision isn’t
necessary to replicate the practical portion. Crunching visual data and
producing a map of high-level features like walls, obstacles and doors is one
of the core capabilities of the HoloLens, so the team decided to let it do its
thing and recreate the environment for the user from these extracted features. They
designed the system around sound, naturally. Every major object and feature can
tell the user where it is, either via voice or sound. Walls, for instance, hiss
(presumably a white noise, not a snake hiss) as the user approaches them. And
the user can scan the scene, with objects announcing themselves from left to
right from the direction in which they are located. A single object can be
selected and will repeat its callout to help the user find it.
The team recruited seven blind
people to test it out. They were given a brief intro but no training, and then
asked to accomplish a variety of tasks. The users could reliably locate and
point to objects from audio cues, and were able to find a chair in a room in a
fraction of the time they normally would, and avoid obstacles easily as well. Then
they were tasked with navigating from the entrance of a building to a room on
the second floor by following the headset’s instructions. A 'virtual guide'
repeatedly says 'follow me' from an apparent distance of a few feet ahead,
while also warning when stairs were coming, where handrails where and when the
user had gone off course. All seven users got to their destinations on the
first try, and much more quickly than if they had had to proceed normally with
no navigation. One subject, the paper notes, said 'That was fun! When can I get
one?' Microsoft actually looked into something like this years ago, but the
hardware just wasn’t there — HoloLens changes that. Even though it is clearly
intended for use by sighted people, its capabilities naturally fill the
requirements for a visual prosthesis like the one described here. Interestingly,
the researchers point out that this type of system was also predicted more than
30 years ago, long before they were even close to possible:
More information: