You’ll interact with smartphones and smartwatches by writing/gesturing on any surface, using sonar signals

March 17, 2016

FingerIO lets you interact with mobile devices by writing or gesturing on any nearby surface, turning a smartphone or smartwatch into an active sonar device (credit: Dennis Wise, University of Washington)

A new sonar technology called FingerIO will make it easier to interact with screens on smartwatches and smartphones by simply writing or gesturing on any nearby surface. It’s is an active sonar system using the device’s own microphones and speakers to track fine-grained finger movements (to within 8mm).

Because sound waves travel through fabric and do not require line of sight, users can even interact with these devices (including writing text) inside a front pocket or a smartwatch hidden under a sweater sleeve.


University of Washington Computer Science & Engineering | FingerIO

Developed by University of Washington computer scientists and electrical engineers, FingerIO uses the device’s own speaker to emit an inaudible ultrasonic wave. That signal bounces off the finger, and those “echoes” are recorded by the device’s microphones and used to calculate the finger’s location in space.

Using sound waves to track finger motion offers several advantages over cameras — which don’t work without line-of-sight or when the device is hidden by fabric or another obstructions — and other technologies like radar that require both custom sensor hardware and greater computing power, said senior author and UW assistant professor of computer science and engineering Shyam Gollakota.

But standard sonar echoes are weak and typically not accurate enough to track finger motion at high resolution. Errors of a few centimeters would make it impossible to differentiate between writing individual letters or subtle hand gestures.

So the UW researchers used “Orthogonal Frequency Division Multiplexing” (used in cellular telecommunications and WiFi), allowing for tracking phase changes in the echoes and correcting for any errors in the finger location.

Applications of fingerIO. a) Transform any surface into a writing interface; b) provide a new interface for smartwatch form factor devices; c) enable gesture interaction with a phone in a pocket; d) work even when the watch is occluded. (credit: R. Nandakumar et al.)

Two microphones are needed to track finger motion in two dimensions, and three for three dimensions. So this system may work (when available commercially*) with some smartphones (it was tested with a Samsung Galaxy S4), but today’s smartwatches typically only have one microphone.

Next steps for the research team include demonstrating how FingerIO can be used to track multiple fingers moving at the same time, and extending its tracking abilities into three dimensions by adding additional microphones to the devices.

The research was funded by the National Science Foundation and Google and will be described in a paper to be presented in May at the Association for Computing Machinery’s CHI 2016 conference in San Jose, California.

* Hint: Microsoft Research principal researcher Desney Tan is a co-author.


editor’s comments: This tech will be great for students and journalists taking notes and for controlling music and videos. It could also help prevent robberies. How would you use it?


Abstract of FingerIO: Using Active Sonar for Fine-Grained Finger Tracking

We present fingerIO, a novel fine-grained finger tracking solution for around-device interaction. FingerIO does not require instrumenting the finger with sensors and works even in the presence of occlusions between the finger and the device. We achieve this by transforming the device into an active sonar system that transmits inaudible sound signals and tracks the echoes of the finger at its microphones. To achieve subcentimeter level tracking accuracies, we present an innovative approach that use a modulation technique commonly used in wireless communication called Orthogonal Frequency Division Multiplexing (OFDM). Our evaluation shows that fingerIO can achieve 2-D finger tracking with an average accuracy of 8 mm using the in-built microphones and speaker of a Samsung Galaxy S4. It also tracks subtle finger motion around the device, even when the phone is inside a pocket. Finally, we prototype a smart watch form-factor fingerIO device and show that it can extend the interaction space to a 0.5×0.25 m2 region on either side of the device and work even when it is fully occluded from the finger.