NASA-Microsoft augmented-reality system allows scientists to ‘work on Mars’

January 22, 2015

A screen shot from OnSight, a software tool developed by NASA’s Jet Propulsion Laboratory in collaboration with Microsoft. OnSight uses real rover data to create a 3D simulation of the Martian environment where mission scientists can “meet” to discuss rover operations. (Credit: NASA/JPL-Caltech)

NASA Jet Propulsion Laboratory and Microsoft have developed software called OnSight that will enable scientists to work virtually on Mars with Mars Curiosity rover, using wearable technology called Microsoft HoloLens.

“OnSight gives our rover scientists the ability to walk around and explore Mars right from their offices,” said Dave Lavery, program executive for the Mars Science Laboratory mission at NASA Headquarters. “It fundamentally changes our perception of Mars, and how we understand the Mars environment surrounding the rover.”

OnSight will use real rover data and extend the Curiosity mission’s existing planning tools by creating a 3D simulation of the Martian environment where scientists around the world can meet. Program scientists will be able to examine the rover’s work site from a first-person perspective, plan new activities, and preview the results of their work firsthand.

Rover operations previously required scientists to examine 2D Mars imagery on a computer screen, and make inferences about what they are seeing.

NASA Microsoft | Microsoft HoloLens & NASA OnSight

The OnSight system uses what Microsoft calls “holographic computing” to overlay visual information and rover data on the user’s field of view. Holographic computing blends a view of the physical world with computer-generated imagery to create a hybrid of real and virtual.

To view this holographic realm, members of the Curiosity mission team don a Microsoft HoloLens device, which surrounds them with images from the rover’s Martian field site. They then can stroll around the rocky surface or crouch down to examine rocky outcrops from different angles.

“Previously, our Mars explorers have been stuck on one side of a computer screen. This tool gives them the ability to explore the rover’s surroundings much as an Earth geologist would do field work here on our planet,” said Norris.

The OnSight tool also will be useful for planning rover operations. For example, scientists can program activities for many of the rover’s science instruments by looking at a target and using gestures to select menu commands.

JPL plans to begin testing OnSight in Curiosity mission operations later this year. Future applications may include Mars 2020 rover mission operations and other applications in support of NASA’s journey to Mars.