Teaching robots to do household chores

Future goal is to allow robots to learn by simply watching a YouTube video
May 30, 2018

MIT’s Sims-inspired “VirtualHome” system aims to teach artificial agents a range of chores, such as setting the table and making coffee. (credit: MIT CSAIL)

Computer scientists at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and the University of Toronto* have created a Sims-inspired “VirtualHome” system that can simulate detailed household tasks.

The idea is to allow “artificial agents” to execute tasks — opening up the possibility of one day teaching robots to do such tasks.

Using crowdsourcing, the researchers created videos that simulate detailed household activities and sub-tasks in eight different scenes, including a living room, kitchen, dining room, bedroom, and home office.  A simple model can generate a program from either a video or a textual description, allowing robots to be programmed by naive users, either via natural language or video demonstration.

“Hey, Jeeves — get me a glass of milk” would require several subtasks — the first five are shown here. (credit: MIT CSAIL)

The researchers have trained the system using nearly 3,000 programs for various activities, which are further broken down into subtasks for the computer to understand. A simple task like “making coffee,” for example, would also include the necessary step, “grabbing a cup.”

Next: Anticipating personalized wants and needs

The end result is a large database of household tasks described using natural language. Companies like Amazon that are working to develop Alexa-like robotic systems at home could in the future use such data to train their models to do more complex tasks.

Robots could eventually be trained to anticipate personalized wants and needs, which could be especially helpful as assistive technology for the elderly, or for those with limited mobility.

The team hopes to train the robots using actual videos instead of Sims-style simulation videos. That would enable a robot to learn directly by simply watching a YouTube video. The team is also working on a reward-learning system in which the robot gets positive feedback when it does tasks correctly.

The project** will be presented at the Computer Vision and Pattern Recognition (CVPR) conference in Salt Lake City, June 18–22, 2018 .

Reference: CVPR paper (open-access). Source: MIT CSAIL.

* Researchers from McGill University and the University of Ljubljana were also involved.

** This project was partially supported by a “La Caixa” fellowship, Canada’s National Sciences and Engineering Research Council Strategic Partnership Network on Machine Learning Hardware Acceleration (NSERC COHESA), Samsung, the Defense Advanced Research Projects Agency (DARPA) and the Intelligence Advanced Research Projects Activity (IARPA).