How to train a robot to do complex abstract thinking

February 16, 2018

Robot inspects cooler, ponders next step (credit: Intelligent Robot Lab / Brown University)

Robots are great at following programmed steps. But asking a robot to “move the green bottle from the cooler to the cupboard” would require it to have abstract representations of these things and actions, plus knowledge of its surroundings.

(“Hmm, which of those millions of pixels is a ‘cooler,’ whatever than means? How do I get inside it and also the ‘cupboard’? …”)

To help robots answer these kinds of questions and plan complex multi-step tasks, robots can construct two kinds of abstract representations of the world around them, say Brown University and MIT researchers:

  • “Procedural abstractions”: bundling all the low-level movements composed into higher-level skills (such as opening a door). Most of those robots doing fancy athletic tricks are explicitly programmed with such procedural abstractions, say the researchers.
  • “Perceptual abstractions”: making sense out of the millions of confusing pixels in the real world.

Building truly intelligent robots

According to George Konidaris, Ph.D., an assistant professor of computer science at Brown and the lead author of the new study, there’s been less progress in perceptual abstraction — the focus of the new research.

To explore this, the researchers trained a robot they called “Anathema” (aka “Ana”). They started by teaching Ana “procedural abstractions” in a room containing a cupboard, a cooler, a switch that controls a light inside the cupboard, and a bottle that could be left in either the cooler or the cupboard. They gave Ana a set of high-level motor skills for manipulating the objects in the room, such as opening and closing both the cooler and the cupboard, flipping the switch, and picking up a bottle.

Ana was also able to learn a very abstract description of the visual environment that contained only what was necessary for her to be able to perform a particular skill. Once armed with these learned abstract procedures and perceptions, the researchers gave Ana a challenge: “Take the bottle from the cooler and put it in the cupboard.”


Ana’s dynamic concept of a “cooler,” based on configurations of pixels in open and closed positions. (credit: Intelligent Robot Lab / Brown University)

Accepting the challenge, Ana navigated to the cooler. She had learned the configuration of pixels in her visual field associated with the cooler lid being closed (the only way to open it). She had also learned how to open it: stand in front of it and don’t do anything (because she needed both hands to open the lid).

She opened the cooler and sighted the bottle. But she didn’t pick it up. Not yet.

She realized that if she had the bottle in her gripper, she wouldn’t be able to open the cupboard — that requires both hands. Instead, she went directly to the cupboard.

There, she saw that the light switch was in the “on” position, and instantly realized that opening the cupboard would block the switch. So she turned the switch off before opening the cupboard. Finally, she returned to the cooler, retrieved the bottle, and placed it in the cupboard.

She had developed the entire plan in about four milliseconds.


“She learned these abstractions on her own”

Once a robot has high-level motor skills, it can automatically construct a compatible high-level symbolic representation of the world by making sense of its pixelated surroundings, according to Konidaris. “We didn’t provide Ana with any of the abstract representations she needed to plan for the task,” he said. “She learned those abstractions on her own, and once she had them, planning was easy.”

Her entire knowledge and skill set was represented in a text file just 126 lines long.

Konidaris says the research provides an important theoretical building block for applying artificial intelligence to robotics. “We believe that allowing our robots to plan and learn in the abstract rather than the concrete will be fundamental to building truly intelligent robots,” he said. “Many problems are often quite simple, if you think about them in the right way.”

Source: Journal of Artificial Intelligence Research (open-access). Funded by DARPA and MIT’s Intelligence Initiative.


IRL Lab | Learning Symbolic Representations for High-Level Robot Planning