Using large-scale brain simulations for machine learning and AI
June 27, 2012
The Google research team has been working on some new approaches to large-scale machine learning, Google Official Blog reports.
Today’s machine learning technology takes significant work to adapt to new uses. For example, say we’re trying to build a system that can distinguish between pictures of cars and motorcycles.
In the standard machine learning approach, we first have to collect tens of thousands of pictures that have already been labeled as “car” or “motorcycle” — what we call labeled data — to train the system. But labeling takes a lot of work, and there’s comparatively little labeled data out there.
Fortunately, recent research on self-taught learning (PDF) and deep learning suggests we might be able to rely instead on unlabeled data—such as random images fetched off the web or out of YouTube videos. These algorithms work by building artificial neural networks, which loosely simulate neuronal (i.e., the brain’s) learning processes.
Neural networks are very computationally costly, so to date, most networks used in machine learning have used only 1 to 10 million connections. “But we suspected that by training much larger networks, we might achieve significantly better accuracy,” said the Google team.
“So we developed a distributed computing infrastructure for training large-scale neural networks. Then, we took an artificial neural network and spread the computation across 16,000 of our CPU cores (in our data centers), and trained models with more than 1 billion connections.”
“We then ran experiments that asked, informally: If we think of our neural network as simulating a very small-scale ‘newborn brain,’ and show it YouTube video for a week, what will it learn? Our hypothesis was that it would learn to recognize common objects in those videos.
“Indeed, to our amusement, one of our artificial neurons learned to respond strongly to pictures of… cats. Remember that this network had never been told what a cat was, nor was it given even a single image labeled as a cat. Instead, it “discovered” what a cat looked like by itself from only unlabeled YouTube stills. That’s what we mean by self-taught learning.
“Using this large-scale neural network, we also significantly improved the state of the art on a standard image classification test—in fact, we saw a 70 percent relative improvement in accuracy. We achieved that by taking advantage of the vast amounts of unlabeled data available on the web, and using it to augment a much more limited set of labeled data. This is something we’re really focused on—how to develop machine learning systems that scale well, so that we can take advantage of vast sets of unlabeled training data.”
“We’re actively working on scaling our systems to train even larger models. To give you a sense of what we mean by ‘larger’—while there’s no accepted way to compare artificial neural networks to biological brains, as a very rough comparison an adult human brain has around 100 trillion connections. So we still have lots of room to grow.
“And this isn’t just about images — we’re actively working with other groups within Google on applying this artificial neural network approach to other areas such as speech recognition and natural language modeling. Someday this could make the tools you use every day work better, faster and smarter.”
The research happened inside Google’s secretive X laboratory, known for inventing self-driving cars and augmented reality glasses, where a small group of researchers began working several years ago on a simulation of the human brain, the New York Times reports.
There Google scientists created one of the largest neural networks for machine learning by connecting 16,000 computer processors, which they turned loose on the Internet to learn on its own.
Google researchers are not alone in exploiting the techniques, which are referred to as “deep learning” models, the Times said. Last year Microsoft scientists presented research showing that the techniques could be applied equally well to build computer systems to understand human speech.
“The Stanford/Google paper pushes the envelope on the size and scale of neural networks by an order of magnitude over previous efforts,” said David A. Bader, executive director of high-performance computing at the Georgia Tech College of Computing. He said that rapid increases in computer technology would close the gap within a relatively short period of time: “The scale of modeling the full human visual cortex may be within reach before the end of the decade.”
- Quoc V. Le, Marc’Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, Andrew Y. Ng, Building High-level Features Using Large Scale Unsupervised Learning, to appear in: ICML 2012: 29th International Conference on Machine Learning, Edinburgh, Scotland, June, 2012 (open access)
- Quoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeff Dean, Andrew Y. Ng, Building high-level features using large scale unsupervised learning, arXiv:1112.6209