IBM simulates 530 billion neurons, 100 trillion synapses on supercomputer
November 19, 2012
A network of neurosynaptic cores derived from long-distance wiring in the monkey brain: Neuro-synaptic cores are locally clustered into brain-inspired regions, and each core is represented as an individual point along the ring. Arcs are drawn from a source core to a destination core with an edge color defined by the color assigned to the source core. (Credit: IBM)
IBM Research – Almaden presented at Supercomputing 2012 last week the next milestone toward fulfilling the ultimate vision of the DARPA’s cognitive computing program, called Systems of Neuromorphic Adaptive Plastic Scalable Electronics (SyNAPSE), according to Dr. Dharmendra S. Modha, Manager, Cognitive Computing, IBM Research – Almaden.
Announced in 2008, DARPA’s SyNAPSE program calls for developing electronic neuromorphic (brain-simulation) machine technology that scales to biological levels, using a cognitive computing architecture with 1010 neurons (10 billion) and 1014 synapses (100 trillion, based on estimates of the number of synapses in the human brain) to develop electronic neuromorphic machine technology that scales to biological levels.”
Simulating 10 billion neurons and 100 trillion synapses on most powerful supercomputer
IBM says it has now accomplished this milestone with its new “TrueNorth” system running on the world’s second-fastest operating supercomputer, the Lawrence Livermore National Lab (LBNL) Blue Gene/Q Sequoia, using 96 racks (1,572,864 processor cores, 1.5 PB memory, 98,304 MPI processes, and 6,291,456 threads).
IBM and LBNL achieved an unprecedented scale of 2.084 billion neurosynaptic cores* containing 53×1010 (530 billion) neurons and 1.37×1014 (100 trillion) synapses running only 1542 times slower than real time.
“We have not built a biologically realistic simulation of the complete human brain,” explains an abstract of the Supercomputing 2012 (SC12) paper (open-access PDF), selected from the 100 SC12 papers as one of the six finalists for the Best Paper Award. “Computation (‘neurons’), memory (‘synapses’), and communication (‘axons,’ ‘dendrites’) are mathematically abstracted away from biological detail toward engineering goals of maximizing function (utility, applications) and minimizing cost (power, area, delay) and design complexity of hardware implementation.”
Neurosynaptic core (credit: IBM)
Two billion neurosynaptic cores
“Previously, we have demonstrated a neurosynaptic core* and some of its applications,” continues the abstract. “We have also compiled the largest long-distance wiring diagram of the monkey brain. Now, imagine a network with over 2 billion of these neurosynaptic cores that are divided into 77 brain-inspired regions with probabilistic intra-region (“gray matter”) connectivity and monkey-brain-inspired inter-region (“white matter”) connectivity.
“This fulfills a core vision of the DARPA SyNAPSE project to bring together nanotechnology, neuroscience, and supercomputing to lay the foundation of a novel cognitive computing architecture that complements today’s von Neumann machines.”
To support TrueNorth, IBM has developed Compass, a multi-threaded, massively parallel functional simulator and a parallel compiler that maps a network of long-distance pathways in the macaque monkey brain to TrueNorth.
* The IBM-Cornell neurosynaptic core is a key building block of a modular neuromorphic architecture, according to Modha. The core incorporates central elements from neuroscience, including 256 leaky integrate-and-fire neurons, 1024 axons, and 256×1024 synapses using an SRAM crossbar memory. It fits in a 4.2mm square area, using a 45nm SOI process.
PAST IBM PRESS RELEASES:
DARPA SyNAPSE Phase 0
DARPA SyNAPSE Phase 1
DARPA SyNAPSE Phase 2
Comments (71)
by Leonard Lehman
Don’t worry folks, this idea will burn out like a long ago memory. This will take 200+ years to become a reality, by that time we humans will have destroyed are planet with no hope of leaving in time to find another world.
Sorry for this statement.
by I heart grammar
*our
by Leonard Lehman
Is it true that what ever logic can be made in hardware can be done in software? Its not a question of speed as much as how wide the bus is. What the brain does well is discounts or priortizes information and saving it, depending on the age of the person. Short term memory and long term memoy is too simple of a term. It is safe to say that facts are rememberd on the base of emotion, which is how the brain grades the importance of a rememberance. this is done in small degrees. This is evolution.
by Prasad
Great step forward.
by M
…and the worlds first AI asks for a banana…
by netesq
Editors: Please note that “billon” is misspelled in the title. Should be “billion.”
by Editor
Thanks. Ten years ago, I couldn’t even spell “billon” — I still can’t!
by derek
each step toward technology being able to match human thinking power is one step closer to making humans the second dominant species. do not enable computers to learn by themselves. keep them only capable of what theyre programmed to do. these guys are destroying mankind and will see it in our lifetime.
by Adam
Disgusting. You cower in the face of greatness because it exceeds your own limitations.
by seeker
Interesting annagram below :)
ABCDEFGHIJKLMN
HI
AB
LM
HAL -> IBM
by Mallén
Congratulations!
by Anon User
4:35 ‘…cognitive computing is only possible within IBM…’
BS :)
‘…where else could you bring together so many disparate technologies and people…’
within any enthusiastic community with sufficient technology quotient?
aaanyway :) best news all week! txs!
by Tony Stender
Does anyone know how the IBM synapse machine design compares to Jeff Hawkins design for cortex columns he is using in his latest work?
As for emotion, I have given it some thought. It seems to be designed to inform the brain of two things. Either move toward or away from, and the amount of energy to use if this event occurs again. More involved emotions seem to include the thinking and language elements ans additional elements.
Who knows if these are close to the real thing?
by James
Presumably if they had an exaflop machine they’d be able to run it at only 16 times slower than real time, and that’s probably only 5-6 years away.
by widsss
Why don’t they build it out of GPUs?
by Editor
This project milestone involved a software emulation of special-purpose neurosynaptic chips, not a hardware project, and the Sequoia doesn’t use GPUs.
by A4i
IBM should build that neurosynaptic core on 20nm process and pack at least 250 cores on a die. Also MCM should be used to incorporate 4 dies (1000 cores) on a module. Also at least 4 modules should be fitted on a rack mount blade and 24 blades in a cabinet – that’s 4 000 neurosynaptic cores on a blade and 96 000 in a cabinet. So 2 billion neurosynaptic cores should fit in 21 000 cabinets. Consider that Titan supercomputer, currently number one in top500 has only 200 cabinets,
by MrFriendly
I’m glad to see that DARPA hasn’t completely given up on this project. When Todd Hylton resigned from DARPA, it looked like it was a complete failure. Fortunately, though, it seems they’ve reduced their scope from blue sky ambitions (“intelligent” robots), to more realistic goals, such as
real-time modeling and processing of large amounts of data streams for business and military applications.
by Editor
“This project” refers to SyNAPSE? I’m not aware of any reduction in scope from the original BAA:
https://www.fbo.gov/index?s=opportunity&mode=form&id=b7b66ad9c0d5a7df21d9488b107256ae&tab=core&_cview=1&cck=1&au=&ck=
by MrFriendly
Darpa has dropped quite a few labs/collaborators since last year, including Stanford, Boston University, and HP.
Apparently, the goal of developing autonomous robots for the battlefield have been supplanted with more modest goals of data analysis applications. Boston University and HP were trying to get memristor circuits to control virtual agents that could see, hear, and react/make decisions in ways similar to rodents. This research is ongoing, but it’s no longer part of the SyNAPSE project. Max Versace, head of the neuromorphic labs at Boston University, wasn’t sure if the SyNAPSE project was still ongoing, when I asked him about it.
Also, the brainchild of the SyNAPSE project, Tod Hylton, left in February 2012 to join Eugene Izhikeivch’s Brain Corporation.
This is why I’m surprised and delighted to hear that they’re still working on neuromorphic processors with IBM. Perhaps they’ll resurrect their plans for autonomous machines, but Modha has stated a number of times that their goal isn’t to develop robots. If you want citations, I’ll dig some up.
by Editor
Thanks for the info. Much appreciated.
by Matthew J Price
Just a small mistake: LLNL’s Sequoia is no longer the world’s fastest supercomputer. Titan now has that distinction, at least for a little while. http://goo.gl/Xx1XN
by Editor
Updated per new Top 500 list (http://www.top500.org), thanks
by eldras
No Synapse yet. Not mapped.
The pivotal brain component, and weighting’s not the half of it.
But it;s a good progressive effort.
by star0
Very interesting…
I would like to see it do something really neat, though — something like the Google cat-recognizinig neural network, only better.
by GAUSS
You can get a neural network to recognize a wide variety of images with different scales, orientations and such on just 1,000 simulated neurons. I really don’y see why Google needed to use thousands of servers and zillions of neurons to do what any hobbyist can do in a few hours of coding. :\
by Justin
Oh wow, this is crazy because I drive past this lab every day lol
by GAUSS
Great work, but unfortunate that it’s all military/DoD.
by James Bleep
A lot of useful civilian technology comes out of DoD research, like err, the internet.
by Gorden Russell
Well, they won’t be able to put those “96 racks (1,572,864 processor cores…) into the skull of a terminator robot anytime soon.
But!
Using satellite communications, a Berserker in Afghanistan could be controlled from Lawrence Livermore.
by Marcos Marin
The lengths to which people will go to make supposedly cool sounding acronyms…
by Casey
1542 times slower than the real thing? Does this mean that, going by Moore’s Law, we should create a human brain simulation around 2023? (Computers doubling each year would mean a 1024x increase 10 years from now, and a 2048x increase in 11)
by Gorden Russell
I can’t follow your math, Casey. When I figure it out step-by-step, doubling every 1.5 years, I get seven doublings in 10.5 years, resulting in 128 times the processing power. Continuing on that way, I get ten doublings 15 years for a total of 1024 times the number of transistors. Then in 16.5 years you get to 2048x.
by Casey
Indeed, but I thought computer power was supposed to double every year, not every year and a half.
by DJ
The doubling rate for cpus is 11 months
by MarkL
2013=2, 4, 8, 16, 32, 64, 128, 256, 512, 2022=1024.
Count ‘em. 10 years.
by Editor
No, as noted below, simulating human nervous system speed is not the system design objective, which is to maximize complexity to the level of a cat brain, and minimize power requirements and size; Moore’s law does not relate to speed, but to transistor density, which is expected to hit a ~6 nm. limit around 2025; and the target architecture is non-von Neumann, so the metrics are different.
by obilesk
I love this answer. You have won the internet-etiquette award for the day. Thank you.
by Hologram
Exactly. Thank you :)
by tim the realist
Maybe this team should collaborate with the team trying to simulate a honey bee brain. The honey bee brain is certainly 1500 times less complex than this simulation so it should be able to run in real time on this hardware.
by Jabbah
The problem currently with simulating actual brains is the lack of a method to accurately determine the synapse levels. Without these all we have is a connection map without the connection potentials. Whilst we have the hardware power available for simulation, it is best used modelling general architectures rather than direct brain mappings with guessed synapse levels or those derived using machine learning algorithms.
by Editor
What does “synapse levels” mean?
by Jabbah
The synaptic weights – how strong the connection is between two neurons.
by Editor
The BAA does not mention weighting but does mention synaptic conductance, which is apparently related?
Dynamic range of synaptic conductance > 10
• Synaptic conductance increase >1%/pulse for presynaptic spike applied somewhere within 80-1 msec before a postsynaptic spike
• Synaptic conductance decrease >1%/pulse for presynaptic spike applied somewhere within 1-80 msec after postsynaptic spike.
• 0%-0.02% conductance decrease if presynaptic spike applied > 100 msec before or after postsynaptic spike
https://www.fbo.gov/index?s=opportunity&mode=form&id=b7b66ad9c0d5a7df21d9488b107256ae&tab=core&_cview=1&cck=1&au=&ck=
by John
Jabbah probably means synapse conductivity. The tune parameter. It’s like the kids who found a piano without strings, then found strings, put them somehow “in the right places” (= connectome), but still piano doesn’t play as it supposed to. The tune factor was missing.
by Jabbah
Yes, that’s a very good way of putting it too!
To put this in perspective, the connectome for the C. elegans worm has been known for a number of years and is well documented, yet numerous projects to create a virtual representation of the worm have failed due to the lack of information such as synapse weights, ion channel details and whether nurons are excitatory or inhibitory. The Virtual C. elegans project in Japan was close and used machine learning algorithms to determine the synapse weights, but it didn’t react in a realistic manner. There are a few on going projects that are looking to incorporate actual synaptic and ion channel data into the connectome models such as OpenWorm.
Now this is just a worm, with 302 neurons and around 7,000 synapses. We are quite a way off modelling actual brains of more complex creatures. Projects such as OpenWorm will help though as it will allow researchers to modify and run the worm simulation in real time. It’s also close to being run on a single workstation requiring ~5TFlops. 4 top of the range gpu cards can almost provide that level now.
by Jabbah
Just to add: The reason it helps researchers is that it provides a way to quickly validate methods of extracting this kind of information. Once reliable methods are found, applying them to more complex brains will provide a much greater chance of accurately modelling it.
by Ralph Dratman
At least when you work with a worm you know what the behavior is supposed to look like. When the simulated human brain starts up, what would it do? Breathe and eat and cry, mainly, for the first several thousand real-world years (the first few simulated years). Then what?
by John
Newborn brain crying for thousand years… oh man, what are these people doing ;’( …
by snake0
Pretty sure that’s not how it works.
by Editor
Great material for Colbert, though :)
by snake0
Very interesting, so in addition to the Connectome, a “Conductome” is also required.
by melajara
I concur!
by Gorden Russell
“Maybe this team should collaborate with the team trying to simulate a honey bee brain. ”
Maybe they already are, tim the realist, but they are keeping it top secret.
They want it to be a big surprise when a robot bee flies through the window and stings an al-Qaida bombmaker with the venom of a Coral Snake.
by Editor
The objective here is not to create a machine that behaves like a human, but to reverse-engineer the human brain’s functionality to create more useful machines. So operating in real time is not the primary objective — intelligence is. The honey bee’s limited functionality would not achieve DARPA’s objective. This is opposite of Watson, where the objective is to respond in real time. This statement from one of the referenced IBM press releases may help:
“The goal of SyNAPSE is to create a system that not only analyzes complex information from multiple sensory modalities at once, but also dynamically rewires itself as it interacts with its environment – all while rivaling the brain’s compact size and low power usage. The IBM team has already successfully completed Phases 0 and 1. This is a major initiative to move beyond the von Neumann paradigm that has been ruling computer architecture for more than half a century,” said Dharmendra Modha, project leader for IBM Research. “Future applications of computing will increasingly demand functionality that is not efficiently delivered by the traditional architecture. These chips are another significant step in the evolution of computers from calculators to learning systems, signaling the beginning of a new generation of computers and their applications in business, science and government.”
More info on the DARPA requirements and specs: Broad Agency Announcement, Systems of Neuromorphic Adaptive Plastic Scalable Electronics
Defense Sciences Office, DARPA-BAA 08-28,
https://www.fbo.gov/index?s=opportunity&mode=form&id=b7b66ad9c0d5a7df21d9488b107256ae&tab=core&_cview=1&cck=1&au=&ck=
by Ralph Dratman
Has it occurred to anyone to ask how an artificial, fully non-von-Neumann brain could be persuaded to cooperate? Are you going to stimulate its pleasure centers each time it correctly answers a question? If so, the most likely outcome will be madness (for the unfortunate brain) and severe emotional trauma (for the experimenters). Try sitting up all day and night trying to persuade a simulated child to eat its simulated peas.
Or maybe this construct will just be something like a wired-up cat: shine virtual light into its virtual eyes and record the resulting virtual neural activity. But if that is the objective, think: an actual, off-the-shelf cat operates at 100% of real time, can be pre-trained for many helpful perceptions and behaviors, and consumes no electric power. The cat would obviously be a much better (not to mention much cheaper) platform for study.
by Sam
The way you teach a child to behave is with love. Limitless amounts of love and patience, kindness, guidance and boundary setting. Smart machines will have to be managed the same way. We’ll raise them, present them to the world well-adjusted and then try to ensure they’re not abused into becoming dangerous.
by Editor
That assumes addition of reptile brain functionality. Are there any such plans?
by Ralph Dratman
Exactly. And to incorporate a reptile brain into a human brain means first building a reptile brain whose reptile can live on its own. And to build a reptile brain means building whatever simpler brain preceded the reptile, which means… well, back to the worm lab.
More fundamentally, I think “neural Darwinism” implies that a brain cannot be built, but must be grown, in connection with the embryonic growth of the actual organism which the brain will control.
Our epigenetic heritage (in the broadest sense), without which human life could not continue, is inseparable from our world, our universe.
by Editor
Perhaps one day, millennia hence, our machine-merged progency will say the same: “To incorporate a cerebrum into a metahuman brain means first building a human brain….”?
by Editor
digital catnip? :)
by Oneironaut
Let me laugh at IBM’s ‘neurons’.
by Liventruth
Now if only we new how to fine tune our computers…hmmm. I wouldn’t be surprised if you were able to create an independently intelligent entity with this. If so, a few questions. Have you planned past the “oh shit this thing is alive” factor and run the tests necessary to make sure that the new conscious entity you are creating is stable or sane? If so, have you considered that by all appearances a bunch of scientists are making a very powerful conscious being. Will entity be able to judge? If so, by what means will it judge? All of us are delusional to an extent as humans. If this is true I am just throwing out a couple of suggestions and comments. Realtime risk-vs-truth/reality analysis may be needed. I would appreciate it as a citizen and human if y’all did due diligence as if this were the most important space flight was about to happen.
Because you are placing our entire planet online with iT.
Capeesh?
by Gianluca
Define “sane” and “stable” related to a true general purpose A.I. . We don’t know. I guess we first have to find out how it actually behaves.
by Ben
The question is: Will it suffer?
If we bring an “innocent” being with perfect innocence in our universe, we could theoretically create a whole new kind of suffering. If the being can think a million times faster than humans, its suffering could be greater than all human suffering combined so far.
by Dan
It will not suffer. Suffering is a biological beings way of informing itself that something is to be avoided in order to maintain its ultimate value which is life. A simulated being would need to be simulated in a suffering state (for tests) for it to actually suffer. Otherwise it would feel no such thing.
by koko
I hope so.
by Aaron
Wow! Big step forward. Can’t wait ’til they refine this process and improve upon current designs. Wonder where it’ll be in five years…
by Liventruth
Quick question: maybe I am stupid. I’ve been to college numerous times. Maybe it is strange to me. I’ve been researching life for 30 years on November 28th.
Analysis : it is amazing that to this day I still don’t understand time as we use it. Does volume ever change according to gravity?
I am a Newton. My lineage is Newtonian. Issac newton wasn’t to keen of a fellow. Lol.
by Gorden Russell
” Does volume ever change according to gravity? ”
Of course, Liventruth. Your volume is decreased markedly if you go past the event horizon of a black hole.
by Jabbah
Around 48 times slower than real time ;-)