First map of how the brain organizes everything we see
December 20, 2012
These sections of a semantic-space map show how some of the different categories of living and non-living objects that we see are related to one another in the brain’s “semantic space” (credit: Shinji Nishimoto, An T. Vu, Jack Gallant/Neuron)
How do we make sense of the thousands of images that flood our retinas each day? Scientists at the University of California, Berkeley, have found that the brain is wired to organize all the categories of objects and actions that we see, and they have created the first interactive map of how the brain organizes these groupings.
Continuous semantic space
The result — achieved through computational models of brain imaging data collected while the subjects watched hours of movie clips — is what researchers call “a continuous semantic space.”
Some relationships between categories make sense (humans and animals share the same “semantic neighborhood”) while others (hallways and buckets) are less obvious. The researchers found that different people share a similar semantic layout.
“Our methods open a door that will quickly lead to a more complete and detailed understanding of how the brain is organized. Already, our online brain viewer appears to provide the most detailed look ever at the visual function and organization of a single human brain,” said Alexander Huth, a doctoral student in neuroscience at UC Berkeley and lead author of the study published Wednesday, Dec. 19 in the journal Neuron.
Medical diagnosis/treatment, brain-machine interfaces
Semantic-space map construction process (Credit: Shinji Nishimoto, An T. Vu, Jack Gallant/Neuron)
A clearer understanding of how the brain organizes visual input can help with the medical diagnosis and treatment of brain disorders. These findings may also be used to create brain-machine interfaces, particularly for facial and other image recognition systems. Among other things, they could improve a grocery store self-checkout system’s ability to recognize different kinds of merchandise.
”Our discovery suggests that brain scans could soon be used to label an image that someone is seeing, and may also help teach computers how to better recognize images,” said Huth, who has produced a video and interactive website to explain the science of what the researchers found.
It has long been thought that each category of object or action humans see — people, animals, vehicles, household appliances and movements — is represented in a separate region of the visual cortex. In this latest study, UC Berkeley researchers found that these categories are actually represented in highly organized, overlapping maps that cover as much as 20 percent of the brain, including the somatosensory and frontal cortices.
To conduct the experiment, the brain activity of five researchers was recorded via functional magnetic resonance imaging (fMRI) as they each watched two hours of movie clips. The brain scans simultaneously measured blood flow in thousands of locations across the brain.
Researchers then used regularized linear regression analysis, which finds correlations in data, to build a model showing how each of the roughly 30,000 locations in the cortex responded to each of the 1,700 categories of objects and actions seen in the movie clips. Next, they used principal components analysis, a statistical method that can summarize large data sets, to find the “semantic space” that was common to all the study subjects.
The results are presented in multicolored, multidimensional maps showing the more than 1,700 visual categories and their relationships to one another. Categories that activate the same brain areas have similar colors. For example, humans are green, animals are yellow, vehicles are pink and violet and buildings are blue.
“Using the semantic space as a visualization tool, we immediately saw that categories are represented in these incredibly intricate maps that cover much more of the brain than we expected,” Huth said.
Other co-authors of the study are UC Berkeley neuroscientists Shinji Nishimoto, An T. Vu and Jack Gallant.
An MRI viewer on the Gallant Lab website (gallantlab.org/semanticmovies) shows how information about the thousands of object and action categories is represented across human neocortex (credit: Shinji Nishimoto, An T. Vu, Jack Gallant/Neuron)
Comments (32)
by Michaela
Another question would be if thinking about things would trigger the same location or if this is only in connection with sight?
by Michaela
How come sex is not mentioned? Is it just too all consuming that it can fit under ‘change’, ‘rooms’, devices’ , ‘touch’ and everything else?
I must be really weird, since that was the first thing in my mind and not the chess playing ;-)
by Chris Reynolds
Must be a guy’s brain – I don’t see the ‘shoes’ map.
by William Sheridan
How does the “natural epistemology” from this research compare to the formal epistemology underpinning Knowledge Engineering? Does the natural epistemology have a logic of its own, of is it simply an evolutionary product of convenient justaposition? Can we produce an amalgam of natural and formal epistemologies that gives us the best of both? Let’s try!
by Editor
Yes, very interesting idea. And what happens when neuromorphic models combine with “semantomorphic,” to coin a word?
by Ian Clarke
That semantic-space map picture can’t be right. Look at the distance between ‘people’ and ‘reptiles’… and no overlap!
by Bri
In all fairness Ian , there are some piney green ones in there, just like in real life. A general gradient from green to yellow.
by jwmapp
Could the article I just read on Kuzwiel AI help organize and understand the brain map?
“A new ‘network-extracted ontology’ model of the cell”
Turning vast amounts of genomic data into meaningful information about the cell is the great challenge of bioinformatics, with major implications for human biology and medicine
Researchers at the University of California, San Diego School of Medicine and colleagues have proposed a new “network-extracted ontology” (NeXO) method that creates a computational model of the cell from large networks of gene and protein interactions, discovering how genes and proteins connect to form higher-level cellular machinery.
by Justin
It would be fun to apply this data to designing a home based on one of the dimensions.
Or find a nice mix of a couple of them.
by JBerry
I wonder if something similar to this would work to map out the areas of the brain involved in making ethical decisions? The subject could watch short videos that involve moral choices, choose an action, and the researchers could follow the changes in blood flow as it happens.
by Gorden Russell
Did you notice that they are looking at voxels to get into semantic space? Can you see what this is leading to? Using a petaflop computer to look at an fMRI scan in real time will allow individual words to be read out of the semantic space. Homeland Security will look into the head of a captured underwear bomber and find out who his contacts are and where he was trained. They’ll give up water boarding altogether.
by GAUSS
Doesn’t sound so bad. :)
by franco
Each brain works different in different tasks
by melajara
Impressive. This study will be a landmark for the definition of new ontologies.
Now a synthetic map collected over 5 researchers is not enough of a sampling to achieve definitive conclusions. There is perhaps more variability across a larger sample set of subjects.
For example, it is a little known fact that strong chess grandmasters are achieving their remarkable chess positions memorization ability by subverting the brain area normally used for face recognition (cf. Zsuzsa Polgar’s brain scans when she was memorizing/recollecting chess positions).
More generally, it would be interesting to compare the submaps of laymen vs the ones developed by specialists of the domain under scrutiny.
by Whittaker
So, by learning from how the chess masters memorize chess position, we can do slight further improvement upon our computers that can already beat human in chess (Of course, I know this is not the only application of learning about chess master’s brain functions). I think it would be great for human beings to learn the chess-playing computers’ strategies and tactics, albeit to actually play as good as a strong computer, we need cybernetic implants in our brains to give us the computational brute force that would not be possible for human beings otherwise. Let’s hope, this would not be considered cheating in the future, in the same way that no one considered Deep Blue cheating.
by Gorden Russell
In the future, Whittaker, such implants will be as common as I-phones.
by GAUSS
They most certainly will. I’ve done a lot of musing on things like this “semantic space”; in my earlier research I formulated this as a vector space, but I can see here it’s much better suited as a large asymmetric graph. The graph will have no problem dealing with nonlinearities. Fascinating.
by Mr.X
@Whittaker:
Unless you insert a joy button, all kinds of competition would become boring as hell, with your implants.
If you insert that button, why not push it without reason!?
Did you ever try to play a first-person shooter with others, while everyone used aimbots!?
Ps: Afaik the chess computers don’t use strategies/tactics in the sense you seem to think of.
Instead, they have models/algorithms to evaluate positions and moves.The computers go mechanically through all possible moves (to a predetermined extent, else they’d be doing this for what would feel like forever) on other (virtual) boards, essentially simulating the game while playing with their opponent.At last,they choose the “best” move they’ve found.
Of course, if algorithms executed by some kind of special/general (depending on the implant) purpose computer count as strategies learned, then why do you even bother to play a game!?Your implant would play alone.
by GAUSS
“Your implant would play alone.”
Surely you’d be able to find some will to do something about this?
by Mr.X
@GAUSS:
Why should I!? If the implant executes optimized algorithms, any inference will only make my “play” get worse.It follows, that the dominant strategy, if you care about winning, will be “let the implant do it’s work”.
A question that -imho- makes more sense is: Why use an implant for “non-critical” tasks?
by GAUSS
Right – and if you care about playing, for enjoyment, you’ll probably worry less about winning all the time since it would get boring. Agreed.
by Mr.X
You didn’t get my point.
If you just play for fun, why use enhancements at all!?You see, it would be unessecary and add nothing.You underestimate the “mechanical” nature of most games.
And, if you play normally against enhanced, you will lose all the time.I don’t see all too many people playing chess against software, all the time, for pure enjoment ;)
And again: Why not just push a enjoy button on your implant instead!?
So, disagreed.
by melajara
@Whittaker. As the modus operandi of the strongest chess programs (e.g. Houdini 3) is glorified brute force i.e. with a more precise eval function for leaf nodes and clever tree pruning techniques there is not much to be adapted from such algorithmic refinements for the sake of “human” chess playing.
Btw, This is the fundamental reason chess ceased to be the Drosophila of AI, even if nowadays mundane smartphones are “outwitting” Carlsen or Anand.
No, the real interest in investigating the neurological basis of chess grandmaster play level, IMHO is for better emulation of this process for the rest of us.
Rather than fiddling with preposterous and, as for now, imaginary brain implants, I’m finding more interesting to try to devise new sets of mental exercises (e.g. computer supported variants on blind chess) to converge to the kind of successful neural plasticity grandmasters are achieving.
Actually I did some work on this. Alas, I found no time to complete it yet with a true CBT platform but I’m closely monitoring progresses on related areas like multimodal or gesture UI.
by melajara
better emulation of this THOUGHT process.
@Editor, no way to have the edit function back, even for 1 minute?
by Gabriel
I wouldn’t be surprised Whittaker, if, rather then being called cheating, their will be two different groups to keep things fair — no-implants and implants-only.
Imagine how exciting it must be to watch two people with implants playing – two ordinary people suddenly playing better then the best unenhanced…in contrast, no-implant groups would be equally exciting for the very reason they are unenhanced and have no “cheats”.
So long as the playing field is fair, it’s all good. Imagine Olympics designed only for the enhanced….it would seem like no-implant records get broken all the time.
by Mr.X
@Gabriel:
“Imagine how exciting it must be to watch two people with implants playing ”
Do you like watching calculators?Well, each to their own, I guess^^
About the olympics: I think it could be called gullible if you think that most of the records there are achieved without drugs.
” two ordinary people suddenly playing better then the best unenhanced”
How is that suddenly!?Why don’t you take your laptop and a standard chess program and go out there.If you’d use the program to play a real life opponent, then you’re essentially already “doing” what seems to be so amazing to you.The difference is, it is obvious that you’re NOT the one doing it, because your computer is external to you.
If you think you’re the one playing better than the unenhanced persons, then you’ll probably think you sometimes actively make rain drop from your “skin” (while wearing a rain coat).
It is just another way to feel good about oneself through self-delusion and without any exertion.
Reminds me of fat people falling for all these diet pills^^
by Mr.X
Anyway: I think your predictions are not that unlikely.
In the end, if general purpose implants come into existence, people will have to use them in order to stay competive (if such a thing still matters).
This will allow their use to permeate other parts of life.
Have a nice day;)
by Gabriel
People with implants are going to be alot more interesting to watch then calculators….questions to ask like; how far do their implants take them? are their different levels of augmentation? what sort of creativity will they employ with their new brain-power? etc…
More so then the fact they are augmented….is the question and curiosity of what will they DO with said-enhancement…that’s where the excitement comes from – and when you get two people with similar enhancements against eachother, the game is still fair and exciting, especially when they are undoubtedly doing things that, once again, the very best unenhanced could never dream of thinking/doing.
Of course, if that doesn’t interest you, their would probably still be games purely for the “natural” which is exciting for that very reason, and probably will still surprise and excite those who have enhanced themselves – as long as the opponents are on fair-grounds, their’s no reason not to enjoy a close and unpredictable game.
Their’s also the very real plausibility of, at least with brain-extenders, of turning them ‘off’ and, at least, emulating how you used to be, so you could play with the “Mosh’s” again. Everything is speculation, but when performance-enhancers start showing up, their will obviously have to be alot of shuffling around and new regulations for those who opt for them, and for those who don’t…..but again, when it comes to games and competition, their is no reason not to enjoy them nor say competition no longer exists — if you are adamant on staying ‘natural’ or watching only those sorts of games, then so be it…..their will probably be an entire genre of games and such directed for that, the same for the augmented.
by Mr.X
First paragraph: Ask them which implants they use and look them up on Amazon.All questions will be answered.There are not many kinds of creativity, it consists mostly of association/pattern matching.
If you’d enough “calculating-power”, most games like chess would be a big math problem.
” the very best unenhanced could never dream of thinking/doing.”
Chess is mechanical, the computers use BRUTE-FORCE.The persons themselves could not interfere without ruining their performance, and if they do that, why use enhancements at all!?It is no achievement of the enhanced.
If you want to see the best performance, which you could not appreciate anyway if you’re not able to do it yourself, watch two supercomputers play.Where is the need for the monkey in that?
Once again, these games are only in so far interesting to us as our ability to calculate is limited.Do you know this game where you put 0 and + in consequtive turns into squares, till you get a winning combo or every square’s filled!? After the age of 8, or sth like that, when you figured out the dominant strategy, these kinds of games “become” boring to us.
Chess is no different, if you have more “brain-brawn” at your disposal.
” their is no reason not to enjoy them nor say competition no longer exists ”
Chess has certainly a “solution”.If you can calculate enough, there won’t be any competition.Just execution of algorithm.Of course, one could argue that being worse doesn’t make our own algoritms more interesting.
“New category of games”:
That’s what would be needed.But if you believe in the singularity, it could be that you would have to change them “exponentially”!?
And maybe there is a physical limit to intelligence or complexity
of the games.
Of course, this whole discussion assumes that people will essentialy stay the same, will essentially be the same old great apes as before, just with some toys.
People have these grand notions about future change, and then assume everything else will stay unaffected.This is, honestly, not smart.
E.g robots will replace labour, but our economy won’t change^^
Ps: Stop the strawman please. Where did I mention any “natural” esoteric “crap”.If you think technology and it’s users are not natural, termites aren’t natural either. I’m neither for nor against these technologies.I will go with the flow^^
Have a nice day:)
by Gabriel
….Mr.X, calm down — there was no strawman whatever in my post; I did not intend to put words in your mouth, so please don’t do the same.
by Whittaker
I am happy to see that my little comment had spawned such well written, intellectually stimulating discussions. Much appreciation from me to everyone who replied.
by gregorylent
time to drop looking at the meat, and to start looking for, and into, the field, where all of this first happens.