How to identify and predict human activities from video
October 30, 2012
The Mind’s Eye program will automate video analysis — recognizing current behavior, interpolating actions that occur off-camera, and predicting future behavior (credit: Carnegie Mellon University)
A video shows a woman carrying a box into a building. Later, it shows her leaving the building without it. What was she doing?
Carnegie Mellon University’s (CMU) Mind’s Eye program is creating intelligent software that will recognize human activities in video and predict what might happen next. It will also flag unusual events and deduce actions that may be occurring off-camera.
Automating the time-consuming job of viewing and interpreting video images will speed intelligence-gathering, improve monitoring, and provide new tools for research. Autonomous systems could employ Mind’s Eye technologies in applications ranging from defense to medical and consumer robotics.
Recognizing and predicting human activity in video footage is a difficult problem. People do not all perform the same action in the same way. Different actions may look very similar on video. And videos of the same action can vary wildly in appearance due to lighting, perspective, background, the individuals involved, and more.
To minimize the effects of these variations, Carnegie Mellon’s Mind’s Eye software will generate 3D models of the human activities and match these models to the person’s motion in the video. It will compare the video motion to actions it’s already been trained to recognize (such as walk, jump, and stand) and identify patterns of actions (such as pick up and carry). The software examines these patterns to infer what the person in the video is doing. It also makes predictions about what is likely to happen next and can guess at activities that might be obscured or occur off-camera.
Carnegie Mellon is one of twelve research teams and three commercial integrators participating in this five-year program, which is sponsored by DARPA’s Information Innovation Office. The project kicked off in September, 2010 and is currently in the early stages of software development.
Comments (16)
by johnny flash
this is f-in boring
by steve
This is creepy.
by Horst
Blessed be the geek(ette)s and the nerd(ette)s who bring this eternal blessing to humanity.
Before human beings become fully rational, this is an intermediate stopgap measure to prevent ‘those who know not what they do’ from harming others. it’s in everyone’s interest – that includes you.
by Gabriel
Tell me about it – makes me think of Minority Report.
I don’t like being cynical, but I want to also be reasonable – how many ways could something like this possibly be used?
This has abuse written all over it. At least we’re not there ‘yet’.
by MrFriendly
We won’t be there for a good while. Just check the pop-sci archives for failed AI project after failed AI project. The problem with these announcements is that virtually nothing has been done or proven. It’s just a plan, like Darpa’s recently discontinued/failed SyNAPSE program.
Progress has been pretty good, lately, but developing computer vision at this level is like taking baby steps to the moon, and occasionally having the moon pushed back another few lengths.
Nobody needs to panic :P At least, not until the ’20′s.
by Editor
“failed AI project after failed AI project”
Citations?
“recently discontinued/failed SyNAPSE program”
Citation? I’m unaware of any cancellation. SyNAPSE has been successful, and DARPA awarded approximately US$21 million in new funding for phase 2 of the SyNAPSE project last year, according to IBM: http://www.ibm.com/smarterplanet/us/en/business_analytics/article/cognitive_computing.html. Latest significant achievement (June 2012):
http://www.kurzweilai.net/building-block-of-a-programmable-neuromorphic-substrate-a-digital-neurosynaptic-core
“developing computer vision at this level is like taking baby steps…”
Not sure what that metric means
by SurveilAInce
Gotta keep tabs on crazy people.
The only solution would be to have no crazy people, but that’s harder to solve and takes more time. Keeping tabs on them comes first.
by MrFriendly
lol.
by MrFriendly
I’ll believe it when I see some actual results/progress. Computer vision is still an extremely tough nut to crack for any kind of practical application.
I remember when Google was claiming they’d have their Goggles app ready to accept photos of chess games, which it would then analyze to determine board pieces so that recommended moves could be computed. They quietly killed that project because the computer vision aspect of it is far too difficult.
by aus
Total surveillance is near.
by Trinity Alps
Creeeeepy!
A scenario using Bush-era logic, the woman should have been shot upon leaving the building as a pre-emptive safety measure ala the Iraq War strategy.
by Mike
MSNBC would be a better place for this comment. This is just technology stuff here.
by MrFriendly
Agreed. I’m sick of politics.
by Terry
You’ve got it! The assumptions are absurd. Maybe the woman went into the building to say hello, used the john, and left the package there by mistake. Lots of tech projects evince a complete lack of imagination, sort of like, in logic, a faulty logical division–lots of things that could be causal aren’t thought of. As with all the babies killed when expectant moms were given thermaldahide and lots of babies were born with various horrible defects. Oops, didn’t think that might happen.
by geekette
The U. Paris is also working on this problem and is using EOP – Emotion Oriented Programming, with common sense reasoning to help narrow the possible interpretations.
by Connections
“#ALERT—DROPPED—ITEM: Person observed carrying object into building and leaving without it.” a quote from a book released this year.
http://thedaemon.com/killdecisionsynopsis.html