Poison attacks against machine learning
July 23, 2012
Poisoning attack strategy (credit: Battista Biggio, Blaine Nelson, Pavel Laskov)
New results indicate that it may be easier than we thought to provide data to a learning program that causes it to learn the wrong things by by feeding it wrong data — a “poison attack,” I Programmer reports.
Three researchers, Battista Biggio (Italy) Blaine Nelson and Pavel Laskov (Germany), have found a way to feed a Support Vector Machine (SVM) with data specially designed to increase the error rate of the machine as much as possible with a few bogus data points.
SVMs are fairly simple learning devices that use examples to make classifications or decisions. They are used in security settings to detect abnormal behavior such as fraud, credit card use anomalies and even to weed out spam. SVMs learn by being shown examples of the sorts of things they are supposed to detect, including examples while doing its job.
The approach assumes that the attacker knows the learning algorithm being employed and has access to the same data to the original training data (which can be simulated).
What they discovered is that their method was capable of having a surprisingly large impact on the performance of the SVMs tested. They also point out that it could be possible to direct the induced errors so as to product particular types of error. For example, a spammer could send some poisoned data so as to evade detection in the future.
Comments (9)
by Phillfrog
I think humans don’t just learn a pattern in order to recognise it, but also have memory of how they have/would have reacted to the input in the past. So perhaps what is needed here is for the learning program to keep various historical snapshots of its pattern matching algorithm and to run the input through many of these snapshots. This may highlight poisoning attempts or at least make poisoning a more drawn out process (perhaps over years) like brainwashing a human takes time (and incidentally entails distorting the person’s memory of past beliefs)
by thane stroop
Considering the nature of learning, and how many of the 7billion learners have/havent had their learning corrupted, I think it would be useful to understand what allows some peoples minds to resist corrupting, like skeptisism or stubborness. A faulty teacher vs faulty sensory perception.
by Gorden Russell
When prison inmates figure out the learning algorithm they will mess over the robot prison guards.
by melajara
Easy, one inmate has just to pretend to be dead. Then the robots will call doctors and disappear as no corpse needs to be guarded.
Then you, the inmate, can wake up and go wherever you want unhampered by any prison custodian as every robotic guard in the prison, acting as as swarm intelligence, now knows that you are dead and a dead person is not walking.
Ergo, this walking person (you) cannot be the deceased but is either from staff or a visitor.
That’s the impeccable logic of machine thinking ;-)
by PirateRo
Really? So the designers of the machine will not have thought this out? There will not be independent verification from a variety of different tools? Instead, life will be like a Hollywood movie where the guards are tricked into the cell to give bad writing an out? Worse, you don’t think the device so placed will not figure the thing out for itself?
Even today, machines are allowed to learn until they can do the job and then their learning is stopped. What is there to mess with? Also, it is a sad commentary to think there may still be a need for prisons like we have no in the future. They may serve an immediate, practical purpose but they do no service to those individuals inside by remediating talents, attitudes or education.
by melajara
Of course, I was deliberately exaggerating.
However my point is that machines to have a truly complete model of human beings should be trained too in the art of deception as practiced by some humans.
Humans are very manipulative and (sometimes) very perverse beings.
A machine should have a model of “evil” too, would it be only for not to be tricked too easily.
An interesting point is the connection of “evil” with twisted logic. That’s what I wanted to convey through my feeble joke.
As for the necessity of prisons, some people would say that they prefer to have prisons than a society with reconditionned human beings devoid of free will.
The problem of evil is a very old one and chances are that it will still be a very prominent one in the future notwithstanding any technical advances.
by Rick
But then the biometrics will take over and recognize you are alive we will also have medical sensors attaches to combat this.
by GatorALLin
…insert skynet joke here…..
by Marcos Marin
Ramona will try to classify this as Social Engineering, as opposed to mere hacking, today @ 3pm ET.