Can computers understand art?
September 27, 2012
Computer scientists Computer scientists Lior Shamir and Jane Tarakhovsky of Lawrence Technological University in Michigan have developed a program that analyzes paintings in a manner similar to how expert art historians perform their analysis, and conducted an experiment that showed that machines can outperform untrained humans in the analysis of fine art.
In the experiment, the researchers used approximately 1, 000 paintings of 34 well-known artists, and let the computer algorithm analyze the similarity between them based solely on the visual content of the paintings, and without any human guidance. Surprisingly, the computer provided a network of similarities between painters that is largely in agreement with the perception of art historians.

A computer-generated graph of similarities between 34 different painters, reflecting the similarities between the artistic styles of painters as were automatically deduced by the computer. The analysis shows that the computer was clearly able to identify the differences between classical realism and modern artistic styles, and automatically separated the painters into two groups, 18 classical painters and 16 modern painters. Inside these two broad groups the computer identified sub-groups of painters that were part of the same artistic movements. Overall, the computer automatically produced an analysis that is in large agreement with the influential links between painters and artistic movements as defined by art historians and critiques. (Credit: Lior Shamir)
The analysis showed that the computer was clearly able to identify the differences between classical realism and modern artistic styles, and automatically separated the painters into two groups, 18 classical painters and 16 modern painters.
Inside these two broad groups the computer identified sub-groups of painters that were part of the same artistic movements. For instance, the computer automatically placed the High Renaissance artists Raphael, Leonardo Da Vinci, and Michelangelo very close to each other. The Baroque painters Vermeer, Rubens and Rembrandt were also clustered together by the algorithm
Similarly, the computer algorithm deduced that Gauguin and Cézanne, both considered post-impressionists, have similar artistic styles, and also identified similarities between the styles of Salvador Dali, Max Ernst, and Giorgio de Chirico, all are considered by art historians to be part of the surrealism school of art.
Overall, the computer automatically produced an analysis that is in large agreement with the influential links between painters and artistic movements as defined by art historians and critiques.
While the average non-expert can normally make the broad differentiation between modern art and classical realism, they have difficulty telling the difference between closely related schools of art such as Early and High Renaissance or Mannerism and Romanticism. The
The experiment was performed by computing from each painting 4,027 numerical image context descriptors — numbers that reflect the content of the image such as texture, color and shapes in a quantitative fashion. This allows the computer to reflect very many aspects of the visual content, and use pattern recognition and statistical methods to detect complex patterns of similarities and dissimilarities between the artistic styles and then quantify these similarities.

Comments (19)
by BPFNE
Understanding is not what art is about. Art is about appreciating the months of work in one paining. The feeling of the artist at the time. The emotion felt and perceived will never be understood by an automated machine.
by asiwel
Pardon me for going on here, but lately the discussion of “algorithms” has been becoming metaphysical. If we have a clever route-finding computer algorithm capable of “machine learning” even .. and we properly connected it up to a rat’s brain, so that it assisted the rat solve mazes to get food, I would call that combination a “super-intelligent” rat. But I would not call the algorithm itself intelligent … instead I would call it a very intelligently programmed algorithm. Google’s automated cars, Watson’s medical diagnoses, even Siri’s anwsers areexamples rapidly evolving into something I would begin to call pretty smart devices. I would want to give my car a name.
by none
how about kitt
by asiwel
“Kitt??”… uh … Probably not. In retrospect, I would want, instead, to politely ask my car to please tell me its name, before we drove off together …
by Gorden Russell
The article didn’t mention if the Campbell soup cans of Andy Warhol were show to the program.
by Bri
My father used to say that he didn’t like andy’s art, but that it was a meaningful step in arts evolution. He did think that Andy was a very good graphics artist( think of the Maralyn photo and it’s strong use of color. It’s a bold eye catching graphic!).
by Christian Gehman
Andy’s soup cans are to art as his beans in the bung hole are to literature.
by PAUL
good point
by Liabna
Well,.., Art is not just painting ya kno deer komputr
by Bri
These are broad technical differences. They are a skeleton of changes in techniques. Branching off from this are the more subtle changes in artistic thought that have evolved over time. If we look at a art historian before they get an education, they would be hard pressed to make these categorical assumptions. So the program is already more intuitive than even someone who is drawn to art. All the other analyses were formed as the painters themselves explored art as a means of expression. Art is a language of communication. It’s not about the craft of applying paint. Artists use techniques to relate a thought to peoples unconscious minds. Micheal Angelos statue of the Jesus lying dead on the virgin Mary’s lap is a classic example. Jesus is life sized, but Mary would stand about 14 feet tall. The statue is about her love and suffering, and you are drawn to her image more than Jesus’s because of the unconscious effects of the scale difference. These attributes can be added into the AI program. It’s just another layer of analysis.
by Marcos Marin
Obnoxiously citing names and buzzwords is indeed something “untrained humans” cannot do very well. :-D
by Editor
You’re right. That requires art critics who are trained to cbnoxiously cite names and buzzwords. :)
by Bri
I thought that was their job. They do it all so well.
by Frank Krasicki
The title and conclusion of this article are profoundly disingenuous. The ability to categorize visual material is quite different from the ability to “understand” the material.
While the cross-referencing of information about these artists and their works is an essential component in understanding the material, it is quite another thing to presume that this data is any more meaningful to a computer than the dental records of artists. The retrieval, no matter how accurate, of mere data is not evidence of understanding the quality of that data, the weight of the ideas involved, or the latent trajectory of those ideas to create new Art.
The good news is that programs like these will contribute to an augmented art experience for visitors to galleries and museums.
by Editor
The algorithm exhibits far greater apparent understanding of art than mine (admittedly not a high threshold), but does not extend (yet) to the depth of an art expert, although the researchers don’t claim that. So the question is: what does “understanding” mean? I admit that’s above my understanding. :)
by asiwel
As Bri said, “It’s just another layer of analysis” and, indeed, in this case, a very interesting and potentially useful and enlightening one. We can produce such “similarity” networks using good computer algorithms and most any data – to wit, evolutionary biology and genomic patterns, citation studies in professional literature, etc.. Whether they make sense is whether the underlying patterns they reveal are reliable and valid – stable and interpretable. But “understanding”, “appreciating”, “learning from”, and changing one’s attitudes and behavior in light of exposure to art – being an entity that does that, whether machine or human, is on a somewhat different order. Even if the data the researchers provide are considered “perceptions” and the results of the analysis when applied to answering questions (posed by others) are the “behaviors”, and the whole thing is the “entity,” I am still not sure I would call that “intelligence.” If the algorithm starts acting independently and begins to tweet (i.e., talk about and publish) patterns it finds itself in data of interest to it, that would be a different story entirely.
by Bri
@Frank and Editor (Amara) this is my point. It is beginning to understand. It formulated the relationships on it’s own. A art historian starts as someone who is drawn to art. They have no basis to evaluate what they see. As they explore art, they learn it’s relationships, whether intuitively or through conversations with other people. When I go to museums with people who have no understanding of art, I try and see it through their eyes. The only handle they have is in the pictorial aspects and subjective feelings. By pictorial I mean it’s photographic qualities. The subjective is in relation to what is depicted and unconscious emotional reactions, which are mostly driven by use of color and what the subjects actions depict, such as a war scene, a land scape, a soup can,etc. Those things are part of our ability to identify and react to the world. Like how they have to train a robot that a rubber ball is round and bounces. The google caris learning this. Nicko is learning this. This program is now learning another layer of understanding and it is doing ot far faster than a normal human can. Of the program also had access to the google cars programs of interrelationships, it would be one step closer to how we perceive the world, and also be a step ahead because people don’t intuitively break down paintings into the groupings this program does. To go through a quick art history lesson. We started making representations of meaningful objects in the cave painting days. These depictions grew in complexity. In Egyptian art, they didn’t know how to use perspective, hence the ackward poses. By the time you get to the dark ages you still have very flat subjects, but perspective is being explored. Each of these steps is an understanding that is learned and passed down as a tradition. These traditions are used to express concepts, emotions. They are like words that gain meaning by the relationships of usage in a particular culture. All of that are deeper subtle levels of analysis, that can be added to this programs impressive understanding of different styles of expression. Our understandings are much more nuanced, but this program is already superior in it’s ability to find relationships. It’s just that there ate many levels of relationships that humans evaluate thing by. Nicko was learning another level of relationships.
by asiwel
I think I would agree with almost all of Bri’s comment above, except perhaps for the second sentence “..it is beginning to understand” … If we are just talking about this example of an AI-type algorithm, my argument is that there still is no “it” there yet to do any “understanding.” The algorithm is essentially an analytical capability. There is no question that these sorts of tools are multiplying and improving. In fact they are really what is now called “narrow AI.” Compared to past programs, they are “brilliant” but compared to the idea of “intelligence” they are necesssary but still way insufficent conditions. There still seems to be very little “I” in narrow AI.
by asiwel
Let me be a little clearer. The result in the article, the similarity graph, is something I would expect to be produced by. e.g., SAS multivariate analysis procedures and graphics programs. I honestly don’t think of factor analysis or path analysis or discriminant functions or pattern recognition, etc., as “intelligent behavior.” I simply regard and respect these procedures as mathematical tools.for analyzing data.