Automated method beats critics and Academy Awards in picking great movies

Data analysis trumps critics, wisdom of crowds, and number of movie awards
January 19, 2015

Subgraph of film connections network. Films are ordered chronologically, based on year of release, from bottom to top (not to scale). A connection between two films exists if a sequence, sentence, character, or other part of the referenced film has been adopted, used, or imitated in the referencing film. For example, there is a connection from 1987’s Raising Arizona (22) to 1981’s The Evil Dead (23) (second column from left) because the main characters of both films drive an Oldsmobile Delta 88. Values represent the time lag of the connection, measured in years. (credit: Max Wasserman et al./PNAS)

According to a new Northwestern University study, the best predictor of a movie’s significance is how often a movie is referenced by other movies.

In other words, a movie’s significance is decided by today’s and tomorrow’s film directors — not the critics.

“Movie critics can be overconfident in spotting important works, and they have bias,” said Luís Amaral, the leader of the study and co-director the Northwestern Institute on Complex Systems. “Our method is as objective as it gets.”

Amaral also is a professor of chemical and biological engineering in Northwestern’s McCormick School of Engineering and Applied Science.

He and his colleagues are the first to systematically compare different approaches for estimating a film’s significance. They evaluated metrics for measures that are both subjective (critical reviews, awards, public opinion) and objective (citations, box office sales).

The researchers found their automated method of movie citations is better at predicting greatness, especially in movies 25 years old or older, than the expertise of movie critics (a group of critics or a single critic), the wisdom of the crowd, the number of awards won, and the amount of box office sales, among other measures.

The study will be published the week of Jan. 19 in Proceedings of the National Academy of Sciences (PNAS).

The research team conducted a big data study of 15,425 U.S.-produced films listed in the Internet Movie Database (IMDb). They looked to see how well an approach predicted a movie’s inclusion in the National Film Registry of the U.S. Library of Congress.

Citation analysis

In their analysis, the researchers found the number of times a movie 25 years or older is referenced by other movies best predicts inclusion in this registry of American films deemed “culturally, historically or aesthetically significant,” such as “The Wizard of Oz,” “Star Wars,” “Psycho,” “Casablanca” and “Gone With the Wind.”

“Directors keep coming back to movies that are significant,” Amaral said. “If you show a little bit from ‘Pscyho,’ such as referencing the shower scene, you are putting that whole movie in front of the viewer of the new movie.”

“There is something about a movie that is hidden to us, but there are measurable things, such as critic ratings, awards and referencing by other filmmakers, that hint at this hidden element — a movie’s significance,” he said. “We find that ultimately it is the creators, the filmmakers themselves, who will determine which movies are important, not the expert critics.”

Also important, the researchers write, is that the automated method can easily be applied to older films for which no other rating may be available.

But what Amaral really wants to do is develop a method for identifying the most significant scientific papers. Given the varied sizes of scientific fields, the sheer number of citations is not sufficient for determining greatness, he said.

“More than 1 million scientific papers are published each year worldwide,” Amaral said. “It can be difficult to distinguish a good scientific paper from an average one, much like the movies. My next goal is to develop a good measure of scientific citations to get inside what is going on in the scientific literature.”

The U.S. Army Research Office supported the research.


Abstract for Cross-evaluation of metrics to estimate the significance of creative works

In a world overflowing with creative works, it is useful to be able to filter out the unimportant works so that the significant ones can be identified and thereby absorbed. An automated method could provide an objective approach for evaluating the significance of works on a universal scale. However, there have been few attempts at creating such a measure, and there are few “ground truths” for validating the effectiveness of potential metrics for significance. For movies, the US Library of Congress’s National Film Registry (NFR) contains American films that are “culturally, historically, or aesthetically significant” as chosen through a careful evaluation and deliberation process. By analyzing a network of citations between 15,425 United States-produced films procured from the Internet Movie Database (IMDb), we obtain several automated metrics for significance. The best of these metrics is able to indicate a film’s presence in the NFR at least as well or better than metrics based on aggregated expert opinions or large population surveys. Importantly, automated metrics can easily be applied to older films for which no other rating may be available. Our results may have implications for the evaluation of other creative works such as scientific research