Ugly Truth: Shady Metrics are Shady

Posted on

“85% of statistic are made up on the spot.”

“Numbers lie.”

“Tell me what you want and I’ll find the numbers to prove it.”

We have a problem with metrics. We love them, but they’re not always helpful. As soon as you get a client to understand why their gut feeling or their ancedotal evidence is not necessarily the best foundation for their strategy, you have to start battling metrics.

Don’t get me wrong. I love statistics. I love metrics. For all their faults, they’re so much more trustworthy than the fore-mentioned gut feelings or anecdotal evidence. I’ve learned not to trust my eye. I can scan a Facebook page and get a feel for how successful their strategy is, but when I actually take a look at the statistics, I find that it’s never quite as bad or as good as I expect it to be. I take it for the (humbling) learning opportunity it is, but it’s tempting to present data so that they say what you want them to say.

For example, take this recent article from vulture.com. The point they want to make is that there’s a huge age-gap between male and female leads in a movie with a romantic plot/subplot. To visualize the issue, they’ve put up a bunch of graphs comparing the ages of some famous leading men and their female costars over the last decade. The results are impressive. There is a massive disparity between Tom Cruise and his costars, Johnny Depp hops and skips, but even he isn’t immune. As a counterpoint, they even charted Tom Hanks. The bulk of his costars land within 10 years, apparently that’s because he’s a good guy who’s been married to the same woman for 25 years.

But when you look at the graphs a bit more closely, some problems show up. Suddenly all I can see are flaws in the logic. Flaws that make me wonder what is really going on. How carefully did they cherry-pick these actors and movies? What would the graphs look like if they were presented consistently.

Now, to be fair, my “gut” tells me that everything this article is saying is true on some level. I’m pretty sure if I went and looked at the same subset of data, the same over all trend would emerge, but not this …spectacularly. And that’s where my frustration lies. Nobody wins when the data is presented in misleading ways.

What exactly stands out to me? A lot.

  • The scale from graph to graph is inconsistent. ie: Cruise goes from 1983-2013, Clooney goes from 1996-2010. Clooney’s costars are almost all within 10 years. His trends are actually closer to Tom Hanks, but the scale makes him look like a “young” Richard Gere. The rapid-fire presentation of these graphs are begging us to compare one star to another, but the graphs make that hard to do accurately.
  • The year scale within each graph is inconsistent. While the actor’s age line technically communicates the gap in years from movie to movie, that’s a weak visual clue. It looks like a graph over time, even though it’s not. We expect the x-axis to be consistent and it’s not. Would any of these graphs look as impressive if the male lead’s age looked like a straight line increase (similar to Steve Carrel)?
  • Vulture says the movies are a “representative sample”, and that’s probably true. But does that mean it’s the best choice to answer this question? The list include films where the age difference is a plot point, where the story spans decades and the characters are both artificially aged/youthened, and action movies where the “lead female” is a draw in her own right and is barely (if at all) romantically involved with the male lead. Are these interesting things to look at? Absolutely. Does it get in the way of looking at this question? Hells yes.
  • Similarly, isn’t there a different casting issue for movies where the couple is established (and a tertiary plot point) and where one of the primary plot points is wooing like a romantic comedy. See also: happy endings vs “maybe this was a terrible idea” endings, female leads who are “midlife crisis” second marriages, etc.
  • The article says the ages of the female costars don’t change, but that’s not really true. Many of them have a clear trend upward (especially when you remove the may/december plots). Clooney’s female leads are consistently 10 years younger than him. They’re certainly not closing the gap, but they are increasing at or about the same rate.
  • Richard Gere, really? If you’re going to talk about leading men who are a box office draw, choose men who are a box office draw. I totally get he’s got history but… no.
  • For that matter, where’s Leo? Hugh Jackman? Robert Downey Jr? Will Smith? Or go the other way: Nicholson, Eastwood, DeNiro. (Bruce Willis? Morgan Freeman?) I’m not saying charting these men would be better or worse than they ones Vulture chose, but they may be more representative of “leading man above the age of 45″, you know?
  • One last shot at the “representative” selection of films: Many of these movies (especially some of the recent ones with the widest age gaps) were not “wins” at the box office or critically acclaimed. What if this means the formula isn’t working as well as people might think it is. I don’t know because it isn’t taken into account.
  • Clearly, I have strong feelings about this, in terms of both how the graphs are misleading and also what we trends we could really learn from looking at this data. Let’s compare to “leading men” who were past their prime in 80s, the 50′s. Let’s cross reference by genre, critical acclaim, box office draw. Let’s identifying how this trend has changed, and under which circumstances (other than Tom Hanks and George Clooney) it’s better or worse.

    Bottom line: when you present data with too many independent variables that haven’t been accounted for and misleading scales, it can get in the way of the core message. Suddenly, instead of focusing on the trends you’ve identified, your audience is left challenging your methods and by extension, the point you’re trying to make. Shady metrics might get your website some traffic, and it might help you keep/win business. But ultimately, they don’t do you or your audience any good and they don’t propel the conversations forward in useful ways.


Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>