Data and Discourse: A Review of The Data Detective

Title: The Data Detective: Ten Easy Rules to Make Sense of Statistics
Author: Tim Harford
Publisher: Riverhead Books
Publishing Date: February 2, 2021
Pages: 336 (Hardcover)
ISBN: 978-0593084595

 

“There are lies, damn lies, and statistics.” Have you heard that before? Said it before? The quote is often wrongly attributed to Mark Twain and has been popularized in many places, including a little book called How To Lie With Statistics, written by Darryl Huff in the 1950s. 60 years later, Huff’s persuasive demonstration of the power of statistics, when used to back misinformation with seemingly solid data, remains popular. Readers tend to leave the book feeling quite cynical about statistics. Economist Tim Harford used to think that Huff had the right idea, but in the years since he began hosting the BBC program More or Less in 2007, he has become increasingly uncomfortable with Huff’s view. “I grew increasingly uneasy when fans of More or Less complimented the way we ‘debunked false statistics slowly…came to appreciate that the real joy was not in shooting down falsehoods but in trying to understand what was true” (11). His ambitious goal with his recent book, The Data Detective, is to create an alternative to Huff’s skepticism: to provide his readers with a set of common sense principles that will allow them to sift statistical truth from lies. 

In The Data Detective, Harford lays out ten rules that help separate facts from fiction. The first two principles focus on personal experiences, principles 3-7 consider errors that data can fall into, and principles 8-9 discuss the value of sources and the presentation of data. The tenth principle reminds readers that changing your mind on the basis of evidence is not only okay, it’s commendable. Each principle has its own chapter, and in each one, Harford weaves multiple stories together to illustrate both what he means and why the principle is important in validating the integrity of statistics. 

In the chapter discussing the first rule (“Search your feelings”), Harford warns readers to be aware of how claims (backed by statistics or not) make us feel, because “how you feel shapes what you think” (46) and thus makes us highly susceptible to misinformation. At this point, most of us are aware of the “click-bait” media/social media tactic of making exaggerated claims that trigger our emotional response to garner more views, likes, and shares. Throughout the first chapter, Harford illustrates the danger of letting our emotions have undue influence on our actions with a story about Abraham Bredius, a 20th century art critic and expert on Dutch painter Johannes Vermeer. In 1937, Bredius identified a newly found painting, Christ at Emmaus, as a lost work of Vermeer. At the time, Bredius was in his 80s and had spent his whole career searching for a link between Vermeer and the Italian artist Caravaggio. He believed that this painting, though “quite different” from all known Vermeers, provided the link.

“…instead of acting immediately, slow down and fact-check the claim, or read an opposing point of view first.”

Unfortunately, his passion made him a soft target for the emotional manipulation of a skilled forger. Christ at Emmaus had in fact been painted by Hans Van Meegeren, who confessed to the crime in 1945 when he was accused of selling another “Vermeer” to the influential Nazi Herman Göring. Bredius was deceived, Harford concludes, because he “desperately wanted to believe that this painting was the Vermeer he’d been looking for all his life.” In short, Bredius was fooled because his emotions motivated his reasoning. “It’s tempting,” the author warns, “to assume that motivated reasoning is just something that happens to other people. I have political principles; you’re politically biased; he’s a fringe conspiracy theorist” (38). Harford’s first rule encourages you to be aware of how claims make you feel, and if they engender strong emotions (positive or negative), to “slow down, control our emotions and our desire to signal partisan affiliation, and commit ourselves to calmly weighing the facts” (46). You can practice this the next time you find yourself on social media, wanting to either re-share or rebut something; instead of acting immediately, slow down and fact-check the claim, or read an opposing point of view first. 

Each of the remaining rules are similarly illustrated with a collection of fascinating stories—some of data-based successes, others cautionary, and still others simply curious. Through these narratives, Harford is asking his readers to: 

2. Ponder your personal experience.
   Is your experience representative? Do the big picture statistics match your experience? If not, what might account for that discrepancy?  

3. Avoid premature enumeration.  
   Do you actually know what’s being counted? Do you understand what the definitions of the groups are? Do the different people discussing agree on the definitions used in the counts? 

4. Step back and enjoy the view.  
   Are the comparisons made in context? Do you understand the scale of the numbers being reported? Is it a big number or a small number? 

5. Get the back story.
   Is there publication or survivorship bias at play in this claim? What other data might have been collected and swept behind the curtain into obscurity?  

6. Ask who is missing.  
   Most studies are done on western, educated, industrialized, rich democracies and have a tendency to study men more frequently than women. Who is missing from the data set? Is there sample bias present in some way? If these other groups were included, would you learn something different? 

7. Demand transparency when the computer says no. 
   When a computer algorithm (program) makes a prediction based on a large set of data, question why the algorithm works. What underlying assumptions are present in the model?  

8. Don’t take statistical bedrock for granted. 
   What is the source of (or who is producing) the data being used? Do these statisticians have a reputation for honesty, openness, and integrity? For data coming from national statistics offices, is the government of that country generally trusted to report accurately, or do they have a history of meddling with the official numbers? 

9. Remember that misinformation can be beautiful too. 
   What is under the surface of a beautiful (or not beautiful) graph? Does the graph accurately reflect the numbers? Does the graph convey any useful information? 

10. Keep an open mind.
    Have facts changed? Am I mistaken? Do I know enough? 

These ten principles can be summarized by a common theme: “be curious.” Do you know what you think you know? For example: on a scale of 1 to 10 (with 1 being no understanding and 10 being total understanding), how familiar are you with how a flush toilet functions? Now, pull out a piece of paper and write down an explanation in your own words (and/or draw a picture). Chances are, if you try this, you will struggle to identify how flushing a toilet really works. ¹ Studies have shown that people usually overestimate their familiarity with a subject, a phenomenon sometimes called “the illusion of explanatory depth” (272). The toilet is a bit silly as an example, but consider the exercise an invitation to ask yourself: Do I really understand the details of the policies and issues I am passionate about? As a simple test, try to explain the policy to someone else.  

“People may very well examine the same information and come to different conclusions than you.”

The Data Detective is an enriching read, and I highly recommend it to anyone who wants to work towards a world where we tone down the yelling on social media and have real conversations about actual data. There are two things about the book that readers may want to be aware of in advance. First, Harford re-edited the book in late spring of 2020 to add commentary about how the need for data on COVID-19 will shape the course of the emerging pandemic. Twenty months into the pandemic’s data debacle, these small interludes feel incomplete and already dated. Second, at places in the book where it is relevant to the stories being told, Harford acknowledges his personal views on data regarding climate change, LGBTQA+ issues, and vaccines. Readers for whom these are triggering topics may find these mentions distracting, however, they are cumulatively a small fraction of the book. I encourage you not to be drawn away from the book’s larger message: question the data—all the data. People may very well examine the same information and come to different conclusions than you. 

So, my reader, in conclusion, along with Tim Harford, I want to challenge you to be curious. Pick an issue where your feelings run hot, turn it into a question, then go be curious and look for data—all the data (not just the graphs retweeted by like-minded friends and influencers)—with an open mind. Yesterday, I had a conversation with a coworker who has different feelings than I do about the COVID vaccine; despite those differences, we both came away enriched in our understanding of “the other.” If you engage in this kind of activity, at the very least, you will better understand why some people passionately hold a different opinion and be able to dialogue with them more effectively. 

1. How a toilet works: the handle lifts a lever inside the tank, which is connected via a chain to a flap that lifts to release the water from the tank into the toilet bowl; the added water increases the pressure in the bowl sufficiently to push the waste over the s-bend (which prevents sewer gas from coming into the bathroom) and down into the sewer pipe…when my son was three he was obsessed with toilets and sewer systems, so I happen to have read a few books on the subject.  ↩