Please follow by email! Instead of an annoying popup to ask you this, I put it here. Pretty please?

Tuesday, November 17, 2015

Zombie Statistics, Part 1

By Steven S. Vrooman

In the new edition of The Zombie Guide to Public Speaking, I added a chapter on the ways we fail to do our research. I decided to share a part of that chapter, from a section I call "Zombie Statistics," because the world needs to know the problematic history of this Curve of Attention. It is the prime example of something fake we believe because it is in graphical form. Public speaking textbooks, slideshares, blog posts, tweets . . . it is EVERYWHERE! It took me a few months of interlibrary loaning to gather enough resources to debunk it, but here it is:

The Curve of Attention

This thing has been trotted out for years. We believe it because it makes sense. I even suggested it might have some merit in Chapter 6. The audience pays the most attention to us at the beginning, gets sleepy and then wakes up at the end. Here’s Sam’s version of it:

 Each version has a slightly different shape to The Curve. The reason is that there is no actual data to back this up.
 I first saw this while writing the teacher’s manual (Vrooman & Egan, 2009) for Fraleigh & Tuman (2009), where the authors suggest that it is based on a “Study by TCC Consulting (San Francisco, Calif.), undertaken between 1987 and 1997” (p. N-4). Emails to the authors about this data never got a response. I found The Curve again in Kenny (1982), who suggests it is a result of studies conducted on students. The y axis of his graph is labelled as “Percentage of class paying attention” (p. 13), but he then describes some uncited experiments about how much students can recall after lectures of different lengths, which is a different sort of y axis entirely.[1] He cites Mills (1977) as the source of The Curve and the research behind it, but when I finally tracked down a copy of Mills’ book, his Curve, exactly the same as Kenny’s, is based not on data, but on “an analogy between the learning process and the process of digestion” (p. 18).
A digestion analogy. Really?
What he suggests about this analogy makes intuitive sense, which is why people keep reproducing it, but what we have is something that has become “fact” for us only because it got turned into a graph. And Mills is reasonably clear that this is just a thought-experiment and analogy: “the shape of the curve in the hands of a good instructor can be modified almost at will” (p. 19).
I later found McGrath’s (2015) citation that The Curve is based on Bligh’s (2000) book, first published in 1971. Bligh bases almost all this on Lloyd’s (1968) work, creating a series of diagrams based on what Lloyd “hypothesizes” and “confirms” (Bligh, 2000, p. 49). The trouble is that Lloyd’s two-page article is based on his own observations of student restlessness in his classes, hardly sufficient rigor to “confirm” a hypothesis and graph the data.[2]
Yet, by now The Curve is used as if it were a classic study, you know, with data behind it. Red Magma (2009) tosses their version into a Slideshare which sells their elearning consulting. They use it as the central data point for why lectures are an inefficient method of learning. Sharpe (2012) suggests The Curve is an “amalgam of all sorts of things from physiological responses to recall about part of the session.” She cites additional “numerous studies” on our 10-15 minute attention spans in “passive tasks” like lecture. This is a commonly reported statistic that, according to Wilson and Korn (2007) has no real supporting evidence besides personal experience and anecdotes.
The fun of all of this is that writers keep changing the shape of The Curve for their own purposes while still asserting that it is based on data. Fraleigh and Tuman’s  (2009) Curve jumps up a bit higher at the conclusion than the 50%-ish in Mills’ (1977). Niemantsverdreit (n.d.) has a curve that goes to 100% attention really quickly at the introduction and then never seems to get up to 50% again.[3] McGrath (2015) has another like it. Sharpe’s remains really flat at the end, popping up to what we might imagine is about 20% at best. In contrast to these attention skeptics, MichCommunication (2012) gives us a graph that looks like a bowl, with the conclusion optimistically popping up to what looks like is even over 100%! Reimold and Reimold (2003) have a graph that pops up to the 100% at the end, but really sharply. Like the one I commissioned for this chapter, it looks to be drawn by hand.[4]
If I see you presenting this Curve of Attention as a fact you will lose just about all of your credibility with me. This graph is reproduced because visual aids seem “truthy.” It is what happens when people do simple searches to find something they want to say and need simple support. Everyone just needs to do better research, especially in professional contexts. The Internet makes it easier to find zombie statistics, but it makes it easier to debunk them, as well.




[1] I was going to reprint the graph for you, but it turns out that reprinting one graph from a book 30 years out of print would cost me more than $150. Since I’m trying to keep this book as inexpensive as possible, that wasn’t going to happen. Welcome to the ins and outs of self-publishing! You can, if you’d like, go to Google Books and search “Peter Kenny attention curve” and see the famous graph for yourself, though.
[2] Wilson and Korn (2007) demonstrate the problems with Bligh’s entire approach in their article.
[3] Sam and I decided that our Curve goes to eleven. #ListenToTheSustain
[4] If you Google “Attention Curve” you will see these versions and many others. You will also see some distressing examples of The Curve being adapted to other contexts, especially for Internet content audiences. The virus spreads…

2 comments:

  1. Steven:

    That Attention Curve has a very curious shape with more than one value for attention at some short times. The peak resembles a caricature showing Jimmy Durante lying on his back and focusing on his famously prominent nose:
    http://www.loc.gov/item/2013645898/

    The Attention Curve also resembles a bathtub curve from reliability engineering, as shown in this article:
    http://www.eam2go.com/articles/applyingthepredictiveapproach.html

    A more humorous curve appears in Figure 1 of Rockwood et al’s 2004 article on
    Incidence of and risk factors for nodding off at scientific sessions:
    http://www.cmaj.ca/content/171/12/1443.full

    I’ve also blogged about some other dubious attention statistics:
    http://joyfulpublicspeaking.blogspot.com/2015/04/whats-worse-than-being-boy-who-cried.html

    ReplyDelete