April 28, 2010

Misleading Charts: Science in Pittsburgh Schools

My pet peeve as a chart geek is misleading charts, and the most galling of these is caused by three-dimensional presentations of one-dimensional variables. Although well-intentioned, and recognizing that people are led to this offense through enabling software, the tendency to use PizZaz in Presentations results in misinformation and obfuscation.

I am led to this chartwise anal-retentiveness by Edward Tufte, who says that if you want to communicate clearly, you should study obfuscation and misdirection.

I was reading Infinonymous today and saw a chart from the City Paper, purporting to communicate the percentage of students at some Pittsburgh high schools who perform satisfactorily in science:

This chart, ostensibly intended to rapidly and easily inform the reader, misleads in many ways. Tufte would consider it an exercise in chartjunk.

First, let's talk about the use of colors. Here's a copy of the image with the numbers removed.


How do the colors influence you? Are they in a spectrum? If I were to describe the colors (Occidentally), I'd probably say:

And so, reading/scanning left-to-right (as we do), I'd say that there is a progression, and that the blue must be even better than the green. That conveys that 37% is a good score; I'm not so sure it is. There's no justification presented to make it a good score.

How does the fullness of the container influence you? I'd suggest that these values are intimated by the container's relative fullness:

The "fullness" of the right-hand bottle, for instance, suggests an accomplishment other than 37%. They're almost "all the way"! Surely the Allderdice score is not meant as the measure of maximum performance. (In fact, it's less than the state average value.)

What's with the scales on the lab flasks? We've already discussed this visually, but aren't percentages measured from 0 to 100, not 0 to 40?

What's with the linear scales on the flasks? Doesn't one inch of fluid at the bottom of the flask contain a lot more fluid than an inch at the top of the flask? What kind of flask is that, anyway?

That's an Erlenmeyer flask, thank you very much, and an inch at the bottom is a lot more volume than an inch at the top. In fact, here's a photo of the scale on the side of an accurate Erlenmeyer flask, and you can see that it's a logarithmic scale. The distance on the scale between 0 and 200ml is much different than the distance on the scale between 300 and 500ml.

The scales used in this graphic are completely misleading.






Here's the information in a table:
% of students Proficient or Better in science
WestinghousePeabodyPerryBrashearDistrict Avg.Allderdice
2.34.09.419.820.237.1


Here's a way to present the information graphically:
% of students Proficient or Better in science
Westinghouse2.3
Peabody4.0
Perry9.4
Brashear19.8
District Avg.20.2
Allderdice37.1


One problem with the above depiction is that visually, it looks like Allerdice is doing pretty well. Here's a way to present the information with a bit of context, by showing their percentage values in the context of 100 units:

% of students Proficient or Better in science
Westinghouse2.3
Peabody4.0
Perry9.4
Brashear19.8
District Avg.20.2
Allderdice37.1


The City Paper text contains an interesting datapoint: the overall Pittsburgh percentage is 20%, and Pennsylvania statewide is at 40%. That's a bit of context which would have been great at the top of the article. There's more info that I'd love to see. I'd love to see a chart or table that puts the Pittsburgh data into perspective.
  • How does Pittsburgh rate among systems in other cities with the same number of students?
  • How do Pittsburgh public schools rate against Pittsburgh Catholic schools?
  • How does Pittsburgh compare to systems that spend the same amount (per pupil) as Pittsburgh does?
Data like that in tables and accurate charts would truly be informative.
(edit: snarky exaggerated comment removed)

If this is the way the school system and the newspapers communicate data, it's no wonder that only 20% of 11th-grade students are proficient-or-better in science.

6 comments:

Chris Potter said...

"If this is the way the school system and the newspapers communicate data, it's no wonder that only 20% of 11th-grade students are proficient-or-better in science."

>>> For starters, you may be overestimating our news section's influence on high school kids. Most of them flip right to "Savage Love."

I'd be more willing to concede some of your points if the numbers themselves weren't written in BIG DIGITS right on the flasks themselves.

To me, the numbers communicate a lot more clearly than, say, an attempt to parse the nuances of the color scheme. (Is purple "neutral"? I've always associated it with Barney the Dinosaur and, to a lesser extent, the Baltimore Ravens.)

Yeah, the Allderdice flask is "nearly full," and I suppose some people might ignore the giant "37.1 percent" and say, "Oh, I guess there's not a problem at Allderdice." But considering that the headline for this graphic is "failed formula," and the headline for the story itself is "Down the Test Tubes" ... it might be a BIT of a stretch to say the message is that students ANYWHERE got a "good score."

But I readily admit: If one were to SUBTRACT those headlines ... and one were to ERASE the big percentages written on the flask ... and one were to REPLACE the giant "37.1%" label with the word "FULL" ... if one were to do all those things, then you would certainly have a very misleading image indeed.

You've done an excellent job of proving that.

It's not that this graphic couldn't be better. (For example, we could have used test tubes instead of beakers.) It's just that your critiques are a pretty flimsy basis for the loaded allegations you make here -- that we haven't done journalism, for example.

-- Chris Potter

Vannevar said...

Hello Chris,

You're right that the journalism comment was unjustified, and I've removed it. I apologize.
Vannevar.

Bram Reichbaum said...

The colors were the thing that really struck me. What does light blue represent? Purity at Allderdice?

That's the kind of thing that can hardly be attributable to sloppiness IMO, but to the subconscious and despite oneself. And that's deep.

Chris, I think it's a very fair object lesson -- if one is going to utilize an artistic chart, do make sure all the major artistic aspects encourage a fair reading of the data and the intent if any.

Chris Potter said...

We're cool. I'm a fan of the blog, which is why this bruised my ordinarily calloused exterior. Keep up the fine work!

C. Briem said...

Gotta give Vannevar a subscription to the USAToday for xmas. :-)

Anonymous said...

logarithmic? parabolic, maybe.

Agreed that there's trouble available for anybody who tries to take precise information from those graphics, and messages other than the factual one in the numbers *may* be taken away.

Post a Comment

Comments and Feedback? Love that stuff. Please leave your thoughts in the box below--