Some tips on how to evaluate the numbers

Research, surveys and polls

Stalking the stats

    Educators are awash in surveys, studies, and polls, among other sources of information. It’s important to know how to judge their credibility.
    For example, the terms "study" and "research" (as in "research shows") don’t really mean a thing. To judge any results or conclusions, you need to know exactly what the researchers did – Review existing data? Observe a single classroom? Ask a bunch of people on the street?
    Surveys and polls also don’t mean a thing unless you know (1) how the sample was selected and (2) how the question was worded. For example, surveys by pro-voucher organizations consistently show that parents by large majorities favor vouchers that allow children to attend private and denominational schools. When other pollsters re-word that question, asking if respondents want to see tax money used to send children to private schools, the answer is a resounding "no."
    The sampling method is equally important. "Nine out of ten dentists surveyed use Gumrot Toothpaste" is a survey finding that few would accept as credible. Yet, some surveys use equally questionable sampling techniques. Accurate sampling, that allows the responses of a few to be extrapolated to a larger group, is a science. The reason for the good reputations enjoyed by Gallup and Roper, to name two respected polling organizations, is that their sampling is highly refined and that their questions are worded in a neutral fashion. Reputable pollsters report the level of confidence that can be attributed to their figures: "65 percent responded yes, plus or minus five percent," means that 65 percent of the sample said yes, and that if the whole population were surveyed, between 60 and 70 percent would say "yes."
    Be wary, also, of surveys that invite self-serving answers. As this is being written, headlines are reporting a survey that shows one in five teenagers carries a weapon. More than 16,000 teenagers filled out confidential questionnaires in a study by the Centers for Disease Control that also reveals that one in five drove after drinking and that one in ten had attempted suicide. Those are pretty alarming results, all right.
    But if you’ve ever known a teenager, it might occur to you that (1) some of those who said they carry a weapon might like to think of themselves as tough kids who would carry a weapon and (2) some might choose the most alarming response just to scare their elders. None of the news media headlining the survey raised those possibilities, however. Prediction: this "information" will enter the debate on school violence and will be the basis for further alarm among those who think the public schools are going to the dogs.

Stalking the stats

    Most school board members don’t want to be statisticians, nor do they need to be. But board members do need to know how to evaluate the trustworthiness of statistics, and they need to know what questions to ask.
    As Gerald Bracey puts it in "Understanding Education Statistics," "Statistics often mask a political or ideological agenda. When there is something rotten in the state of Denmark, you need to be able to sniff it out by understanding the statistics used."
    To sniff, you need to know certain information that often is not provided. Bracey’s monograph, written for Educational Research Service, offers a crash course in basic statistics. Following is a brief discussion of some of the concepts.
    One of the trickier concepts is correlation. It is tricky because human beings are predisposed to find relationships and to assume causality. If A follows B, people tend to assume that B caused A.
    For example, if a new reading program is introduced and reading scores improve, it’s natural to assume that the program is responsible for the increase in scores. But it’s not necessarily true. The mere fact of a change, or knowing they were being observed, may have inspired the teacher and students to do better. It is important to sort out these variables before expanding the program to the entire district, where it might prove to be an expensive failure.
    One thing you need to know to judge whether the program is responsible for the improvement is the statistical significance of the difference in test scores. To use Bracey’s example, you might teach reading in two different ways and then test the students. You find there is a difference in the average score of the two groups. Statistical significant tests tell you how likely it is that you would have seen a difference as large as the one you saw if there had been no difference between the groups. If the test tells you the difference could easily have been due to chance or to differences in the students themselves, then the test scores don’t really tell you that one way of teaching is better than the other.
    Following is a brief discussion of other terms and concepts that will help you find the truth in the numbers.

• Mean, median, mode: These are all "measures of central tendency," or what we often loosely call "average." The mean is the simple average: add up the numbers and divide by the number of figures. That can be useful in some cases, but is misleading when extreme numbers are involved. For example, calculating the mean per pupil expenditure in Illinois schools results in a misleading figure unless you note the very high-spending schools and the very low-spending. The median is the mid-point; half the numbers are above the median and half are below. When we refer to the "50th percentile," that’s a median. The mode, less used, is the number that recurs most often. If the mean score on a test is 55, but 75 percent of the students scored 65 and the rest got very low scores, then it’s useful to know the mode.
• Measures of dispersion are ways of describing the distribution of figures, or the shape of the curve. One of the simplest is the range, which lists the extremes: scores ranged from 35 to 92. A less simple way is the standard deviation. Calculated according to a formula that only a statistician could love, the standard deviation allows you, among other things, to determine how reliable the mean is – that is, how likely it is if you would obtain the same or equivalent results if you gave the same test again. Percentiles are another measure of dispersion. Standardized test scores are often reported in quartiles and, nationally, one-fourth of the scores will be in each quartile. If all the students in your district score in the third and fourth quartiles, you can truly say that in your district, "all the children are above average."
      The important thing to know about measures of dispersion is that a mean or median figure is not meaningful unless you know how the figures are "dispersed," or how closely they are or are not clustered around the median. The more closely they are clustered, the more meaningful the mean or median is.
• Measures that are easy to confuse include rank vs. performance and rates vs. numbers. A common confusion: when you say the "fastest-growing fields are high-tech and high-skill jobs," you’re talking about a rate, not a number. It’s true that those are among the fastest-growing fields. But the greatest number of new jobs will be in the service industries. The distinction also must be made when discussing dropout rates. Hispanic students drop out at a much higher rate than white students do – but the number of white students who drop out is much greater, because there are more white students.
• Graphs and charts are meant to clarify, but in fact can do the opposite. When you look at a line graph, for example, examine the axis (the vertical line). The smaller the scale (that is, the less units it shows), the larger the differences in the plotted numbers will appear. Bracey uses the example of two graphs plotting the same SAT scores. One from The Public Interest used a 100-point axis. Another from Sandia Report plots the scores on a 600-point scale. On the 600-point scale, the decline in scores appears slight. On the 100-point scale, the decline appears enormous, even though the two graphs use the same numbers.
      When looking at bar charts, notice where the scale starts. If you are graphing two figures above 50, and start the scale at 50, the differences in the two bars will look far larger than if you had started the graph at zero.

IASB ARCHIVES HOME