by Dr. Fred Worth, Department of Mathematics, Henderson State University, posted May 29, 1996. Dr. Worth notes the much of the material for this paper came from the book "How to Lie with Statistics" and Reader's Digest.
The world today almost seems to run on statistics, surveys and polls. Especially considering that this is an election year, it is essential that people understand what statistics say and do not say and when they are or are not reliable. [note: Much of the data contained in this paper comes from the book "How to Lie with Statistics." The book was published over 40 years ago so some of the data is outdated.] Statistics can cause great deal of worry. We have all heard "the average child walks/crawls/talks/etc. by the age of ... ." A mother hears that and thinks, "Oh no, my child doesn't walk yet, what's wrong with him?" Very likely, there is nothing wrong with the child. My son didn't walk until he was about 19 months. He didn't crawl until around the same time. Was there something wrong with him? No, he was just very heavy and needed to develop more strength than most kids do. Variation is important to consider. IQ testing is another area that can cause a great deal of stress, especially for parents. Suppose you want to measure off a football field. Your stride is about 1 yard long. So you step off 100 yards. How accurate will it be? Within 3 feet one way or another? Long by more than 3 feet? Short by more than 3 feet? Reasonable guesses (where P(A) denotes the probability that event A happens) would be P(97
103). IQ tests (by themselves) are likely no more accurate than that as a measure of intelligence. All kinds of other factors may enter in. Background, health on the day of the test, what you read just before you took the test, fatigue, etc. Suppose the statistics ARE accurate. What do they prove? Cigarette brand X has 50% less tar/nicotine/carbon monoxide/etc. than the other leading brands. Does that make them healthy? More doctors smoke brand X than any other brand. That may very well be true. Does that mean that smoking brand X is a healthy choice? A mechanical juicer was advertised as being able to extract "26% more juice." That sounds pretty good until one realizes that the ad did not say "26% more juice" than what. It turns out it was 26% more juice than a hand juicer. It was not being compared to a comparable product. More car accidents occur in clear weather than foggy weather. We therefore conclude that it is safer to drive in the fog than in clear weather, right? Of course not. Is the statistic false? No. How meaningful is the statistic. Not very. Consider that, in most places, clear weather is much more common than foggy weather. Consider also that when it is foggy, many people will avoid driving. Thus, it is to be expected that more accidents will occur during clear weather than foggy weather. A more useful statistic to consider is NOT number of accidents but number of accidents per mile driven. It was reported that the death rate in the Navy during the Spanish-American war was 9 per 1000. The death rate in New York City during the Spanish-American war was 16/1000. Are we then to conclude, that during the late 19th century, it was safer to be in the Navy during a war than it was to live in New York City? That is simply absurd. The two statistics do not compare the same kinds of populations. The Navy would have young, healthy adults. New York would have a far more diverse population. It would include infants, the elderly, the ill. All of those populations have much higher mortality rates than young, healthy adults. Arkansas ranks 49th in ... . We hear statistics like that all of the time. The intended reaction is "Oh, no. We have to improve" whatever it is that was being measured. Consider, however, that no matter what is being measured, if you have 50 states being ranked, one of them must be 49th. That of itself does not mean that any of the states are deficient. "Joe Smith is the paid major league baseball player." Does that mean Joe is ready for the poverty line? Consider that all major league players make more than $100,000 per year. That hardly qualifies them for public assistance. A study once showed that students who smoke have lower grades. Some people concluded from that data that smoking causes lower grades. Such a conclusion is not valid without considerably more evidence. The problem here is distinguishing between causation and correlation. Simply having two events happen together does NOT mean either one necessarily caused the other. There are more weddings in June than any other month. There are more suicides in June than any other month. Therefore, we conclude that weddings cause suicide. Again, we have an absurd conclusion. How many of the suicides were people who recently married or knew someone who recently married? Is it possible that other factors could better explain the data? It would be just as silly to say that since June has more weddings and that June has only 30 days, that all of the weddings cause June to have fewer days. Cornell did a study (approx. 1950) of middle age alumni. It showed that 93% of male graduates were married and 65% of female graduates were married. Female college graduates were three times as likely to be unmarried as female non-college graduates. Therefore, some people concluded that graduating from college makes it less likely a woman will marry. Such a conclusion ignores vital information. Isn't it possible that those women who married during their college careers were more likely to drop out to pursue family goals rather than educational or career goals? Another vital conern is how reliable the data are. Suppose I ask 100 people how much they slept last night. I add up their answers and get 783.1 hours. I report the average person slept 7.831 hours. Sounds like a precise study. It is worthless. Do you really know how long you slept last night? Around 1949, Yale took a survey of their graduates. The survery showed that the "average" salary of Yale grads was $25,111 per year (a princely sum in those days). What does this mean? Does it prove that Yale graduates make more money than those of other colleges, or more than people in general? Actually, no, it doesn't. Several questions must be considered. First, what "average" was used? There are three frequently used statistics that are often called averages. First is the mean. It is what most people think of when they think of an average. The mean is obtained by adding up the numbers and dividing by the number in the sample. For example the mean of 5, 6, 6, 7 and 11 is (5 + 6 + 6 + 7 + 11)/5 or 7. Second, is the median. The median measures the middle value. The median of 5, 6, 6, 7 and 11 is 6. Lastly, the mode denotes the value that occurs most frequently. The mode of 5, 6, 6, 7 and 11 is 6. Each of these "averages" are useful but all can be deceptive. Back to the Yale salary study. If "average" implied the mean, consider the following possibility. Suppose 100 graduates answered the survey, one made $2,000,000 and one made $500,000. Then the other 98 "average" $113 per year. If the mode was used, it is possible that 2 graduates made $25,111 while all the others made under $1000 per year. In general, the median is a better measure for populations with possibly wide variations, like salary. But even if the median is used, is this useful information? How was the study done? It was done by mail and depended on the respondent giving his/her salary. This creates several problems. Are people honest? They may lie, giving a higher salary (to look good) or giving a lower salary (lied to IRS so they want to stay consistent). They may not know how much they make so they may have guessed. It is highly probable that they did not get all of the living graduates to respond since some would have whereabouts unknown (likely unsuccessful) or others would not want to respond (likely unsuccessful). These various problems, unless addressed somehow, make the study relatively useless. Suppose I want to do a study of the average height of people who play miniature golf at a particular miniature golf course. I do the mean and get a mean height of 4'8" and a median height of 4'8". I thus conclude that most of the people there that day are around 4'8". Good conclusion? Well, it turns out that it was father-son day at the local day care. So I found 24 fathers with their 3 year old kids and one 10 year old brother. Variance also is a valuable tool when doing statistics but then it reporting results starts getting so complicated that the general public (and the media reporting the statistics) is not going to understand whether the information is valid or not. It is sort of like the guy who had his head in a hot oven and his feet in liquid nitrogen who said "on the average I feel fine." A classic example of a bad use of statistics occured during the presidential election of 1936. Literary Digest magazine did a presidential election poll in 1932. Their results were very accurate. So they did it again in 1936 using the same methodology. Their results showed Alf Landon would win over Franklin Roosevelt by an electoral vote margin of 370 - 161. The actual results were a little different. Roosevelt won 523 - 8. What happened? They did their survey via a telephone poll of their subscribers. Why would that be a problem? What happened in 1929? The stock market crashed and the depression began. By 1932, things were getting bad but not too bad. By 1936, things were getting really bad. People gave up "unnecessary" things like magazine subscriptions and phones. Who still had magazine subscriptions and phones? Rich people. In the 1930s rich people tended to be Republicans so it was natural that a majority of those polled would vote for the Republican candidate. Another problem to consider is emotion. Whenever you deal with emotionally charged issues you need to be especially careful. Abortion is a prime example. Both pro-abortion groups and pro-life groups can show polls that show that the majority of the public supports their view. How do we word those questions? If I want to show support for a pro-abortion view, I would ask "Should women be allowed to make their own medical decisions without the intrusion of an overbearing federal bureaucracy?" If I want to show support for a pro-life view, I would ask "Should doctors be allowed to mercilessly butcher innocent little babies?" Obviously the wording affects the response. It is often approximately 10% of the population is homosexual. That statistic has been used so many times that it has come to be accepted as fact by many people including the media. It is instructive to look at the study from which those numbers were derived. In the 1940s, Dr. Alfred Kinsey did a study at the University of Indiana. It is in this study that the oft repeated claim of 10% homosexuality originated. This study, however, was badly flawed. To take results of a sample and draw inferences about the entire population one must be sure that the sample is representative of the entire population. Kinsey used primarily volunteers for his study. He used second and third groups of volunteers, many of whom were referred by the first group. These facts by themselves make any comparison to the actual population highly suspect. In addition to this, however, is the fact that a disproportionate percentage of Kinsey's study was made up of prison inmates. One of the results of Kinsey's study was that about 10% of his sample was "more or less" exclusively homosexual for about three years of their lives. This is the statistic that people claim "proves" that 10% of the population is homosexual. A careful reading shows that not even Kinsey, with his poor methodology, supports the claim of 10% homosexuality. Kinsey's results only show four per cent of males and 1 per cent of females to be exclusively homosexual for more than three years of their lives. Between 1981 and 1984, three different studies (published by Indiana University Press, Journal of Psychology and Theology and Playboy Press) all came up with results very different from 10%. They found that 96% of the population consider themselves exclusively heterosexual. Between one and three per cent consider themselves exclusively homosexual. In a recent study, a group from the University of Chicago, reported that only one per cent of Americans identify themselves as homosexuals. In the July 3, 1992 issue of Science magazine, a study of human sexuality was reported which showed less than 4.1% of the male population had participated in homosexual intercourse at least once in their life. Only 1.1% of the male population had participated in homosexual intercourse in the past year. In the female population, the rate of lesbian activity was 2.6% for at least once in the lifetime and 0.3% in the past year. It is also often reported that heterosexuals are responsible for more cases of child abuse than homosexuals. This is true. But the intent of many who report that data is to try to make it seem that heterosexuals are more likely to be abusers. While homosexuals make up between one and two per cent of the population, they account for more than 33% of all child abuse. Thus, homosexuals are between 24 and 48 times more likely than heterosexuals to be child abusers. Another emotionally charged statistical claim is that men earn more than women. Many problems come up in such a statistic. Are they comparing similar jobs? Are they considering years in the job? Are men more likely to put in more hours because of women spending more time at home with their children? Some statements are worded so as to make them sound good. In 1948, it was reported "Today, electric power is available to more than 3/4 of U.S. farms." It could have been worded "Almost 1/4 of U.S. farms do not have electrical power available to them." Also, note that it said "available." It didn't say that they had it. Suppose I say, "Today, Mercedes Benz automobiles are available to more than 80% of the American public." This could just mean that more than 80% of the American public lives within 50 miles of a Mercedes dealer. Graphs of data, even though they contain no words, can also be very deceptive. Consider the graph below, which shows the production of widgets by two competing factories. Factory A XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Factory B XXX The graph seems to indicate that Factory A makes considerably more widgets than Factory B. The graph is deceptive because no labels are given. Here is the graph again, with labels. Factory A XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Factory B XXX 1950 2000 It now is clear that the difference in production between the two is not very significant. Another example of a deceptive graph was once used to demonstrate the amount of tax money collected by the federal government. A map of the United States was shown. States were shaded in. The taxes collected by the federal government were supposed to be equal to the amount of revenue produced by those states. [Keep in mind that this study was over 40 years ago, so it was before the large population shift to California and the southwest.] Shaded in were Washington, Oregon, California, Nevada, Idaho, Montana, Wyoming, Utah, Arizona, New Mexico, Colorado, North Dakota, South Dakota, Nebraska, Kansas, Oklahoma, Texas, Minnesota, Iowa and most of Missouri. That covered more than half of the country. The map could just as easily been done shading New York, Pennsylvania, New Jersey, Massachusetts and Connecticut. That shading covered very little of the country. Both were correct since the eastern states at that time, though much smaller, produced significantly more revenue than the western states. Polls can be used to show the opposite of the truth. Princeton's Office of Public Opinion Research once did a poll of people to determine racial attitudes. People were asked if they felt that blacks had as good a chance to get a job as a white. Other questions were asked to determine the racial attitudes of the person. Two-thirds of those who were sympathetic to blacks said that blacks had a poorer chance of getting a job. Two-thirds of those showing prejudice felt blacks had an equal chance of getting a job. Thus, the poll could be done during a time of relative racial harmony, then again during a time of racial strife. As racial attitudes worsened, the poll could show "Blacks have an increasing likelihood of being able to get jobs." The frame of reference is vital to proper interpretation of statistics. Suppose you have an investment. Last year you got a return of 3%. This year you have a return of 6%. You could say your return was 3% better than last year. You could say your return was 100% better than last year. Both statements are "true" though the certainly give a very different impression. Another example of poor use of statistics and polls is "Selective Reporting." In January 1995, the U.S. Agency for International Development (AID) reported (to great media response) that Americans support foreign aid. A Reader's Digest poll showed that 86% of the people don't know what AID does. It also showed that 67% want foreign aid cut. The same question was asked, in the context of cutting the deficit, and 83% said cut foreign aid. So where did AID get there information? A poll asked if the US should "share at least a small portion of its wealth with those in the world who are in great need." Over 80% said yes. They were asked if foreign aid should be cut a little, somewhat, a lot or "eliminate it entirely." 8% chose the last option. They took this to mean 92% "support" foreign aid. They ignored that 75% said there is "too much" foreign aid. "2 out of 3 doctors recommend XYZ toothpaste." How did they decide that? Easy. Find two dentists that recommend XYZ and then find one other dentist who doesn't. Polls show support for increasing spending on PBS and support for decreasing spending. If you want to get support for decreasing spending then emphasize the budget deficit. If you want to get support for increasing spending then emphasize violence on regular television. Polls show support for increasing spending on the National Endowment for the Arts and support for decreasing spending. If you want to get support for decreasing spending then emphasize the budget deficit and morals. If you want to get support for increasing spending then emphasize censorship and freedom of speech. Probability is another source of confusion with statistics. In basketball, if a player who typically makes 50% of his shots misses five in a row, we tend to think that something is badly wrong. However, the probability of that 50% shooter having at least 5 consecutive misses in any span of 11 shots is 255/2048 = .1245, almost 1 in 8. It is not at all rare. The probability that in a span of 6 sets of 11 shots, that he will miss at least 5 consecutive shots in one of those sets of 11 shots is 1 - (1 - 255/2048)6 = .5496, better than a 50-50 chance. If I flipped a coin and got heads 75% of the time the tendency might be to think that it must be a biased coin. The problem here is the need to know how many tosses are involved. If there were only 4 tosses then the probability of 3 heads is 1/4. With a "fair" coin, as the number of tosses gets "large," P(half heads) goes to 1/2. I wrote a computer program to simulate 10,000 tosses of a coin. I ran it 10 times. Here are the results. HEADS TAILS 4974 5026 5036 4964 4963 5037 5000 5000 5024 4976 4995 5005 5038 4962 4983 5017 4989 5011 5023 4977 50,025 49,975 The greatest disparity was in the 7th run. There were 76 more "heads" than "tails." But that only means 76/100 of 1% difference. Overall there is only a 1/20 of 1% difference. In standardized testing, percentiles are used to compare scores. Suppose in a collection of 300 students, a test is given with 200 possible points. Suppose John is on the 99-th percentile, Dave is on the 90-th percentile, Mike is on the 60-th percentile and Bill is on the 40-th percentile. Let J denote John's score, D Dave's score, etc. We might guess that J - D