Standard Deviations: Flawed Assumptions, Tortured Data, and Other Ways to Lie with Statistics
Now with data so plentiful, researchers often spend too little time distinguishing between good data and rubbish, between sound analysis and junk science.
Patterns in the data are considered statistically persuasive if they have less than a 1-in-20 chance of occurring by luck alone.
One out of every twenty tests of worthless theories will be statistically significant.
Selective reporting and data pillaging—are known as data grubbing.
If a theory was made up to fit the data, then of course the data support the theory! Theories should be tested with new data that have not been contaminated by data grubbing.
Some percentage changes are misleading, for example, when comparing the percentage change in something small to the percentage change in something big.
A statistical fluke can make a big difference if the base is small.
One way to deal with a small base is to use data for several years to get a bigger base.
There can be a statistical correlation without any causal relationship.
There isn’t necessarily any relationship between things that increase with the population—other than that they increase with the population.
A graph should reveal patterns that would not be evident in a table.
Watch out for graphs where zero has been omitted from an axis. This omission lets the graph zoom in on the data and show patterns that might otherwise be too compact to detect. However, this magnification exaggerates variations in the data and can be misleading.
Worst of all are graphs with no numbers on the axis, because then there is no way of telling how much the variations have been exaggerated. Watch out for data that have not been adjusted for the growth of the population and prices.
Graphs should not be mere decoration, to amuse the easily bored. A useful graph displays data accurately and coherently, and helps us understand the data.
We should be cautious about calculating without thinking.
A test may be very likely to show a positive result in certain situations (for example, if a disease is present), yet a positive test result does not insure that the condition is present. It may be a false positive. False positive are more common when the condition is rare (like a malignant tumor) or when there are a large number of readings
Misperceptions are part of our natural tendency to look for patterns and believe that there must be a logical explanation for the patterns we see.
When we see a data cluster, we naturally think that something special is going on—that there is a reason that these heads (or tails) are bunched together. But there isn’t.
When data are used to invent a theory, the evidence is unconvincing unless the theory has a logical basis and has been tested with fresh data.
A study that leaves out data is waving a big red flag.
Extraordinary claims require extraordinary evidence. True believers settle for less.
Data without theory can fuel a speculative stock market bubble or create the illusion of a bubble where there is none. How do we tell the difference between a real bubble and a false alarm? You know the answer: we need a theory. Data are not enough.
If we have no logical explanation for a historical trend and nonetheless assume it will continue, we are making an incautious extrapolation that may well turn out to be embarrassingly incorrect.
Before we extrapolate a past trend into a confident prediction, we should look behind the numbers and think about whether the underlying reasons for the past trend will continue or dissipate.
A careful selection of when to start and stop a graph can create the illusion of a trend that would be absent in a more complete graph.
If the beginning and ending points seem to be peculiar choices that would be made only after scrutinizing the data, these choices probably were made to distort the historical record.
Theory without data—a semi-plausible theory that is presented as fact without ever confronting data. A theory is just a conjecture until it is tested with reliable data. For predictions decades or even centuries into the future, that is pretty much the norm.
We are hardwired to make sense of the world around us—to notice patterns and invent theories to explain these patterns. We underestimate how easily patterns can be created by inexplicable random events—by good luck and bad luck.
Experiments often involve changing one thing while holding confounding factors constant and seeing what happens. For example, plants can be given varying doses of fertilizer while holding water, sunlight, and other factors constant. In the behavioral sciences, however, experiments involving humans are limited. We can’t make people lose their jobs, divorce their spouses, or have children and see how they react. Instead, we make do with observational data—observing people who lost their jobs, divorced, or have children. It’s very natural to draw conclusions from what we observe. We all do it, but it’s risky.
Don’t overlook the possibility of errors in recording data or writing computer code.
Watch out for graphs that exaggerate differences by omitting zero from a graph’s axis.
Be doubly skeptical of graphs that have two vertical axes and omit zero from either or both axes.
Watch out for graphs that omit data, use inconsistent spacing on the axes, reverse the axes, and clutter the graph with chartjunk.
Before you double-check someone’s arithmetic, double-check their reasoning.
The probability that a person who has a disease will have a positive test result is not the same as the probability that a person with a positive test result has the disease.
Correlation is not the statistical term for causation. No matter how close the correlation, we still need a logical explanation.
Don’t be fooled by successes and failures. Those who appear to be the best are probably not as far above average as they seem. Nor are those who appear to be the worst as far below average as they seem. Expect those at the extremes to regress to the mean.
Good luck will certainly not continue indefinitely, but do not assume that good luck makes bad luck more likely, or vice versa.
Don’t be easily convinced by theories that are consistent with data but defy common sense.
Watch out for studies where data were omitted, especially if you suspect that the omitted data were discarded because they do not support the reported results.
If a theory doesn’t make sense, be skeptical. If a statistical conclusion seems unbelievable, don’t believe it. If you check the data and the tests, there is usually a serious problem that wipes out the conclusion.