Bias, empiricism, and the hierarchy of evidence

How can we make sense of all the evidence in the midst of all this noise—how can we decide what’s true? There no simple answer. Experts will disagree. Sometimes research showing causation doesn’t exist, and sometimes multiple research studies come to conflicting conclusions. These simple guidelines, however, should help.

First, consider the bias of the source. What might they gain financially or otherwise by having the world believe their conclusions?

Second, always favor empiricism over dogma. Empiricists make careful observations of what really happens in the world—they measure; whereas, dogmatists, simply draw conclusions from their imaginations. Consider the DARE program, a program that attempts to keep school kids from using drugs and alcohol. From about 1983 to 2009, the program used dogma—those who ran the program imagined that if they got kids in a room and lectured them about the dangers, substance abuse would go down. When researchers carefully measured the outcomes empirically, they found that the program didn’t lower substance abuse at all. Those involved in the program were slow to accept this. They wasted (mostly) all those resources for twenty-six years. Finally in about 2009, they changed the approach, carefully measured the outcomes empirically, and found that the new approach works. (Nordrum, Amy. “The New D.A.R.E. Program—This One Works.” Scientific American. September 10, 2014 http://www.scientificamerican.com/article/the-new-d-a-r-e-program-this-one-works/)

Last, use this simple “Hierarchy of Evidence,” which lists types of evidence from most to least compelling.

Hierarchy of evidence:

  1. Consensus of the experts based on research proving causation
  2. Multiple pieces of research showing causation
  3. A single piece of evidence showing causation
  4. The opinion of a group of experts, based on their experiences
  5. The opinion of a single expert, based on her experiences
  6. Many correlations
  7. A single correlation
  8. A single story (anecdote)
  9. Someone’s opinion based on his or her imagination