Scarlet & Grey
Ohio State University
School of Music

Questioning the Obvious

Asking Grandma's Advice

A common criticism of empirical studies is that they simply confirm what we already know. Such studies are a waste of time.

It is indeed true, that a large number of empirical studies generate results that are utterly expected. There is a good likelihood that your grandmother could predict the results of most experiments.

Suppose that your grandmother was 95% correct in her predictions. You design an experiment, and then, rather than carrying out the experiment, you visit your grandmother:

"Grandma, I'm planning to do this experiment ... the subjects will hear these kinds of stimuli and be asked to give these kinds of responses ... what do you think will happen?"

If your grandmother is 95% correct in her predictions, why would we waste time, effort and money actually doing the experiment? Why investigate the obvious?

The answer to this question is that your grandmother will be wrong 5% of the time. Unfortunately, there is no way to predict which 5% she will get wrong, so the only way to find out is to run the experiment.

Now for "grandmother" feel free to substitute whatever expert you want in the above discussion.

Intuition is important in life. We can't test every hypothesis experimentally, so we must rely on experience, "instinct" and intuition to guide us through the innumerable problems and situations we encounter. But our intuitions can fail us. Even the intuitions of the best informed experts regularly fail.

If our expert does indeed guess 95% correct, we are still not absolved from doing the work.

It is not what we don't know that is the principal problem in scholarship. It is what we "know" that ain't so that is the main problem.

The Ethics of Experimental Efforts

As noted above, we can't test every conceivable hypothesis. In research, our activities are limited by physical, human and financial resources. So what should motivate which experiments a research pursues?

Three kinds of experiments are especially important:

  1. The first kind of valuable experiment is the experiment that tests ideas that are important for society. Ideas that concern human lives, health, well-being, and the allotment or potential allotment of scarce public resources.

  2. The second kind of valuable experiment is the experiment that resolves or addresses a pressing argument or debate in a field. If two groups of scholars disagree about something, I feel an experiment coming on.

  3. The third kind of valuable experiment is one that investigates a foundational assumption that is widely presumed to be true -- a presumption from which elaborate theories are spun, but an assumption that has never been tested.

Tempting Ridicule: Scholarship Near the Foundations

Testing the third kind of hypothesis will most often generate results that conform with our intuitions. Much of the time our intuitions are correct, so it is natural that such an experiment might draw ridicule. But what would have happened if the results had not been consistent with our intuitions?

The scholar who regularly tests foundational assumptions in a discipline is most likely to experience derision and ridicule from his or her colleagues. But the scholar who regularly tests foundational assumptions is also the scholar who is most likely to instigate a scholarly revolution. In Popperian terms, this is the scholar who tests hypotheses located closest to the "trunk" of the tree of possibilities. To most people in a discipline, such scholars seem timid and unimaginative. On the contrary, such scholars are engaged in a bold high-stakes venture.

Finally, don't forget the phenomenon of hindsight bias: results always tend to look obvious in hindsight. It is surprising how many scholars respond to experiments that overthrow long-held ideas in the following way: "I always thought that idea was wrong."