Chapter 2 Potential problems with empirical science

Let’s look at some fictitious case studies (science fiction if you wish). We will use them to remind ourselves of the benefits of experimental methods and of the perils of naivety about their limitations.

2.1 Objective evidence

Smith has talked to a lot of people during the last 10 years and made extensive notes. He claims that using the right toothpaste makes you smarter. Smith knows this because he talked to a lot of people and made extensive notes.

Q: Why do we not believe him?

2.2 Observation vs. manipulation

Smith subjected 700 people to an IQ test. He also recorded for each participant which toothpaste they use regularly. (There are only two brands: bling and shiny.) Here’s a visualization of his data:

Figure 2.1: Distribution of IQ-scores for different brands of toothpaste!

A statistical test reveals that there is a significant difference between the two groups of toothpaste users. Smith publishes a paper with the title: ``shiny makes you smart.’’

Q: Why do you strongly dislike this paper?

2.3 The publication-generating process

Smith recruited 50 participants. Each used one brand of toothpaste for 4 weeks before taking an IQ test. A statistical test reveals that there is significant difference between the two groups. Smith submits a research paper with the title ``You are what you brush: shiny makes you smart.’’ to a top-tier journal.

Meanwhile, another researcher, Jones, has independently carried out the same experiment. A statistical test on Jones’ data reveals no significant difference between groups. Jones still submits a research paper to a top-tier journal with the title ``Expect the expected: toothpaste does not influence IQ scores.’’

Three months later, Smith’s paper gets published, Jones’ doesn’t.

Q: Do you think that something like this could happen in reality? If so, why would this be disturbing?

2.4 Researcher degrees of freedom

Jones is frustrated by the rejection. She looks at her data again. She realizes that toothpaste does have a significant effect on IQ scores after all, but only for right-handed participants and the subset of IQ-questions related to language. She also realizes that this ties in with Prof. Brainstawn’s work on lateralization. She submits a paper to a different journal. The paper is accepted as: ``Brush up your language the right way: toothpaste influences on IQ and lateralization in the brain.’’

Q: Why is this bad for science?