Experimental Design, Repeated Measures and Order Effects

Between-subjects and Within-subjects

Let's suppose you're interested in the effects of head trauma on memory recall, and you devise an experiment that compares the performance of people across two different conditions: the first, a standard test of recall; the second, a standard test of recall taken after having been hit over the head with a large mallet. (Don't try this at home.)

You then have a choice to make: do you get the participants in your experiment - your experimental subjects - to do the test in both conditions (allowing you to compare the results against each other); or do you split them into two groups, and get one group to do the test in the frst condition, and the second, to do the test in the second condition (allowing you to compare the results of the two groups)? If you opt for the first choice, then your experiment is a within-subjects or repeated measures design; if you opt for the second, it's a between-subjects design.

A within-subjects design tests each subject under all conditions. This has one big disadvantage, namely, that participation in one condition may affect performance in another. So, for example, somebody might do better in the second memory test, despite having been hit over the head with a mallet, simply because they'd had the chance to practice the first time around. This is the problem of what are known as "order effects" or "carryover effects". It really is a big deal, as the data below will show (well, hopefully - if it doesn't - oops!)

Order Effects and the Envelope

In the first part of The Envelope and the Vintage Sedan, people are asked to make a moral judgment about three scenarios - the Envelope, the Sedan and the Thesis. Two of these scenarios - Sedan and Thesis - are designed to elicit the response that there is bad behavior on display; the other scenario - Envelope - is designed to elicit the response that there isn't.

Everybody who completes The Envelope and the Vintage Sedan is required to make a judgment about all three scenarios, which makes it a within-subjects design (where the dependent variable is the moral judgment). This means it is susceptible to order effects. To put it simply, it seems likely that the judgments people make will be affected by the order in which the scenarios are presented.

In order to control for order effects, the experiment randomizes the order in which the three scenarios appear, and then assigns a code to the responses. This makes it possible to determine whether order effects are in fact operating. The chart above shows the percentage of people who responded "I behaved badly" to the Envelope condition - where the expectation is that most people won't make this judgement - depending on whether they saw the Envelope scenario first, second or third.

If there are no order effects, then the percentage will be more or less the same in each case. However, if there are order effects, then one would expect the "Behaved Badly" percentage to increase with each case. This is because both the Sedan and the Thesis scenarios will tend to prime people to respond that there is bad behavior in the Envelope case (and also because people might be aiming for consistency). In particular, one would expect a significant increase in the Behaved Badly score where the Envelope scenario is the last to appear (i.e., in third place).

As of writing, there is not enough data to know whether this will turn out to be the case. However, if it does, then not only will it show the impact of order effects, it will also raise questions about the nature of moral judgments. If moral judgments are susceptible to order effects, then clearly, sometimes at least, they rest on shaky grounds, even if moral agents themselves are not aware of it.