Hidden secret of multiple choice questions

One of the exciting changes in my work over the past few years has been the wide range of new question types that are available in online tests. There are so many more ways for students to interact with assessments and new ways for them to demonstrate their skills and knowledge.

Despite this, there is still a big place for the humble multiple-choice question. They’ve not always had the best reputation and they are often perceived as being only suitable for testing shallow concepts and as being prone to guessing. However, a well-constructed multiple-choice question has a lot of power and subtlety and can be a very effective way of surveying what students know and can do.

What is the secret of a good multiple-choice question? It all comes down to the options. The options consist of one correct response which we call the ‘key’, and a number of incorrect responses that we call ‘distractors’. At a basic level, the job of the distractors is to hide the key but if that is all they do then you can end up with a test question that genuinely is shallow and prone to guessing. Back when I was a maths teacher (a long time ago now) I’d see multiple-choice questions that were written with distractors that were three numbers similar in size to the key. This is sometimes characterised as one-below, one-above and one-very close. This approach is very inadequate for several reasons! The most obvious problem is that it encourages students to guess that the answer is somewhere in the middle of the range of numbers!

A more sophisticated method of distractor choice is to consider the likely errors students will make. Not only does this provide meaningful distractors, it is also a useful way to consider the quality of the item in general.  Identifying the common errors involves a degree of guess work but can be made less subjective by examining the performance of similar items in previous tests. A knowledge of the skills and knowledge of the test population is also helpful.

One approach to identifying common errors in a maths question is to look at the steps needed to find the key. For example, the addition 27 + 26 requires the student to add several digits together whilst being aware of place value. The most common error would be to treat 7 + 6 as 3 rather than 13. If students were asked this problem as a free-response question, the most common incorrect answer would be 43.  As we need three plausible distractors, the task in the item has to have enough complexity to generate three errors. The addition 27 + 26 would be inadequate for most audiences. More complex problems may generate so many plausible errors that selecting the best three can be difficult.

Using likely errors for distractors also provides rich information. Each distractor is essentially a code for a specific mistake. That makes it easier to see what kind of errors a large number of students are making. This is a very powerful feature of multiple-choice question and yet it is often overlooked. Few other forms of assessment so neatly enable a teacher to see the kinds of mistakes groups of students are making.

So while I definitely enjoy the bigger range of choices we now have for creating assessments, I still have a place for the humble and oft-maligned multiple-choice question.