Here is a “secret” to remember when preparing for a high-stakes multiple choice exam: the people writing the test questions look at the available practice problems too. This seemingly simple statement helps explain why some students fail to appropriately utilize a readily available resource. While the use of practice questions is common, a strategy to maximize their efficacy is quite rare.
Over the last decade I have asked hundreds of medical students how they prepare for high-stakes multiple choice exams. Their answers almost always include reviewing practice questions. Question banks for high-stakes exams are ubiquitous and represent big business. For several hundred dollars, students gain access to online questions that closely resemble those they will be asked on their upcoming high-stakes exams. The line between “test relevant” content and illegally stolen material can be murky. Ethical and legal questions aside, new question banks pop up every year. If practice questions are so common then why, you may ask, do students still struggle with these tests?
The answer to this question can be explained by the table above depicting the difference between the examinee’s subjective impression of a question (I know it!) and the actual objective result (correct or incorrect). Quadrant “A” indicates a question where the correct answer was selected and the examinee believed he/she knew the answer. This was a “right answer” selected for the “right reason”. While this type of question feels satisfying, it is essentially a waste of time for the examinee preparing for a high-stakes exam. Because the amount of material is so great, and time so limited, spending time on quadrant “A” should be minimized. Many poor test takers spend too much time focusing on this quadrant, at the expense of a more efficient strategy, because it feels good to get practice questions right.
Quadrants “B” and “D” are also of limited value to test takers, especially if the preparation time is limited. (Author’s note: preparation time will always feel limited.) An examinee is unlikely to sufficiently address large knowledge gaps if an exam is imminent. Investing time in quadrants B and D is relatively low-yield. (Note: If sufficient lead time exists, there are excellent strategies to address both quadrants. A blog post specifically addressing these quadrants is available HERE.)
That leaves quadrant “C”. This is where the money is. Questions that examinees believe they got right and in fact got wrong should be the primary area of focus when using practice questions. Given that most practice banks have thousands of questions, examinees should be able to find numerous questions that fall into quadrant C. This content is familiar to the examinee but somehow an incorrect answer was selected. These questions are worthy of deep exploration. What went wrong? Addressing issues in this quadrant produces the most immediate and longstanding benefit. Quadrant C maximizes your return (score) on investment (time).
Simply memorizing questions that exist in a question bank is falling directly into a well-laid trap. Test writers can easily write question stems that look deceptively similar to questions in popular test banks. This twist can throw off test takers who have simply seen questions without understanding the content. If you think test writers are above subscribing to test banks stop kidding yourself. Like an academic version of spy-versus-spy, those writing questions are obsessively trying to stay one step ahead of the newest question bank to ensure the exam generated will create a meaningful distribution. Multiple choice exams are a full-contact sport. Focusing on quadrant C is an effective and efficient way to prepare for battle.
Life Is Not A Multiple Choice Exam 
Leave a Reply