Our second assumption asserts that students interpret the NGCI as intended. Specifically, students understand what
each question is asking, and distractor choices function effectively as probes of student misconceptions.We have two
primary pieces of evidence to support these claims. First, student-supplied responses to the open-ended questions in
Williamson and Willoughby (2012) were foundational in describing students’ na€ıve ideas. Examples of typical
responses allowed question wording to mimic students’ natural language. And, the most common misconceptions
formed the basis for distractor choices.We can gauge the effectiveness of these distractors by looking at the
distribution of student responses. Again, we provide concrete examples of this in Section 5. The second piece of
evidence we have to support our assumption includes information from student interviews. These interviews were
conducted on a voluntary basis with an incentive of $5 to participate. Eighteen students were interviewed while
taking Version 2 of the NGCI, and seven were interviewed while taking Version 3. Students were given a copy of the
NGCI, paper and pen, and they were asked to provide as much information as possible about their thought processes
while working through the survey, including whether they were confused, if they were wavering between answers, or
if they were guessing. Students were audio recorded as they engaged in this “think-aloud” process (Willis 2005),
while the interviewer engaged in “back channeling” by nodding and saying, “Okay,” to encourage the students to
continue (Bolton and Bronkhorst 1996), interrupting the students only to remind them to clarify or elaborate. The
interviewer took notes during the interview and elaborated on relevant issues as soon as possible after the interview
was over. As discussed in Section 5, these interviews were essential in determining how students interpreted
questions and chose their answers. In some cases, information from interviews even suggested new distractors that
proved to be popular choices. Interviews with students who were taking Version 3 of the NGCI became repetitive,
indicating that students were interpreting and answering questions in a predictable way.