The most difficult item on the NGCI, Item 25, also falls within the Force Law Domain. As seen in Figure 1, this
question maintained a high CTT difficulty value throughout the development process, and Table 7 shows its
evolution through drastic changes. Item 25 probes how changes in mass and distance affect gravitational force by
using a hypothetical comparison of two planets of different sizes but equal gravitational force for a person on
their surfaces. At first encounter, Item 25 looks very similar to two other items on the NGCI (Items 14 and 17),
where simple reasoning about mass, rather than size (or distance), can lead to the correct answer. Indeed, the
most effective distractor corresponds to the idea that “same gravity implies same mass,” which is only true if
distance is the same. However, here, students must apply their reasoning about mass coherently with the idea of
size (or distance), since a person on the surfaces of the planets would be at different distances from the centers of
mass. So, even students that perform very well on the NGCI may get this question wrong because they do not
double check the factor of distance. This likely explains the high difficulty and poor discrimination index of this
question. However, even though CTT statistics did not improve throughout the development process, student
interviews suggest that the changes did lead to a conceptually sound question that minimizes non-gravity related
errors in student reasoning. Specifically, because Versions 1 and 2 compared Earth and Saturn, rather than
hypothetical planets, students reasoned with their preconceptions about Saturn rather than their understanding of
gravity. Some students were confused because it would be impossible for a person to stand on the surface of
Saturn, and some students simply did not believe what the question was asserting. Furthermore, changing to a
hypothetical scenario allowed distractor choices to represent common misconceptions in the Mixing of Forces
Model. The purpose of this question is well aligned with the concept domains presented in Section 3, and it
probes the vitally important student ability to correctly distinguish situations where either mass or distance is
different. Therefore, despite its poor performance in terms of CTT statistics, we support the inclusion of Item 25
on the NGCI.