Reliability. In addition to validity, our study investigated the issue of instrument
reliability (AERA et al., 2004, p. 25). A high-quality and valuable instrument is
expected to provide consistent measurement outcomes (i.e., it produces reliable
inferences). We investigated the reliability of the Lombrozo et al. instrument using
both Classical Test Theory (CTT) and IRT (specifically, Rasch modeling). Within
CTT, one can distinguish between several reliability coefficients depending on
which approach is used for replicating the measurement (e.g., test–retest reliability,
reliability of parallel forms, or internal consistency). Given that only one form and
one single test were administered, the appropriate CTT approach available for
investigating reliability in our study is internal consistency. Internal consistency
splits the test into two or more parts, and consistency within those parts is calculated.
The most widely used coefficient of internal consistency is Cronbach’s α
(Haertel, 2006). Using α eliminates a source of error associated with an arbitrary
split choice. It is also one of the most commonly used measures of reliability in
science education.