Our study explores three interrelated aspects of Lombrozo et al.’s NOS tool: (1)
validity and reliability; (2) instrument dimensionality; and (3) item scales, properties,
and qualities. We examine these attributes within the context of Rasch modeling, an approach for constructing and evaluating measures, which has gained in importance
in recent years (Bond & Fox, 2001; Wilson, 2004). We illustrate some of the
strengths and limitations of the Lombrozo et al. instrument and provide suggestions
for improving it. Given the putative importance of NOS knowledge, our study
attempts to advance instrument quality in this research area. More broadly, we hope
that our investigation will help to outline a general approach for evaluating many
important aspects of Likert-type instruments—and interpreting resulting scores—in
science education research using Rasch modeling.