Crocker and Algina (1986) used the term “degrees of consistency” (p. 212), suggesting that p0 is“perhaps the simplest measure of consistency of mastery decisions” (p. 212), but noting that it does not
reflect the degree of consistency; should we be interested in degree, then an index such as Φ(λ) is appropriate. Crocker and Algina’s text has a practical example of how these two statistics are affected by the magnitude, or degree, of classification errors (Crocker and Algina, 1986, Table 9.6, p. 205).
As a matter of interest, when I had Lertap derive the values of Φ(λ) plotted in Figure 5, I took note of corresponding p0 values. Data are displayed in Table 2.
Figure 6: Graph of Φ(λ) and p0 values seen in Table 2
Figure 6 plots the values found in Table 2. (Note that the p0 values in Table 2 were computed using the Peng-Subkoviak estimation method.)
As indicated in Figure 6, differences in the two consistency measures, p0 and Φ(λ), were slight in this case. However, it cannot necessarily be concluded that this is a typical situation. p0 is an indicator of the proportion of consistent classifications; Φ(λ) reflects not only this proportion, but also the magnitude of any misclassifications. It should not be concluded that these measures will always be as similar in value as those seen in this case.
Of general interest is the pattern seen in Figure 6: both measures dip close to the mean of the observed scores, and climb as the cut score departs from the mean – this is a general pattern, and a typical situation.