The levels shown above were derived from a multistep process [PISA 2005b]
as follows: (a) Mathematics curriculum experts identified possible subscales in
the domain of mathematics, (b) PISA items were mapped onto each subscale,
(c) a skills audit of each item in each subscale was carried out on the basis of a
detailed expert analysis, (d) field test data were analyzed to yield item locations
on subscales, (e) the information from the two previous steps was combined.
In this last step, the ordering of the items was linked with the descriptions of
associated knowledge and skills, giving a hierarchy of knowledge and skills
that defined possible values of the progress variable. This results in natural
clusters of skills, which provides a basis for understanding and describing the
progress variable. The results of this last step were also validated with later
empirical data, and by using a validation process involving experts. Note that
this method of developing a progress variable is much less precise than the
approaches described in the references above (e.g., [Wilson et al. 2000; Wilson
and Scalise 2006], and will thus usually result in a progress variable that is much
broader in its content.