Analyses
For each test taker, a score was computed for each of the four six-item self-assessment
measures by summing responses over all six items. Correlations were computed among
the four TOEIC scores and the four self-assessments.
For the sake of parsimony, we wished to report analyses based on a single sample that
combined test takers from Japan and Korea. However, pooling heterogeneous samples
can sometimes result in biased or even nonsensical results (e.g., Hassler & Thadewald,
2003). Thus, to justify combining the separate samples from Japan and Korea, we conducted
a preliminary analysis to compare data from each of the two countries.
To address the main objective, we performed hierarchical regression analyses (Cohen,
Cohen, West, & Aiken, 2003) for each domain in order to compute the incremental contribution
of additional TOEIC test measures to the prediction of self-assessed performance.
The order of entry of the explanatory variables (scores from the four TOEIC
measures) was as follows. The TOEIC measure that corresponded to the particular selfassessment
scale being used as the dependent variable (listening, reading, speaking, or
writing) was always entered first, followed by the next most influential predictor, and so
on. (Not surprisingly, the first variable selected in this way also had the most explanatory
power.) The increment in the multiple R2 indicated the value added by considering additional
TOEIC scores. Because TOEIC Listening and Reading scores are always available
in tandem (because the two measures are always given together), they were always considered
in combination as predictors here, instead of separately. Hence, both TOEIC
Listening and TOEIC Reading scores were always entered first when either the listening
or the reading self-assessment measure was used as the dependent variable.
We could have followed other less empirical, more theoretically based orders to enter
variables. For example, TOEIC Writing score could have been entered as the second
variable when predicting performance on speaking tasks, as both speaking and writing
are productive skills. We believe, however, that the central thesis of our study (that there
158 Language Testing 32(2)
is incremental value in considering additional TOEIC measures) applies regardless of the
specific strategy used to determine the entry of variables.