CONCLUSION
This short note has sought to point out some concerns related to use of the test-retest correlation as a reliability indicator. These shortcomings are not generally recognized or discussed in the literature. Some hypothetical examples were chosen to demonstrate certain possible occurrences in actual situations. We hope to encourage awareness of factors that influence the value of this frequently used reliability measure and recommend consideration of supplementary statistics, especially when attempting a comparison involving two or more data sets. Comparisons of rebability from one study to another may be most reasonable when comparable product categories are involved and when the same scale of measurement (such as 0-100) is used at each replication.
The test-retest correlation has other potential problems as discussed in (6) and (7). Both Gibson (3) and Adams et al. (1) also discuss factors which can affect reliability of recall test scores and the need for equivalent test-retest conditions. The concerns expressed in this paper are essentially independent of and in addition to these. While this paper has focused on the test-retest correlation's use in the context of advertising research and copy testing methods, the major points of the discussion are generalizable to other research settings where the test-retest correlation is used as an indicator of reliability.