Our findings for students’ academic achievement seem paradox- ical: the intervention appears to show positive effects at the end of the quarter after finding no effects half-way through the marking period. However, we think this apparent paradox results from a logistical issue rather than a finding of substantive interest. In an unfortunate oversight, we finalized our prespecified hypotheses before reviewing the timing of each key aspect of the study. Although the direction of the estimate for students’ midquarter grades is the same as the end-of-quarter grades, we suspect that the intervention occurred too close to teachers’ grade-submission deadline to have a meaningful effect in most classes. In other words, students may not have had a sufficient opportunity to do enough graded work between the time that they (and their teach- ers) completed their feedback sheets and the date that midquarter grades were due. As a result, we do not discuss this outcome further. Students’ performance on their final quarter grades, by contrast, suggests that the intervention probably caused students’ grades to increase. Our point estimate of this increase corresponds to a little less than a fifth of a letter grade.
To better understand our initial pattern of results, we examined whether our intervention might have had differential effects on different subpopulations of students. By fitting a series of multi- level models (for observed outcomes) and models with robust SEs