Data Processing and Analysis
An initial coding schema derived from the focal research questions was developed during the final
stages of the sampling process. From this procedure, a preliminary structured data abstraction form
was created (see Miller & Campbell, 2006). The data abstraction form predominately consisted of
fixed items that were binary in nature (i.e., 0 ¼ absence of the characteristic/trait or 1 ¼ presence of
the characteristic/trait). A small number were multiple-selection items (e.g., for theory development
there were multiple coding options and coders could code using any combination of codes) and others
were open-ended (e.g., populations targeted by programs evaluated, design or method used to
support causal inferences).
The data abstraction form predominately consisted of low inference items (i.e., requiring little
judgment), whereas others were high inference items (i.e., requiring a greater degree of judgment).
Corresponding to the focal research questions, fixed items in the data abstraction form were predominately
constructed as they pertained to the specific research questions investigated as well as to
what were perceived as observable enactments of the principles and subprinciples enumerated in
Table 1. Subprinciple 3.b., for example, was excluded from the data abstraction form, since relevant
contingencies related to conducting evaluations (e.g., time and budget) typically are not described in
published research reports. To describe how theory-driven principles are enacted in practice (i.e.,
Research question #3 and its corresponding subquestions), data were coded and analyzed primarily
using open coding methods.5