12.2 EFFECTIVENESS OF EMERGING TECHNOLOGIES
Few issues have the broad appeal of, yet remain as elusive as, the basic question: Did it work? Interest in testing the effectiveness, or lack thereof, of computer technologies has been long standing. Several approaches have been advanced. Some have attempted to identify the unique learning contributions of computer-based learning systems. Others have focused on the overall effectiveness of computer-based versus noncomputer-based learning systems, while still others have emphasized cost effectiveness. Not surprisingly, results have proved highly variable depending on the focus of the effort and the methods employed to address the question.
This section contains three subsections: cost effectiveness, findings from meta-analyses, and design research. The cost effectiveness subsection analyzes the ways in which technology-supported learning has been approached from a cost-benefit perspective. The meta-analysis subsection emphasizes research undertaken to address "big questions" about the effectiveness of computers. Finally, research is organized according to its implications for the design of computer-based learning systems.
12.2.1 Cost Effectiveness
The value of computers in instruction has been of considerable interest for some time [see Niemiec (1989) for a review of several cost effectiveness studies related to computer-based instruction]. Early in the evolution of computer-assisted instruction, it became clear that developing courseware and acquiring needed hardware would be an expensive proposition. Often, researchers used "cost-added" models, where the marginal gains associated with such systems were evaluated relative to the additional costs incurred in obtaining them [see, for example, the methods described by Levin & Meister (1986) and Niemeier, Blackwell & Walberg (1986) in Kappan ~ issue on the effects of computerassisted instruction]. The approaches were often near-term in nature, with capital costs (computer hardware) and software associated with specific learning tasks evaluated over relatively short durations. Comparisons of recurring costs and maintenance were rare. Using cost-added models, the costs associated with computer-aided learning were rarely considered feasible.
Cost replacement approaches evolved to evaluate the relative costs associated with learning via "traditional" approaches-usually teacher-led, textbook-based methodsversus computer-aided methods. The underlying question of these approaches shifted from assessing the marginal gains using "add-on" technologies to one in which the costs and outcomes associated with the overall delivery system were evaluated. This was the essence of Bork's (1986) "grand experiment," in which he advocated that computerintensive learning systems, designed for optimal impact versus educational convenience, be developed and tested fairly. Judgments as to the true value of computers versus traditional classroom-based teaching on learning could then be assessed, appropriate designs and models could be implemented, true costs (immediate, recurring, long term) associated with each could be identified, and the relative effectiveness of each method could be benchmarked without undue confounding.
Needless to say, these ideas have met with considerable resistance over the past 30 years. The political consequences of altering laws, revising policies and procedures related to teacher preparation and certification, and other educational traditions were, and remain to this day, formidable. The resistance is largely the product of concern over one possible model-total elimination and replacement of teachersrather than a response to a widely held view. In truth, it is not yet known what roles, if any, teachers might play in truly alternative technology-based learning systems.
While Bork's proposed experiment has yet to be implemented on a significant scale, technology has become the centerpiece of many school reform initiatives. Collectively, educators have become more amenable to, and at the forefront of, technology in school reform. A wide range of innovative efforts has been reported, with roles for technology varying from primary delivery of new information via integrated learning systems,.to knowledge-building tools, to management of performance and educational decision making, to comprehensive multimedia in both teaching and learning. The promise is great; however, comparatively little hard data have been generated.
Cost effectiveness is arguably the most critical issue to reconcile from a purely pragmatic perspective. Yet, it may be the most fundamentally flawed in how it has been studied. Increased learning outcomes, even at moderate costs, are unlikely to engender much adoption and diffusion. Consumers seek additional value for their investments. They seek not simply to move marginally further along the same yardstick but to evolve different metrics and ways to measure. They seek to spend