Approach 4: Evaluation by Pretext
Evaluation by pretext exists when an evaluator earnestly and honestly proceeds to conduct a sound evaluation to serve a stated purpose that, unbeknownst to the evaluator, is deceptive and false. In such a case, the client is guilty of the indiscretion of misleading the evaluator. The evaluator is guilty of proceeding with the evaluation without confirming that the evaluation’s stated purpose is the actual purpose. The nature of evaluation by pretext is seen in the following true example.
A research center’s director had recently been appointed and wanted a baseline evaluation of the center’s programs. He contracted for an independent evaluation of the programs. When the evaluation team arrived for its three-day site visit, the center director informed them that the evaluation’s purpose should be to identify the full range of flaws in the programs as a basis for program improvement.
He had reviewed previous evaluations of the center’s programs and found them reassuring in regard to the high quality of work in the center. These evaluations had been conducted and reported on rigorously and independently. Previous evaluators mainly had found the center’s programs to be sound and had lauded them for their importance, rigor, productiveness, and accountability. The director said that although such positive reports had no doubt been good for staff morale, they had not included detailed direction for program improvement. This year he said the evaluators should set aside any search for program strengths; instead, they should
concentrate on identifying and cataloging weaknesses.
He said candid reports along these lines would be invaluable to him and the staff for finetuning their already good programs and making them truly outstanding.Heasked the evaluators to present their findings of program weaknesses at a full staffmeeting during the last afternoon of the site visit.
Unfortunately, the evaluators swallowed the director’s request and reasoning—hook,
line, and sinker. For three days, they delved into each of the center’s programs. They were determined to identify and document the full range of weaknesses in each program.
At the end of their visit, the evaluators went to the auditorium where they would orally
deliver the findings to the center’s staff. To the surprise and consternation of the evaluators, the audience included not only the center’s staff but also officials from the federal agency that was funding the center’s work. The evaluators wished they were in a position to present a balanced assessment of each program’s strengths and weaknesses because the center’s funding undoubtedly was at risk. However, they were prepared only to present what they had searched for and found: program weaknesses. The evaluators’ recitations on program weaknesses cast a pall over the entire meeting and undoubtedly misled the federal officials as to the true merit and worth of the programs being reviewed.
Why would the center’s director orchestrate such a disastrous chain of evaluation events? It turned out that he had not liked the previous director of the center, wanted to discredit his leadership, and was seeking to replace the center’s programs with others of his choice. These were the evaluation’s real purposes as viewed by the director. In this evaluation by pretext, the evaluators had unwittingly played into the director’s hands. With some advance exploration before signing on to do the evaluation, they might have learned of the director’s deception and declined the evaluation assignment or insisted on making a valid assessment of strengths
and weaknesses.
Before contracting for an evaluation, it is a good idea to obtain information from a wide range of program stakeholders. It can be particularly enlightening to consider who might be hurt by the evaluation and to invite their reactions. In interacting with stakeholders, prospective evaluators should outline what they have been asked to do and inquire as to what concerns they should consider before agreeing to conduct an evaluation. Also, evaluators probably should not agree to collect and report only strengths or only weaknesses in a program. As stated in the 1994 and 2011 editions of the Joint Committee on Standards for Educational Evaluation’s Program Evaluation Standards, evaluators should fairly appraise both strengths and weaknesses in a program.
The main advance organizer in an evaluation by pretext is the client’s directive to the
evaluator and rationale for the directive—for example, to identify program defects as a basis for program improvement. The client’s purpose is not the purpose given to the evaluator. In the example here, the director’s purpose was not program improvement, as stated to the evaluators, but program termination and discrediting of the previous director. Clients are the source of evaluation questions that guide evaluations by pretext. Although clients should be one source of evaluation questions, they should not be the only source: other stakeholders and evaluators themselves should also contribute evaluation questions. Typical questions in evaluations by pretext focus on negative aspects of a program, but they may be more varied depending on the evaluation’s hidden purpose and could concentrate on only positive features of a program. Because evaluations by pretext employ and are led astray by the evaluation questions stated by the client, methodology is not the source of these evaluations’ problems. This approach to evaluation has no redeeming qualities and can be seen as disturbingly Machiavellian.