xt to a choice of methods, activities, and tools, the planning of an evaluation also requires
the selection of respondents. In the case of teaching materials, four groups of respondents
can be distinguished:
• the design team itself
• experts, including subject professionals, designers, teacher educators, education
supervisors
• users, including teachers and pupils
• other stakeholders, such as parents.
The type of respondents to be selected depends on the particular evaluation question.
Subject matter experts will be able to provide a clear answer to the question whether the
design is in line with recent insights within a certain knowledge domain. If, however,
the design team is ready to develop a layout for the design, graphic designers need to be
involved.
Once a certain group of respondents is selected, it is important to consider the number
of respondents to be approached. Especially during the development stages, the most
important objective of a formative evaluation is to locate shortcomings and to generate
ideas for further improvement of the design. For this purpose, it is not necessary to select
large numbers of respondents. More important is to ensure triangulation, i.e. including a
variation of respondents per respondent group (for example, two strong pupils, two weak
pupils, and two average pupils, in the case of a micro evaluation). Thus, the information
from one respondent may be verified or complemented by information from others. For
example, a screening of the design may be performed by several developers, and a try-out
carried out in three different classes rather than a single one.
When planning an evaluation, it is also important to consider the role fulfilled by
developers themselves during the evaluation. During later development stages, it will
be desirable to involve external evaluators, rather than the developers themselves, to
evaluate the actual effectiveness of the curriculum (see paragraph 3.3.3). During the early
stages, however, it seems legitimate and even desirable to let developers conduct the
evaluation activities themselves. In this way it is more likely that the evaluation will lead
to modifications in the curriculum. The development team can carry out the evaluation at
any desired moment and the results are quickly implemented in revisions of the product.
Secondly, the developers learn a lot from their evaluation activities. For example, they can
directly experience how the product is used in practice. However, the developers should be
well aware of their natural inclination to become so attached to their own design that they
might not be able to make objective assessments of shortcomings in the design. In that
case, the involvement of external evaluators becomes desirable.