or educational background. Below we provide more details
pertaining to these dissonant groups.
Characteristics of the Superficial Processing group
The Superficial Processing group was comprised of 17 students.
As noted, these students succeeded on the MCQ, while making low
scores on the CLA. Qualitative analysis of CLA responses revealed
that these students did not show any serious attempt to analyse,
interpret or synthesise the information from the materials in the task.
Students in this group provided isolated ideas in their written
answers. Their written responses were typically composed of
reproduced or slightly modified portions of text sources, meaning
that the students had not explained the content of the materials in
their own words. The students disregarded much of the information
and used only one or two sources. Nor did the students
recognise presuppositions, such as vested interest. The following
extract describes a typical response in this group (reproduced or
slightly modified portions from the text sources are underlined):
QUESTION 1a: What specific information, evidence, facts, and/or
theories discussed in the Document Library support the inbreeding
explanation?
Document 6, transcription of the radio interview with Dr. Leusid
QUESTION 1b: What are the arguments against the inbreeding
explanation?
In Document 2 Charles Stone writes that the catfish were found in
the same pond that Bonaventure Mills poisoned to death 10 years
ago.He thus blames the mills for what has happened.
In Document 6 Thomas Leusid states that duplications of limbs in
frogs and catfish are caused by a parasite.
Student 28, polytechnic, female
The analysis indicated several weaknesses in producing
arguments. Common to all answers in this group was that students
provided vague arguments. A frequent problem in argumentation
was that the students provided claims or conclusions, but did not
provide valid reasons or explanations to support their claims,
thereby leaving the argument weak. In other words, students’
reasoning was very superficial and thus poor.
These students either did not evaluate the quality of information
or their evaluation of the acceptability of the information was
naı¨ve. Typically, the students first copied short portions of the
source text directly, word for word, or made poor paraphrases of
the materials, changing and modifying a source text slightly, for
example, by adding words or using synonyms. They justified the
information by appealing to authority, tradition or probability. In
appealing to authority, a student argued that something was
correct or true because an authority of some kind had asserted it
was so. Appealing to probability means that a student took
something for granted because it would probably be the case,
whereas in appealing to tradition a student argued that something
is correct because it relates to tradition. The students seemed to
put blind trust in verified empirical evidence, as in the following
statement: ‘‘this document contains numbers, so it seems to be
reliable’’ (Student 25, university, female).
In addition to the above problems in argumentation, the
answers contained other weak arguments. We identified ambiguous
claims (i.e., a student provided claims that were unclear),
irrelevant conclusions (i.e., a student used irrelevant information
or misinterpretation as evidence) or hasty generalizations (i.e., a
student argued from a special case to a general rule based on
insufficient evidence). Some students supported conclusions by
refuting competing conclusions. In this case the different conclusions
were not mutually exclusive.
Characteristics of the Thorough Processing group
The second dissonant group, Thorough Processing, contained
17 test responses in which the students made a stronger showing
on a CLA performance task (an average CLA score by two scorers
was at least 11.5) than on the MCQ (less than 43% of the items were
correctly answered). Students identified the major ideas presented
in the CLA materials. They demonstrated thorough and accurate
understanding of the materials: they evaluated the quality of the
information and considered its premises, as well as the implications
of different conclusions. They weighed different options, connected
related ideas and gave reasoned explanations. The
following extract describes a typical response in the Thorough
Processing group:
QUESTION 1a: What specific information, evidence, facts, and/or
theories discussed in the Document Library support the inbreeding
explanation?
Thomas Leusid says in his interview that many catfish are quite
young – at most three years old, which strongly indicates that the
catfish have come to this area only a short time ago, so it could not
be about Db09-caused toxin concentration – especially since the
Db09 toxin has not, according to Sandy Evans’ study as referred to
by Dr. Munt, caused any mutations. In addition, Leusid’s research
team has found some multiplied structures in the catfish
populations of Miracle Lake, although the feature of three eyes
had not been observed earlier. Leusid’s theory – that only a number
of fertilised eggs have come to Bush Creek – is supported in
particular by the fact that Bush Creek and Miracle Lake greatly
restrict the mobility of fish, and adult catfish live at the bottom of
waterways [and] are not likely to get across the rapids.
Student 5, university, female
Although the students in this group provided mostly convincing
arguments, analysis still revealed that some responses contained
isolated weak arguments. Weak arguments pertained most often
to situations in which the students supported their conclusions by
refuting other, competing conclusions. Other weak arguments
identified were appealing to authority and hasty generalizations.
However, none of students in this group appealed to tradition or
probability, nor did they provide ambiguous claims or irrelevant
conclusions. Furthermore, it was typical that a student who
provided weak arguments also presented relevant and coherent
arguments. In contrast to the Superficial Processing group, these
students provided rationales for their decisions.
Taken together, the students’ test scores within the dissonant
groups represent extreme performance levels. Students who
belonged to the Superficial Processing group were much stronger
in identifying the right answers from a set of predetermined
alternatives than in constructing a response, while the students
who belonged to the Thorough Processing group succeeded much
better in constructing a response in writing than in identifying preformulated
correct answers. The written responses of the
Superficial Processing students revealed several problems in
identifying, interpreting, synthesising and evaluating information,
as well as in producing explanations and arguments in their own
words. By contrast, the written responses of the Thorough
Processing students demonstrated that the task materials were
processed in great depth. These students showed that they knew
how to define problems, analyse, interpret and evaluate the
information on all sides of an issue and use relevant data to
produce arguments and explanations. In the background questionnaire
these students reported that they had put their best
efforts into completing the tests. It is thus somewhat surprising
H. Hyytinen 6 et al. / Studies in Educational Evaluation 44 (2015) 1–8