Although the course is taught with an objects first
approach most of the comprehension questions are small
bite size pieces of code and are largely procedural.
Therefore, even if the code is encapsulated in a method,
many of the questions are essentially procedural in
nature.
Of the metrics in Table 1 we selected the subset
which we deemed to be most applicable to measuring the
difficulty of novice code tracing and EiPE tasks:
• Number of statements
• Number of operands (including all identifiers
that are not key words)
• Cyclomatic complexity
• Average nested block depth
• Average number of parameters
One EiPE question involved code that contained two
methods and internal method calls. The object oriented
metric, the number of methods, that had a variation in
value was therefore included as part of our metric set for
EiPE questions.
We did not use KCLID because most of our code
comprehension questions did not contain lines of code
which were not conceptually unique lines of code.
Additionally, we elected not to use the number of
operators metric as the number of operators is
proportional to the number of operands and would
therefore not contribute anything new to the evaluation.
We also supplemented this set of metrics with two
simplified versions of dynamic metrics for the
measurement of the difficulty of the code tracing
questions that we have called the sum of all operands in
the executed statements and the number of executed
program statements. The sum of all operands in the
executed statements was calculated as the sum of all
operands (O) in the executed statements ES where the
total number of executed statements is ν.
Sum of all operands in the executed statements = Σν ESi (O)