Model Comparisons
It is never very informative to evaluate the fit of a single model in isolation (Roberts and Pashler, 2000). Simply fitting a single model to data and finding a high proportion of variance predicted is not very informative because (a) several parameters may be needed to achieve a good fit and so the good fit may simply reflect the use of a lot of free parameters, and (b) simpler models may exist that fit just as well or even better than the model being promoted. Instead it is necessary to compare competing models. Ideally we would like to derive a priori predictions from each model (predictions that do not depend on any parameter estimation from the data), and then determine experimental conditions for which the models make different a priori predictions, and finally experimentally test these a priori predictions. This is the basic approach used for simple axiomatic models described in Chapter 1. Usually, however, our models are too complex to derive a priori and parameter-free predictions. Instead we need to turn to methods that allow predictions to be computed after estimating parameters from the data for the models. In the latter case, we need to perform quantitative model comparisons that evaluate the two scientific criteria of model accuracy and model parsimony.
Usually the competing models differ in their substantive assumptions about the cognitive and or decision processes involved in the decision task. It is important to distinguish between two types of model comparisons. One type is called a nested model comparison in which case a simple model is compared to a more complex model, and the simple model is derived from the more complex model by imposing constraints on the parameters of the more complex model. For example, comparing the four parameter PVL model with a three parameter version produced by setting the loss aversion parameter equal to λ=1 is a nested model comparison. Another type is called a non-nested model comparison in which case one model is not nested within the other model. For example, comparing the decay (4.2a) to the delta (4.2b) learning rule models is a non-nested comparison. Different model comparison methods are used depending on the method of parameter estimation.