MOTIVATION
Figure 1 illustrates a typical example of a prediction problem: given some noisy observations
of a dependent variable at certain values of the independent variable
, what is
our best estimate of the dependent variable at a new value, ✂✁
?
If we expect the underlying function ✄✆☎✞✝
to be linear, and can make some assumptions
about the input data, we might use a least-squares method to fit a straight
line (linear regression). Moreover, if we suspect ✄✆☎✂✝
may also be quadratic, cubic, or
even nonpolynomial, we can use the principles of model selection to choose among the
various possibilities.
Gaussian process regression (GPR) is an even finer approach than this. Rather
than claiming ✄✆☎✞✝
relates to some specific models (e.g. ✄✆☎✞✝✠✟☛✡☞✍✌✏✎),
a Gaussian
process can represent ✄✆☎✞✝
obliquely, but rigorously, by letting the data ‘speak’ more
clearly for themselves. GPR is still a form of supervised learning, but the training data
are harnessed in a subtler way.
As such, GPR is a less ‘parametric’ tool. However, it’s not completely free-form,
and if we’re unwilling to make even basic assumptions about ✄✆☎✂✝
, then more general
techniques should be considered, including those underpinned by the principle of
maximum entropy; Chapter 6 of Sivia and Skilling (2006) offers an introduction.