The second aspect of relevance that we consider is whether it is binary or multivalued.
Binary relevance simply means that a document is either relevant or
not relevant. It seems obvious that some documents are less relevant than others,
but still more relevant than documents that are completely off-topic. For example,
we may consider the document containing a list of U.S. presidents to be less
topically relevant than the Lincoln biography, but certainly more relevant than
an advertisement for a Lincoln automobile. Based on this observation, some retrieval
models and evaluation measures explicitly introduce relevance as a multivalued
variable. Multiple levels of relevance are certainly important in evaluation,
when people are asked to judge relevance. Having just three levels (relevant, nonrelevant,
unsure) has been shown to make the judges’ task much easier. In the case
of retrieval models, however, the advantages of multiple levels are less clear.