where GI
is a query and ri is partial information about the desired ranking, or relevance
level, of documents for that query. This means that if document da should
be ranked higher than db, then (da, db) ∈ ri; otherwise, (da, db) /∈ ri. Where do
these rankings come from? If relevance judgments are available, the desired ranking
would put all documents judged to be at a higher relevance level above those
at a lower level. Note that this accommodates multiple levels of relevance, which
are often used in evaluations of web search engines.