If relevance judgments are not available, however, the ranking can be based
on clickthrough and other user data. For example, if a person clicks on the third
document in a ranking for a query and not on the first two, we can assume that
it should be ranked higher in r. If d1, d2, and d3 are the documents in the first,
pairs (d3, d1) and (d3, d2) being in the desired ranking for this query. This ranking
data will be noisy (because clicks are not relevance judgments) and incomplete,
but there will be a lot of it, and experiments have shown that this type of training
data can be used effectively.