The process of obtaining relevance judgments
and the reliability of retrieval experiments are discussed in the TREC book.
Zobel (1998) shows that some incompleteness of relevance judgments does not
affect experiments, although Buckley and Voorhees (2004) suggest that substantial
incompleteness can be a problem. Voorhees and Buckley (2002) discuss the error rates associated with different numbers of queries.