In this paper, sampling is used to select elements (web pages) to be labeled by human judges. While we present formula for both cases, sampling with replacement is simpler in both formulation and in edge cases in implementation. Additionally, in the large scale, there is a vanishing small probability the same item will be selected. Thus, we use sampling with replacement in the empirical study.