Users provided subjective evaluations. The individuals were given simulated information needs and
were not able to influence on the submitted query. A set of entities were selected, based on a set of
criteria, which formed as a basis for the queries. The simulated information needs were designed to
reflect the queries in a general manner. In this experiment, we observed the difference between the
results of each individual query for each information need (i.e., three queries with different
parameters used for each information need). Based on an information need a user might have liked to
submit an alternative query than used. However, if a user disagrees with a submitted query that
negativism would apply to all three search results for that information need. Consequently, it is
believed to be of minor importance if the submitted query was not optimal according to the users
preferences since that would apply to all the queries for each simulated information need. Moreover,
the Cronbach's alpha scores were above 0,7 indicating reliable evaluation results.