Constructing the item evaluation criteria. More research needs to be done on
choosing or constructing the best set of criteria for evaluating an item. For example,
most of current multi-criteria rating recommenders require users to rate an item on
multiple criteria at a single level (e.g., story and special effects of a movie). This
single level of criteria could be further broken down into sub-criteria, and there
could be multiple levels depending on the given problem. For example, in a movie
recommender system, special effects could be again divided into sound and graphic
effects. More information with multiple levels of criteria could potentially help to
better understand user preferences, and various techniques, such as the analytic hierarchy process (AHP), can be used to consider the hierarchy of criteria [78], as
Schmitt et al. [85] propose to do in their system. As we consider more criteria for
each item, we may also need to carefully examine the correlation among criteria
because the choice of criteria may significantly affect the recommendation quality.
Furthermore, as mentioned earlier, it is important to have a consistent family of
criteria for a given recommender system application because then the criteria are
monotonic, exhaustive, and non-redundant. In summary, constructing a set of criteria for a given recommendation problem is an interesting and important topic for
future research.
Dealing with missing multi-criteria ratings. Multi-criteria recommender systems
typically would require the users to provide more data to such systems than their
single-rating counterparts, thus increasing the likelihood of obtaining missing or incomplete data. One popular technique to deal with missing data is the expectation
maximization (EM) algorithm [18] that finds maximum likelihood estimates for incomplete data. In particular, the probabilistic modeling approach for multi-criteria
rating prediction proposed by [79] uses the EM algorithm to predict values of the
missing ratings in multi-criteria rating settings. The applicability of other existing
techniques in this setting should be explored, and novel techniques could be developed by considering the specifics of multi-criteria information, such as the possible
relationships between different criteria.