duced by the topic retrieval for either the IRTopics or the
TransTopics methods. Figure 5(b) shows the percentage of
watch videos that have at least one new related video among
the top-K results as a result of topic retrieval.
As Figure 5(a) shows, there is a significant number of results
added at the top ranks as a result of topic retrieval.
Figure 5(b) demonstrates that these results are spread among
watch videos. More than 70% of watch videos are affected
by our retrieval, and have at least one new top-20 result
coming from the topic retrieval stage.
Both of the proposed methods retrieve roughly the same
number of new results, with the TransTopics method introducing
slightly more results at the higher ranks (13.6%
compared to 12.9% at the top ten results). This potentially
indicates the higher relevance of the new results introduced
by the TransTopics method.
Figure 5(b) shows a similar trend. At top ten results, the
TransTopics method affects 73.1% of watch videos, compared
to the 64.6% affected by the IRTopics method.
While the simulation method described in this section is
suitable for the purposes of validation and testing different
variants of the proposed methods, it does not provide a
definitive answer whether the proposed topic-based videos
will indeed have a positive effect on the actual user experience.
To this end, we conduct a large scale live experiment
that is described in the next section.
7.3 Live Experiment
7.3.1 Live Experiment Summary
To evaluate the performance of the IRTopics and the
TransTopics methods, we conducted a large scale experiment
on a random sample of live YouTube traffic. The
experiment was run during a single month in 2013, and
affected related video suggestions returned for millions of
watch videos. The topic weights for both retrieval methods
were updated on the daily basis during the experiment, according
to the method descriptions in Section 3 and Section
4, respectively.
Figure 6 presents a summary of the live experiment findings.
The differences in Figure 6 are reported with respect
to a baseline system that does not employ a topic retrieval